Skip to content

Adds structured logging API. #665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jun 2, 2020
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
- Adds `functions.logger` SDK to enabled structured logging in the Node.js 10 runtime. Example:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should add that in Node.js versions < 10, the method still is supported but the data is displayed in a single line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo, should be "to enable"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nit, but in a text flow we prefer "For example:"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

```js
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, does this work? In the d.ts files we use ```javascript, but a shorter form would be nice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup 😄

const functions = require('firebase-functions');

functions.logger.debug('example log with structured data', {
uid: user.uid,
authorized: true,
});
```

The logger can also override default behavior of `console.*` methods through a special require:

```js
require('firebase-functions/logger/compat');
```

In older runtimes, the logger will print to console, no structured data is saved.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest "logger prints to the console, and no..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

105 changes: 105 additions & 0 deletions spec/logger.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
import { expect } from 'chai';
import * as sinon from 'sinon';
import * as logger from '../src/logger';

const SUPPORTS_STRUCTURED_LOGS =
parseInt(process.versions?.node?.split('.')?.[0] || '8', 10) >= 10;

describe(`logger (${
SUPPORTS_STRUCTURED_LOGS ? 'structured' : 'unstructured'
})`, () => {
let sandbox: sinon.SinonSandbox;
let stdoutStub: sinon.SinonStub;
let stderrStub: sinon.SinonStub;

beforeEach(() => {
sandbox = sinon.createSandbox();
stdoutStub = sandbox.stub(process.stdout, 'write');
stderrStub = sandbox.stub(process.stderr, 'write');
});

function expectOutput(stdStub: sinon.SinonStub, entry: any) {
if (SUPPORTS_STRUCTURED_LOGS) {
return expect(
JSON.parse((stdStub.getCalls()[0].args[0] as string).trim())
).to.deep.eq(entry);
} else {
// legacy logging is not structured, but do a sanity check
return expect(stdStub.getCalls()[0].args[0]).to.include(entry.message);
}
}

function expectStdout(entry: any) {
return expectOutput(stdoutStub, entry);
}

function expectStderr(entry: any) {
return expectOutput(stderrStub, entry);
}

describe('logging methods', () => {
let writeStub: sinon.SinonStub;
beforeEach(() => {
writeStub = sinon.stub(logger, 'write');
});

afterEach(() => {
writeStub.restore();
});

it('should coalesce arguments into the message', () => {
logger.log('hello', { middle: 'obj' }, 'end message');
expectStdout({
severity: 'INFO',
message: "hello { middle: 'obj' } end message",
});
sandbox.restore(); // to avoid swallowing test runner output
});

it('should merge structured data from the last argument', () => {
logger.log('hello', 'world', { additional: 'context' });
expectStdout({
severity: 'INFO',
message: 'hello world',
additional: 'context',
});
sandbox.restore(); // to avoid swallowing test runner output
});
});

describe('write', () => {
describe('structured logging', () => {
describe('write', () => {
for (const severity of ['DEBUG', 'INFO', 'NOTICE']) {
it(`should output ${severity} severity to stdout`, () => {
let entry: logger.LogEntry = {
severity: severity as logger.LogSeverity,
message: 'test',
};
logger.write(entry);
expectStdout(entry);
sandbox.restore(); // to avoid swallowing test runner output
});
}

for (const severity of [
'WARNING',
'ERROR',
'CRITICAL',
'ALERT',
'EMERGENCY',
]) {
it(`should output ${severity} severity to stderr`, () => {
let entry: logger.LogEntry = {
severity: severity as logger.LogSeverity,
message: 'test',
};
logger.write(entry);
expectStderr(entry);
sandbox.restore(); // to avoid swallowing test runner output
});
}
});
});
});
});
2 changes: 2 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ import * as storage from './providers/storage';
import * as testLab from './providers/testLab';

import * as apps from './apps';
import * as logger from './logger';
import { handler } from './handler-builder';
import { setup } from './setup';

Expand All @@ -51,6 +52,7 @@ export {
remoteConfig,
storage,
testLab,
logger,
};

// Exported root types:
Expand Down
135 changes: 135 additions & 0 deletions src/logger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
import { format } from 'util';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this import got in the middle of two lines of comments?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, fixed.


// safely preserve unpatched console.* methods in case of compat require
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will these be parsed into the reference by our TypeDoc script? If so, suggest "//Safely . . . require." with init cap and period.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope.

const unpatchedConsole = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this still won't work if somebody did this:

require('firebase-functions/logger/compat');
// Already overwrote console.* methods before requiring "firebase-functions"
const functions = require('firebase-functions');
// unpatchedConsole is actually the patched console
functions.logger.info('something') // will probably do weird things

I think there's many opportunities for there to be weird bugs and edge cases with 'compat', and I'm not sure it's worth the incremental convenience of providing this functionality.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does work, because the first thing compat does is require logger which preserves the unpatched console methods. Added a comment to clarify.

This wasn't in my original proposal but Kato asked if it was possible and it turned out to be easier than expected. I do worry about the chance for bugs here but this is just one way to use the SDK and provides a convenience. Anyone who encounters bugs we can just tell to use the other ways of using the SDK.

Copy link
Contributor

@laurenzlong laurenzlong Apr 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, compat does import logger first, this is really subtle behavior, it's good you added a comment to clarify.

I'm still not a fan of compat and I think it's just making this SDK harder to maintain (it's already way too complicated and has too many subtle behaviors) But if you feel like the risk is worth the user experience benefit, we can keep it in.

debug: console.debug,
info: console.info,
log: console.log,
warn: console.warn,
error: console.error,
};

// Determine if structured logs are supported (node >= 10). If something goes wrong,
// assume no since unstructured is safer.
const SUPPORTS_STRUCTURED_LOGS =
parseInt(process.versions?.node?.split('.')?.[0] || '8', 10) >= 10;

// Map LogSeverity types to their equivalent `console.*` method.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this goes to public docs, let's backtick LogSeverity or just say "log severity" in the text flow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't be in public docs.

const CONSOLE_SEVERITY: {
[severity: string]: 'debug' | 'info' | 'warn' | 'error';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I'm not sure if TS would let you do this, but if it does). This would be clearer as:
[severity: LogSeverity]: 'debug' | 'info' | 'warn' | 'error';

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TS doesn't let you do this 😢

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😢

} = {
DEBUG: 'debug',
INFO: 'info',
NOTICE: 'info',
WARNING: 'warn',
ERROR: 'error',
CRITICAL: 'error',
ALERT: 'error',
EMERGENCY: 'error',
};

/**
* LogSeverity indicates the detailed severity of the log entry. See [Cloud Logging docs](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#logseverity) for more.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest backticks for LogSeverity

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest the actual doc section title, LogSeverity

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

*/
export type LogSeverity =
| 'DEBUG'
| 'INFO'
| 'NOTICE'
| 'WARNING'
| 'ERROR'
| 'CRITICAL'
| 'ALERT'
| 'EMERGENCY';

/**
* LogEntry represents a structured Cloud Logging entry. All keys aside from `severity` and `message` are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest backticks for LogEntry

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* included in the `jsonPayload` of the logged entry.
*/
export interface LogEntry {
severity: LogSeverity;
message?: string;
[key: string]: any;
}

/**
* Writes a LogEntry to `stdout`/`stderr` (depending on severity).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest backticks for LogEntry (and every other literal that appears in the HTML output for the reference docs).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* @param entry The LogEntry including severity, message, and any additional structured metadata.
*/
export function write(entry: LogEntry) {
if (SUPPORTS_STRUCTURED_LOGS) {
unpatchedConsole[CONSOLE_SEVERITY[entry.severity]](JSON.stringify(entry));
return;
}

let message = entry.message || '';
const jsonPayload: { [key: string]: any } = {};
let jsonKeyCount = 0;
for (const k in entry) {
if (!['severity', 'message'].includes(k)) {
jsonKeyCount++;
jsonPayload[k] = entry[k];
}
}
if (jsonKeyCount > 0) {
message = `${message} ${JSON.stringify(jsonPayload, null, 2)}`;
}
unpatchedConsole[CONSOLE_SEVERITY[entry.severity]](message);
}

/**
* Writes a `DEBUG` severity log. If the last argument provided is a plain object,
* it will be added to the `jsonPayload` in the Cloud Logging entry.
* @param args Arguments, concatenated into the log message with space separators.
*/
export function debug(...args: any[]) {
write(entryFromArgs('DEBUG', args));
}

/**
* Writes an `INFO` severity log. If the last argument provided is a plain object,
* it will be added to the `jsonPayload` in the Cloud Logging entry.
* @param args Arguments, concatenated into the log message with space separators.
*/
export function log(...args: any[]) {
write(entryFromArgs('INFO', args));
}

/**
* Writes an `INFO` severity log. If the last argument provided is a plain object,
* it will be added to the `jsonPayload` in the Cloud Logging entry.
* @param args Arguments, concatenated into the log message with space separators.
*/
export function info(...args: any[]) {
write(entryFromArgs('INFO', args));
}

/**
* Writes a `WARNING` severity log. If the last argument provided is a plain object,
* it will be added to the `jsonPayload` in the Cloud Logging entry.
* @param args Arguments, concatenated into the log message with space separators.
*/
export function warn(...args: any[]) {
write(entryFromArgs('WARNING', args));
}

/**
* Writes an `ERROR` severity log. If the last argument provided is a plain object,
* it will be added to the `jsonPayload` in the Cloud Logging entry.
* @param args Arguments, concatenated into the log message with space separators.
*/
export function error(...args: any[]) {
write(entryFromArgs('ERROR', args));
}

function entryFromArgs(severity: LogSeverity, args: any[]): LogEntry {
let entry = {};
const lastArg = args[args.length - 1];
if (typeof lastArg == 'object' && lastArg.constructor == Object) {
entry = args.pop();
}
return Object.assign({}, entry, {
severity,
// mimic `console.*` behavior, see https://nodejs.org/api/console.html#console_console_log_data_args
message: format.apply(null, args),
});
}
8 changes: 8 additions & 0 deletions src/logger/compat.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { debug, info, warn, error } from '../logger';

// IMPORTANT -- "../logger" must be imported before monkeypatching!
console.debug = debug;
console.info = info;
console.log = info;
console.warn = warn;
console.error = error;
2 changes: 1 addition & 1 deletion tsconfig.release.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@
"target": "es2017",
"typeRoots": ["./node_modules/@types"]
},
"files": ["./src/index.ts"]
"files": ["./src/index.ts", "./src/logger.ts", "./src/logger/compat.ts"]
}