Event parsing doesn't match Cloud Functions in background mode

[ericnorris]

Hey there!

Related to #41 and #85, I ran into a lot of confusion trying to sort out what the format should be for the request body to a local background function.

In production a background function has the following for the first parameter:

{ @type: 'type.googleapis.com/google.pubsub.v1.PubsubMessage',
  attributes: null,
  data:  '<base64 data>' }

and the following for the "context" parameter:

{ eventId: '749799261766789',
  timestamp: '2019-09-17T19:56:09.018Z',
  eventType: 'providers/cloud.pubsub/eventTypes/topic.publish',
  resource: '<pubsub topic>' }

The current docs for using the functions framework for events suggests curling data in a format that results in the parameters not matching the above - the first parameter becomes:

{ message:
   { attributes: {},
     data: '<base64 data>',
     messageId: '136969346945' },
  subscription: 'projects/myproject/subscriptions/mysubscription' }

And the "context" parameter:

{ type: 'true',
  specversion: 'true',
  source: 'true',
  id: 'true' }

Digging into the source code,

functions-framework-nodejs/src/invoker.ts

Lines 431 to 447 in de6ef38

	let data = event.data;
	let context = event.context;
	if (isBinaryCloudEvent(req)) {
	// Support CloudEvents in binary content mode, with data being the whole
	// request body and context attributes retrieved from request headers.
	data = event;
	context = getBinaryCloudEventContext(req);
	} else if (context === undefined) {
	// Support legacy events and CloudEvents in structured content mode, with
	// context properties represented as event top-level properties.
	// Context is everything but data.
	context = event;
	// Clear the property before removing field so the data object
	// is not deleted.
	context.data = undefined;
	delete context.data;
	}

seems to expect the body (for "legacy events") to look something like:

{ data:
  { data: '<base64 data>`,
    attributes: null,
    @type: 'type.googleapis.com/google.pubsub.v1.PubsubMessage' },
  eventId: 749799261766789 }

(Further evidence this is expected is in this WIP branch for NodeJS samples.)

Using the above POST body seems to result in function parameters that most closely match production, and although it doesn't match the POST body for push messages I'm assuming that's what users should send if they want their function to act closest to what they will see when they deploy to Cloud Functions.

Is this correct?

[grant]

I'm not sure the expected behavior in this case to be honest.
@ace-n Do you know more about event parsing in GCF background functions?

[ace-n]

cc @mogar1980 @swalkowski

[swalkowski]

The Functions Framework supports multiple event formats:

• The doc that you mentioned describes handling events conforming to the CloudEvents spec, in structured or binary mode.
• However, Google Cloud Functions service currently does not provide events in CloudEvents format. Instead, it uses the structured JSON format in one of the following forms:
  • "data" and "context" attributes.
  • "data" attribute and individual context attributes inlined at the top level, next to "data" (the legacy format).

If you want to trigger your function with CloudEvents, which may be useful for integrations beyond Cloud Functions, use the CloudEvents format.

But if you want to trigger the function like a background function in Cloud Functions service, use one of the other formats mentioned above.

We should clarify this in the Functions Framework docs.

[burtonjc]

@swalkowski, I posted a comment on this issue too explaining the behavior I am seeing, but it seems related to this issue as well.

What I am trying to accomplish is a local dev environment where:

1. functions-framework is serving a GCF pubsub triggered function locally
2. gcloud PubSub emulator is running locally
3. There is a push subscription in the emulator with a push endpoint to the functions-framework server
4. Then publish messages to the subscription topic to trigger the function

Expected behvior: the function is executed with arguments similar to those it receives while running in GCF.

Actual behavrior: the function is executed with the first argument being undefined and the second like:

{
    "subscription": "projects/gcp-nodejs-demo/subscriptions/local-receivePubSub-demo",
    "message": {
        "data": "<encoded string>",
        "messageId": "1",
        "attributes": { "transactionId":"72c28439-5c5d-4e59-a032-e8f622785835" }
    }
}

Is this a problem with the emulator not sending the same message format as production PubSub does? Or is it a problem with how functions-framework is parsing the event?

[m0ar]

@burtonjc Did you find any solutions for this issue? Pains us as well trying to setup a local devenv for cloud functions.

[burtonjc]

@m0ar, unfortunately no, at least not anything elegant. You already commented on it, but there is some good conversation on #41 as well.

[grant]

Here's a sample integration test using the event parsing for Cloud Functions in background mode for Pub/Sub:

https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/master/functions/helloworld/test/sample.integration.pubsub.test.js

We have lots of other samples for GCF too in that repo.
We don't use the Pub/Sub emulator, that may be a source of a problem.