Node 8 functions ending with "finished with status: 'connection error'"

[diginikkari]

Firebase function triggered by firestore trigger is ending prematurely with finished with status: 'connection error'. There are no errors logged from actual function. Phase where function is exiting is varying. Most of the functions (lighter functions, with less communication to firestore) are finishing ok.

When runtime is changed to "Node.js 10 Beta" functions are working correctly.

Region: europe-west1
Memory allocated: 1 GB
Timeout: 300 seconds

Related issues

#429 (already closed)

[REQUIRED] Version info

node:
node 8 (in cloud)
v12.4.0 (locally)

firebase-functions:
"firebase-functions": "3.3.0"

firebase-tools:
"firebase-tools": "7.12.1"

7.12.1

firebase-admin:
"firebase-admin": "8.9.1"

[REQUIRED] Test case

have not been able to extract code to reproduce it outside of my codebase. I have case where this is happening 100% of executions.

[REQUIRED] Steps to reproduce

Run function which is querying multiple documents from firestore using firebase-admin.

repo for reproducing issue:
https://github.com/diginikkari/case00032967

Reproduction steps:

1. create firestore project
2. Initialize firestore database for project
3. get service account credentials and save it root folder with name: service-account.json
4. build and deploy functions npm run build; npm run deploy
5. generate fake test data npm run generate-fake-data
6. call function which add order document to firestore, which then triggers the firestore function trigger npm run create-order-copy
7. check logs npm run logs

When function is using "node 8" it will end with finished with status: 'connection error'.
When changing to "node 10" function is ending finished with status: 'ok'

[REQUIRED] Expected behavior

Function is finished succesfully.

[REQUIRED] Actual behavior

Function is ending prematurely with finished with status: 'connection error' message.

Were you able to successfully deploy your functions?

Yes, there were no errors with deployment.

log from errored execution.txt

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[steven-tib]

Hi, I'm having similar issues since few hours. same message finished with status: 'connection error' everything worked fine earlier and I only did minor changes in the code.

Any progress on your side @diginikkari ?

[diginikkari]

@steven-tib I changed function runtime to node.js 10. This was desperate try because nothing else helped and for some reason it worked at least for me. So perhaps that is something you could try also.

I'm bit worried on using node.js 10 because it's still in beta. But I figured that what I have to lose. I keep my functions in node 10 at least until this issue is solved.

[steven-tib]

@diginikkari it seems that this workaround solves it, at least for now. I'm not confident either to use the node 10 beta so I hope they will find a way to fix this soon. Kiitos avusta!

[yeldarby]

This is also happening to us. Seems to be intermittent:

[c0rnonthec0bb]

We're also experiencing this issue on a scheduled function that runs every minute. We first experienced the "connection error" result at 5:51pm PST on Jan 18th. As with the others in this thread, using the Node 10 Beta runtime fixes the issue, but that obviously is not an ideal solution.

[yeldarby]

We are also getting this error on our Python cloud function :-/

The Node 10 workaround won't work for that one, any suggestion?

[neilpoulin]

Yep, had this issue started showing up recently for me, too. It's a nearly silent error as it didn't' trigger any of my existing alerting. I banged my head against this for many hours today, thinking I was doing something wrong while trying to process ~10,000 records (which I had been doing without issue in many cloud functions using the same shared code for weeks).

Upgrading to the Node 10 (Beta) runtime also seems to work for me, but I'm not thrilled at being forced to update to a Beta product before I was prepared to do so. My dev machine and all CI/CD pipelines all run Node 8, so having using a different version only in production is bad.

I really just wish that Firebase (or GCP) could get ahead of some of these problems by at least alerting us that there may be a problem, rather than us having to notice jobs "randomly" failing with no useful error codes.

[TomasMasiulis]

Same error on our environment.
Function is triggered by pubsub trigger every night.
Error happens on ~30 second of execution regardless that timeout is set to 540 seconds.

Is there a away to upgrade only one function to the Node 10, instead of upgrading full project ?

[diginikkari]

@TomasMasiulis You can change runtime in https://console.cloud.google.com/functions select you project and select the function you wan't to change. Then press edit button and select Node.js 10 (Beta) from runtime dropdown.

[eleith]

please note that two weeks ago, we found that when we switched our functions over to node 10 we saw a non-trivial increase of memory usage (a simple hello world exhibited this behavior, almost doubling the amount of memory needed).

we reported our experience and measurements to the firebase team and their support channel acknowledged that this is a beta product and encouraged us to stick with node 8.

we are also experiencing this error, but are hesitant to use the workaround until we can determine that the memory usage matches that of node 8.

[swftvsn]

We're also seeing this. Any word from Firebasers / Google?

[swftvsn]

So, some of our more busy CF are really taking a hit: This log is from last night: (Filtered out the successful runs.)

4:01:02.098 am myFunc Function execution took 231 ms, finished with status:'connection error'
4:01:00.765 am myFunc Function execution took 128 ms, finished with status:'connection error'
4:00:59.745 am myFunc Function execution took 512 ms, finished with status:'connection error'
4:00:58.982 am myFunc Function execution took 223 ms, finished with status:'connection error'
4:00:58.768 am myFunc Function execution took 207 ms, finished with status:'connection error'
4:00:58.671 am myFunc Function execution took 130 ms, finished with status:'connection error'
4:00:58.081 am myFunc Function execution took 444 ms, finished with status:'connection error'
4:00:57.644 am myFunc Function execution took 614 ms, finished with status:'connection error'
4:00:57.620 am myFunc Function execution took 578 ms, finished with status:'connection error'
4:00:57.463 am myFunc Function execution took 268 ms, finished with status:'connection error'
4:00:57.352 am myFunc Function execution took 500 ms, finished with status:'connection error'
4:00:57.065 am myFunc Function execution took 502 ms, finished with status:'connection error'
4:00:56.453 am myFunc Function execution took 159 ms, finished with status:'connection error'
4:00:56.205 am myFunc Function execution took 312 ms, finished with status:'connection error'
4:00:55.841 am myFunc Function execution took 140 ms, finished with status:'connection error'
4:00:55.814 am myFunc Function execution took 179 ms, finished with status:'connection error'
4:00:55.197 am myFunc Function execution took 343 ms, finished with status:'connection error'
4:00:51.733 am myFunc Function execution took 148 ms, finished with status:'connection error'
4:00:51.247 am myFunc Function execution took 560 ms, finished with status:'connection error'
4:00:50.596 am myFunc Function execution took 217 ms, finished with status:'connection error'
4:00:49.965 am myFunc Function execution took 545 ms, finished with status:'connection error'
4:00:42.818 am myFunc Function execution took 570 ms, finished with status:'connection error'
4:00:38.329 am myFunc Function execution took 407 ms, finished with status:'connection error'

Thank god we have work queues set up due to past unreliability issues. Still, we need to do extra work due to this :/

[TomasMasiulis]

@TomasMasiulis You can change runtime in https://console.cloud.google.com/functions select you project and select the function you wan't to change. Then press edit button and select Node.js 10 (Beta) from runtime dropdown.

@diginikkari thanks a lot !

Cannot deploy function after changing runtime to Node.js 10 (Beta).
After clicking Deploy in it gives me an error :(

Deployment failure:
Build failed: {"error": {"canonicalCode": "INVALID_ARGUMENT", "errorMessage": "npm_install had stderr output:\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/faye-websocket-486c59bd/package.json'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/websocket-driver-ea6ccb07/package.json'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/faye-websocket-486c59bd/CHANGELOG.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/websocket-driver-ea6ccb07/CHANGELOG.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/faye-websocket-486c59bd/LICENSE.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/websocket-driver-ea6ccb07/LICENSE.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/faye-websocket-486c59bd/README.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/websocket-driver-ea6ccb07/README.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/escodegen-8a99b129/LICENSE.BSD'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/escodegen-8a99b129/README.md'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/module.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/net.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/os.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/package.json'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/acorn-776c12f1/dist/walk.es.js'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/path.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/moment-timezone-9832a231/builds/moment-timezone-with-data-2012-2022.js'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/exceljs-7d37de6b/dist/es5/doc/column.js'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@firebase/util-e86a055f/dist/index.node.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/perf_hooks.d.ts'\nnpm WARN tar ENOENT: no such file or directory, lstat '/workspace/functions/node_modules/.staging/@firebase/util-e86a055f/dist/src'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/process.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/exceljs-7d37de6b/dist/es5/doc/column.js.map'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@google-cloud/firestore-18e9025d/build/protos/firestore_v1_proto_api.js'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/punycode.d.ts'\nnpm WARN tar ENOENT: no such file or directory, lstat '/workspace/functions/node_modules/.staging/@firebase/util-e86a055f/dist/src'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/querystring.d.ts'\nnpm WARN tar ENOENT: no such file or directory, open '/workspace/functions/node_modules/.staging/@types/node-585a004e/readline.d.ts'\nnpm WARN tar ENOENT: no such file or directory, lstat '/workspace/functions/node_modules/.staging/@firebase/util-e86a055f/dist/src'\nnpm WARN tar ENOENT: no such fi

[swftvsn]

Also in case it slipped from someone: Cloud Functions is GA, Firestore is GA but Firestore Cloud Function Triggers are still in beta, as stated here:

Cloud Firestore triggers for Cloud Functions is a beta feature with some known limitations

https://firebase.google.com/docs/functions/firestore-events (at the bottom of the page.)

[miketrebilcock]

We had the issue too, having raised a support case with GCP, this has now been reported to us as resolved and we're not seeing the error any more after having redpeloyed our node 8 funtctions.

[dochathuc]

Two days ago, I deployed my clouds function again with minor changes in the code. I got 'connection error'. After looking at the code, I saw that this error only happened in functions which are:

• Scheduled functions (cronjob and pubsub)
• Using await Promise.all()
  I decided to convert 'Promise.all' to 'reduce', then this error happens less, but still exists.

PS: I'm using Node 8

[diginikkari]

I also got info from my support case that this has been internally fixed. I was not able to reproduce this anymore so closing it. I will redeploy my function with runtime node.js 8.

[swftvsn]

Can verify that they reported as fixed for us too and we're not seeing the behavior anymore after full redeploy.

[ciphereck]

@diginikkari I'm still getting this error. Is there any way to get a work around it? or how to resolve it?

[diginikkari]

@ciphereck Issue which affected this error for me was some firebase internal issue which they fixed.

I suggest that you test with node.js 10 as it was one solution when I had the issue. Functions are in anycase now moving from node.js 8 to node.js 10. If that doesn't help, be in contact with firebase support.

[emadalam]

I don't think this has been fixed. It started happening to us of lately, with no code changes whatsoever. Just re-deploying those functions since we use automated deployments with Gitlab pipelines and have all functions deployed with every release. Firebase support team has no answers or acknowledgment of this problem just yet, very annoying. However our internal investigation points to the setup something similar to this comment from here #611 (comment), where the problem is happening consistently.

• Cloud Function Node runtime version 8
• Deployment happening from a Gitlab Pipeline using docker image (https://hub.docker.com/r/partumgmbh/gcp-pipeline) with firebase-tools 7.14.0 and Node v10.19.0 installed
• Cloud Function of type pubsub.schedule (which automatically creates a trigger in Google Cloud Scheduler)
• Cloud Function returning either Promise.all or await Promise.all in all those connection error cases
• Underlying promises arrays are mostly trivial write operations to either Firestore or BigQuery inserts, but a lot of them

Migrating to Node 10 is not the solution just yet as it has lot of other implications. We plan to do the migration during end of the year, but until then we want the Node 8 runtime for cloud functions to be running as before. This sudden platform bug is more than undesirable. Any help, workaround, suggestions would be more than welcome.

[glasperfan]

I also have a clear repro as well if it's useful. Fix was to move to Node 10. I guess Node 8 is more than deprecated 😄

• Node 8 Cloud Function
• CLI Deployment (firebase --version = 8.10.0)
• Scheduled function (pubsub.schedule)
• Async function
• Async requests made - get all documents from Firestore collection (~2K), upload file to Cloud Storage

[prene]

I also have a clear repro as well if it's useful. Fix was to move to Node 10. I guess Node 8 is more than deprecated 😄

• Node 8 Cloud Function
• CLI Deployment (firebase --version = 8.10.0)
• Scheduled function (pubsub.schedule)
• Async function
• Async requests made - get all documents from Firestore collection (~2K), upload file to Cloud Storage

Pretty much the same for me. Async function, getting large number of documents. Problem does not appear on our dev environment, with a significantly smaller number of documents.

[davestimpert]

Just started seeing the same error on node 16

[derrickrc]

Seeing lots of these errors with one of our Cloud Functions, Node 14. I compared this Cloud Function to another of our Cloud Functions that has a higher rate of invocations but very few of these connection errors, and that function has a much higher 2GB memory allocation. I've redeployed our other function with more memory to see if this helps.