crash due to null activeQuery in lib/client.js

[spollack]

We had a crash in production last night in the pg module. (we just updated to pg 4.5.1 javascript, after running pg 2.11.1 native for many many months. this is the first time i've seen this issue.)

The crash was in lib/client.js:123:

  self.activeQuery.handleCommandComplete(msg, con);

stack trace:

TypeError: Cannot call method 'handleCommandComplete' of null
    at null.<anonymous> (/app/node_modules/pg/lib/client.js:123:24)
    at emit (events.js:95:17)
    at Socket.<anonymous> (/app/node_modules/pg/lib/connection.js:109:12)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:765:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)
    at emitReadable (_stream_readable.js:423:5)
    at readableAddChunk (_stream_readable.js:166:9)
    at Socket.Readable.push (_stream_readable.js:128:10)
    at TCP.onread (net.js:529:21)

i do not have a repro case at this time.

obviously we could put guards in this function (and its peers) to check if self.activeQuery exists before calling the function. but not sure how we got in this state in the first place. @brianc thoughts on this? thanks

[vitaly-t]

In cases like this one, it is worth trying to roll back specifically node-postgres version, to see whether the module itself caused the issue or one of its updated dependencies, which I reckon was the latter, by the look of the error log.

But then again, if you cannot reproduce the issue, your guess is as good as anyone's. You won't be able to fix it without knowing how to check that it's gone.

Figure out how to reproduce the issue and get back ;)

[spollack]

Hi Vitaly, not sure i follow, the only code on the stack when the crash happens is node internals and the pg library itself, no other dependency libraries. That of course doesn't mean that a change to a pg dependency didn't lead to us getting in this state, but i'm curious what in the error log leads you to believe that? thanks.

[spollack]

We have now had about 10 crashes in production due to this issue. Last week we made a local patch to pg to add guards as suggested above, and that resolved the issue. We will open a PR here for that patch. cc @jshepard

[spollack]

@jshepard dug into this more, and it appears that this problem was likely caused by an issue in another library we were using on top of pg. given that, closing this.

[brianc]

Glad you got it figured out!!! 💃

On Monday, June 6, 2016, Seth Pollack [email protected] wrote:

Closed #949 #949.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#949 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AADDoQYaNg3LF3KpFiEQOgH_bFrcOoRjks5qJJhbgaJpZM4HkBad
.