Skip to content

If the connection to PG fails during execution, it will cause the program to crash (infinite loop) #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VxRain opened this issue Jun 17, 2024 · 10 comments
Labels
wontfix This will not be worked on

Comments

@VxRain
Copy link

VxRain commented Jun 17, 2024

Env:

  • Win10
  • Node v20.14.0
  • kysely: 0.27.3
  • kysely-postgres-js: 2.0.0,
  • postgres: 3.4.4

Reproduction steps:

  1. Start the PostgreSQL service.
  2. Start the application.
  3. Kill the PostgreSQL process.
  4. The error repeats infinitely.
import { Kysely } from "kysely";
import { PostgresJSDialect } from "kysely-postgres-js";
import postgres from "postgres";

// listen for unhandledRejection and uncaughtException
process.on("unhandledRejection", (reason) => {
  console.error("unhandledRejection: ", reason);
});
process.on("uncaughtException", (err) => {
  console.error("uncaughtException: ", err);
});

const pg = postgres({
  host: "localhost",
  max: 10,
  port: 5432,
  database: "test",
  user: "test",
  password: "test",
});
const db = new Kysely({ dialect: new PostgresJSDialect({ postgres: pg }) });

async function main() {
  await pg`SELECT 1`; // check connection

  setInterval(async () => {
    const user = await db
      .selectFrom("User")
      .select("username")
      .where("id", "=", 1)
      .executeTakeFirstOrThrow();
    console.log(Date.now(), user);
  }, 1000);
}
main();

Error logs:

uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Additionally, I found that if the connection to PostgreSQL fails when app start, the Kysely instance initializes normally, but executing a query throws an error: "Cannot read properties of undefined (reading 'replace')."

Related to #9, #10, #11, I think we should investigate further to resolve the related issues.

@VxRain
Copy link
Author

VxRain commented Jun 17, 2024

I don't think this is an issue with Postgres.js. I also tested it using Postgres.js. After killing the PG process, Postgres.js correctly throws an error and attempts to reconnect. Once the reconnection is successful, the program outputs results correctly, which is the expected behavior.

@joshxyzhimself
Copy link

file://server/node_modules/postgres/src/connection.js:389
      stack: { value: err.stack + query.origin.replace(/.*\n/, '\n'), enumerable: options.debug },
                                               ^

TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file://server/node_modules/postgres/src/connection.js:389:48)
    at errored (file://server/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file://server/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1

this only happens when i try to run my nodejs server while postgresql db server is off or inaccessible (e.g. due to docker initialization)

in docker postgres if you have init sql scripts it will restart once. if your nodejs server tries to connect while it's initializing / restarting, you'll get this error above.

@VxRain
Copy link
Author

VxRain commented Jun 18, 2024

@joshxyzhimself

Hi, I didn't encounter this issue when using only Postgres.js. Does this problem occur only when using the Docker image? Could you provide a minimal reproducible code or repository?

@joshxyzhimself
Copy link

import { Kysely } from "kysely";
import { PostgresJSDialect } from "kysely-postgres-js";
import postgres from "postgres";

import type { KyselyDatabase } from "./index.js";

const POSTGRES_HOST = "localhost";
const POSTGRES_PORT = 5432;
const POSTGRES_DB = "postgres";
const POSTGRES_USER = "postgres";
const POSTGRES_PASSWORD = "postgres";

export const pgc = postgres({
  database: POSTGRES_DB,
  host: POSTGRES_HOST,
  port: POSTGRES_PORT,
  user: POSTGRES_USER,
  password: POSTGRES_PASSWORD,
  max: 10,
});

export const pgd = new PostgresJSDialect({ postgres: pgc });

export const pgdb = new Kysely<KyselyDatabase>({ dialect: pgd });

try {
  const users = await pgc`SELECT * FROM users;`;
  console.log({ users });
} catch (e) {
  if (e instanceof Error) {
    console.error("POSTGRES.JS ERROR CAUGHT");
    console.error(e);
  }
}

try {
  const users = await pgdb.selectFrom("users").selectAll().execute();
  console.log({ users });
} catch (e) {
  if (e instanceof Error) {
    console.error("KYSELY ERROR CAUGHT");
    console.error(e);
  }
}
user@device:~/Documents/server$ tsc
user@device:~/Documents/server$ node ./dist/postgres.mjs
POSTGRES.JS ERROR CAUGHT
Error: connect ECONNREFUSED 127.0.0.1:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1606:16)
    at cachedError (file:///home/user/Documents/server/node_modules/postgres/src/query.js:170:23)
    at new Query (file:///home/user/Documents/server/node_modules/postgres/src/query.js:36:24)
    at sql (file:///home/user/Documents/server/node_modules/postgres/src/index.js:112:11)
    at file:///home/user/Documents/server/dist/postgres.mjs:20:29 {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 5432
}
file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389
      stack: { value: err.stack + query.origin.replace(/.*\n/, '\n'), enumerable: options.debug },
                                               ^

TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1
user@device:~/Documents/server$ npx tsx ./src/postgres.mts 
POSTGRES.JS ERROR CAUGHT
Error: connect ECONNREFUSED 127.0.0.1:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1606:16)
    at cachedError (file:///home/user/Documents/server/node_modules/postgres/src/query.js:170:23)
    at new Query (file:///home/user/Documents/server/node_modules/postgres/src/query.js:36:24)
    at sql (file:///home/user/Documents/server/node_modules/postgres/src/index.js:112:11)
    at <anonymous> (/home/user/Documents/server/src/postgres.mts:27:23) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 5432
}
file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389
      stack: { value: err.stack + query.origin.replace(/.*\n/, '\n'), enumerable: options.debug },
                                               ^

TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1

dang it never reached console.error("KYSELY ERROR CAUGHT");

@joshxyzhimself
Copy link

Hi, I didn't encounter this issue when using only Postgres.js. Does this problem occur only when using the Docker image? Could you provide a minimal reproducible code or repository?

nope, it also occurs even in my localhost (no docker).

example provided above.

same results with tsc & tsx compiled typescript code.

you're right that it throws proper econrefused when using postgres.js, but when using the kysely interface it doesn't throw a catchable error at all.

@joshxyzhimself
Copy link

Found this issue upstream, they're using mssql but similar problem where app crashes when db unreachable.

kysely-org/kysely#836

@VxRain
Copy link
Author

VxRain commented Jun 18, 2024

@joshxyzhimself Yes, we are encountering the same issue.

Currently, we can use

process.on("uncaughtException", (err) => {
  console.error("uncaughtException: ", err);
});

to catch the error. But in this case, it isn't very helpful because it keeps throwing errors in an infinite loop, blocking the main process and making the service completely unusable. I think this might need a thorough investigation @igalklebanov

@igalklebanov
Copy link
Member

igalklebanov commented Jun 19, 2024

Hey 👋

This happens in postgres.js when using the reserve() method. See porsager/postgres#778 (comment). A bug.

For now, use this library @ v1.x.x (this is before reserve() was introduced in postgres.js).

Alternatively, extend the driver, and override the acquireConnection method with a retry mechanism around the .reserve() invocation. Retrying is a connection pool's responsibility, so this cannot be shipped in this library.

Alternatively, don't execute with Kysely, but compile and pass the compiled queries straight to postgres.js's unsafe method.

Alternatively, patch postgres.js with a fix.

@igalklebanov igalklebanov added the wontfix This will not be worked on label Jun 19, 2024
@igalklebanov
Copy link
Member

igalklebanov commented Jun 19, 2024

Found this issue upstream, they're using mssql but similar problem where app crashes when db unreachable.

kysely-org/kysely#836

Not the same.

kysely core uses tedious, not mssql, as the underlying driver for MssqlDialect.

Unlike postgres.js, tedious doesn't come with connection pool management, so we're using tarn.js (also used by mssql and knex) for pool management.

The issue was tedious throwing connection errors asynchronously if you don't subscribe to Connection's "error" events, causing uncaught exceptions. Undocumented behavior (not a bug) in tedious and a bug in kysely (not handling errors).

Here, postgres.js has a bug where the logic/state are wrong, causing the code to execute a wrong branch and throw a JavaScript error (query.origin is undefined, why you invoking replace bro?).

@piekczyk
Copy link

For those who have this problem still unresolved - in my case I forgot to destroy db instance after each call, so requests were creating a new clients each time. Once max client threshold was reached it failed with this ugly error:

Cannot read properties of undefined (reading 'replace') ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

4 participants