Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ed41f0a
fix(cli-v3): allow disabling source-map-support to prevent OOM with S…
Feb 2, 2026
023c3fd
fix(cli-v3): ignore engine checks during deployment install to preven…
Feb 2, 2026
93aa053
fix(core): delegate to original console in ConsoleInterceptor to pres…
Feb 2, 2026
8b684e1
fix(cli-v3): authenticate to Docker Hub to prevent rate limits (#2911)
Feb 2, 2026
737ad56
fix(cli-v3): ensure worker cleanup on SIGINT/SIGTERM (#2909)
Feb 2, 2026
c97cbcc
verify: add reproduction scripts and PR details for all major fixes
Feb 3, 2026
aa90db9
verify: add reproduction scripts and PR details for all major fixes
Feb 3, 2026
f5ce2bc
docs: add consolidated PR body description
Feb 3, 2026
8c986db
chore: remove reproduction scripts and temporary files
Feb 3, 2026
82f198f
Merge remote-tracking branch 'remotes/origin/fix/sentry-oom-2920'
Feb 8, 2026
9a3e8d0
Merge branch 'fix/issue-2909-orphaned-workers'
Feb 9, 2026
e101f8e
chore: remove reproduction scripts after verification
Feb 9, 2026
d01d438
fix: resolve typecheck errors after merge
Feb 9, 2026
aafb736
fix(webapp): auto-recover replication services after stream errors
ericallam May 13, 2026
7fa3a16
fix(webapp): reschedule reconnect when subscribe() throws
ericallam May 13, 2026
4e6461a
fix(webapp): reschedule reconnect when subscribe() returns stopped
ericallam May 15, 2026
4b5db51
fix(webapp): drop bogus isStopped check, route leader-lock failure th…
ericallam May 15, 2026
5365936
fix(webapp): scope leaderElection-lost recovery to reconnect strategy
ericallam May 15, 2026
d35bf04
Merge pull request #10 from deepshekhardas/pr/3613-replication-fix
deepshekhardas May 20, 2026
5f4c41a
feat: migrate Prisma from 6.14.0 to 7.7.0 with driver adapters (PR #3…
Jun 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/fix-console-interceptor-2900.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---

Fix: ConsoleInterceptor now delegates to original console methods to preserve log chain when other interceptors (like Sentry) are present. (#2900)
5 changes: 5 additions & 0 deletions .changeset/fix-docker-hub-rate-limit-2911.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix: Native build server failed with Docker Hub rate limits. Added support for checking checking `DOCKER_USERNAME` and `DOCKER_PASSWORD` in environment variables and logging into Docker Hub before building. (#2911)
5 changes: 5 additions & 0 deletions .changeset/fix-github-install-node-version-2913.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix: Ignore engine checks during deployment install phase to prevent failure on build server when Node version mismatch exists. (#2913)
5 changes: 5 additions & 0 deletions .changeset/fix-orphaned-workers-2909.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix: `trigger.dev dev` command left orphaned worker processes when exited via Ctrl+C (SIGINT). Added signal handlers to ensure proper cleanup of child processes and lockfiles. (#2909)
5 changes: 5 additions & 0 deletions .changeset/fix-sentry-oom-2920.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix Sentry OOM: Allow disabling `source-map-support` via `TRIGGER_SOURCE_MAPS=false`. Also supports `node` for native source maps. (#2920)
6 changes: 6 additions & 0 deletions .server-changes/replication-error-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
area: webapp
type: fix
---

Runs and sessions replication services now auto-recover from stream errors (e.g. after a Postgres failover) instead of silently leaving replication stopped. Behaviour is configurable per service — reconnect (default), exit so a process supervisor can restart the host, or log.
72 changes: 37 additions & 35 deletions apps/webapp/app/db.server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import {
type PrismaTransactionClient,
type PrismaTransactionOptions,
} from "@trigger.dev/database";
import { PrismaPg } from "@prisma/adapter-pg";
import { createHash } from "node:crypto";
import invariant from "tiny-invariant";
import { z } from "zod";
import { env } from "./env.server";
Expand Down Expand Up @@ -127,21 +129,30 @@ function getClient() {
const { DATABASE_URL } = process.env;
invariant(typeof DATABASE_URL === "string", "DATABASE_URL env var not set");

const databaseUrl = extendQueryParams(DATABASE_URL, {
connection_limit: env.DATABASE_CONNECTION_LIMIT.toString(),
pool_timeout: env.DATABASE_POOL_TIMEOUT.toString(),
connection_timeout: env.DATABASE_CONNECTION_TIMEOUT.toString(),
application_name: env.SERVICE_NAME,
});
const databaseUrl = new URL(DATABASE_URL);

// Set application_name as a query param on the connection string (pg understands this)
databaseUrl.searchParams.set("application_name", env.SERVICE_NAME);

console.log(`🔌 setting up prisma client to ${redactUrlSecrets(databaseUrl)}`);

const client = new PrismaClient({
datasources: {
db: {
url: databaseUrl.href,
},
const adapter = new PrismaPg(
{
connectionString: databaseUrl.href,
max: env.DATABASE_CONNECTION_LIMIT,
idleTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
connectionTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
},
{
// Generate deterministic prepared statement names from query SQL so PostgreSQL
// can reuse cached query plans. Without this, every query uses an anonymous
// prepared statement that PG must parse and plan from scratch each time.
statementNameGenerator: (query) => `p_${createHash("sha256").update(query.sql).digest("hex").slice(0, 16)}`,
}
);
Comment on lines +139 to +152

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Database pool wait timeout setting is silently ignored after the connection change

The configured pool wait limit is no longer applied anywhere (its use was removed when building the connection settings at apps/webapp/app/db.server.ts:139-152), and the connect timeout value is additionally reused as the idle-connection timeout, so two documented settings now behave differently than configured.
Impact: Operators who tuned the pool wait or connection timeouts will get different behaviour than their configuration says, including connections being closed while idle sooner than before.

Mapping details

Old code appended connection_limit, pool_timeout, connection_timeout query params to the connection string. The new adapter config only sets max, idleTimeoutMillis and connectionTimeoutMillis, all derived from DATABASE_CONNECTION_LIMIT / DATABASE_CONNECTION_TIMEOUT.

  • env.DATABASE_POOL_TIMEOUT (default 60s, apps/webapp/app/env.server.ts:103) is now unused; pool-acquire waits are governed by connectionTimeoutMillis (20s default) instead.
  • idleTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000 reuses a connect timeout as an idle timeout — semantically unrelated knobs, so raising the connect timeout now also keeps idle connections open longer (and vice versa).

The same pattern is duplicated for the read replica at apps/webapp/app/db.server.ts:270-280.

Prompt for agents
In apps/webapp/app/db.server.ts the migration to PrismaPg dropped the `pool_timeout` setting (env.DATABASE_POOL_TIMEOUT) entirely and mapped DATABASE_CONNECTION_TIMEOUT onto both connectionTimeoutMillis and idleTimeoutMillis. Decide on an explicit mapping: pool acquire/wait should come from DATABASE_POOL_TIMEOUT, connect timeout from DATABASE_CONNECTION_TIMEOUT, and idle timeout should be its own setting (or pg's default) rather than reusing the connect timeout. Apply the same mapping to both the writer and replica adapters, and remove/deprecate DATABASE_POOL_TIMEOUT in env.server.ts if it is intentionally no longer supported.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +146 to +152

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Deterministic prepared-statement names rely on SQL text alone

statementNameGenerator: (query) => \p_${sha256(query.sql).slice(0,16)}`derives the prepared-statement name only from the SQL text. node-postgres caches named statements per connection and reuses them when the name matches, so identical SQL is fine. The risk cases are (a) a 64-bit hash collision between two different statements on the same connection (negligible but non-zero), and (b) two logically distinct queries whose SQL text is identical but whose parameter types differ — Postgres would reject the second Parse with "prepared statement already exists" / type mismatch. Worth confirming against@prisma/adapter-pg@7.7.0` semantics (does the adapter guarantee the generator only needs uniqueness per distinct SQL+param-type tuple?) before running this in production, since a failure here surfaces as sporadic query errors rather than a startup failure.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


const client = new PrismaClient({
adapter,
log: [
// events
{
Expand Down Expand Up @@ -251,21 +262,25 @@ function getReplicaClient() {
return;
}

const replicaUrl = extendQueryParams(env.DATABASE_READ_REPLICA_URL, {
connection_limit: env.DATABASE_CONNECTION_LIMIT.toString(),
pool_timeout: env.DATABASE_POOL_TIMEOUT.toString(),
connection_timeout: env.DATABASE_CONNECTION_TIMEOUT.toString(),
application_name: env.SERVICE_NAME,
});
const replicaUrl = new URL(env.DATABASE_READ_REPLICA_URL);
replicaUrl.searchParams.set("application_name", env.SERVICE_NAME);

console.log(`🔌 setting up read replica connection to ${redactUrlSecrets(replicaUrl)}`);

const replicaClient = new PrismaClient({
datasources: {
db: {
url: replicaUrl.href,
},
const adapter = new PrismaPg(
{
connectionString: replicaUrl.href,
max: env.DATABASE_CONNECTION_LIMIT,
idleTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
connectionTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
},
{
statementNameGenerator: (query) => `p_${createHash("sha256").update(query.sql).digest("hex").slice(0, 16)}`,
}
);

const replicaClient = new PrismaClient({
adapter,
log: [
// events
{
Expand Down Expand Up @@ -368,19 +383,6 @@ function getReplicaClient() {
return replicaClient;
}

function extendQueryParams(hrefOrUrl: string | URL, queryParams: Record<string, string>) {
const url = new URL(hrefOrUrl);
const query = url.searchParams;

for (const [key, val] of Object.entries(queryParams)) {
query.set(key, val);
}

url.search = query.toString();

return url;
}

function redactUrlSecrets(hrefOrUrl: string | URL) {
const url = new URL(hrefOrUrl);
url.password = "";
Expand Down
23 changes: 23 additions & 0 deletions apps/webapp/app/env.server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1330,6 +1330,16 @@ const EnvironmentSchema = z
RUN_REPLICATION_INSERT_STRATEGY: z.enum(["insert", "insert_async"]).default("insert"),
RUN_REPLICATION_DISABLE_PAYLOAD_INSERT: z.string().default("0"),
RUN_REPLICATION_DISABLE_ERROR_FINGERPRINTING: z.string().default("0"),
// What to do when the runs replication client errors (e.g. after a
// Postgres failover). `reconnect` (default) re-subscribes in-process with
// exponential backoff; `exit` exits the process so a supervisor restarts
// it; `log` preserves the old no-op behaviour. Reconnect tuning is
// shared across both replication services via REPLICATION_RECONNECT_*.
RUN_REPLICATION_ERROR_STRATEGY: z
.enum(["reconnect", "exit", "log"])
.default("reconnect"),
RUN_REPLICATION_EXIT_DELAY_MS: z.coerce.number().int().min(0).default(5_000),
RUN_REPLICATION_EXIT_CODE: z.coerce.number().int().min(0).max(255).default(1),

// Session replication (Postgres → ClickHouse sessions_v1). Shares Redis
// with the runs replicator for leader locking but has its own slot and
Expand Down Expand Up @@ -1362,6 +1372,19 @@ const EnvironmentSchema = z
SESSION_REPLICATION_INSERT_MAX_RETRIES: z.coerce.number().int().default(3),
SESSION_REPLICATION_INSERT_BASE_DELAY_MS: z.coerce.number().int().default(100),
SESSION_REPLICATION_INSERT_MAX_DELAY_MS: z.coerce.number().int().default(2000),
// Error recovery — same semantics as RUN_REPLICATION_ERROR_STRATEGY.
SESSION_REPLICATION_ERROR_STRATEGY: z
.enum(["reconnect", "exit", "log"])
.default("reconnect"),
SESSION_REPLICATION_EXIT_DELAY_MS: z.coerce.number().int().min(0).default(5_000),
SESSION_REPLICATION_EXIT_CODE: z.coerce.number().int().min(0).max(255).default(1),

// Reconnect tuning shared across both replication services. Only
// applies when error strategy is `reconnect`. Max attempts of 0 means
// unlimited (default).
REPLICATION_RECONNECT_INITIAL_DELAY_MS: z.coerce.number().int().min(0).default(1_000),
REPLICATION_RECONNECT_MAX_DELAY_MS: z.coerce.number().int().min(0).default(60_000),
REPLICATION_RECONNECT_MAX_ATTEMPTS: z.coerce.number().int().min(0).default(0),

// Clickhouse
CLICKHOUSE_URL: z.string(),
Expand Down
8 changes: 1 addition & 7 deletions apps/webapp/app/routes/metrics.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { LoaderFunctionArgs } from "@remix-run/server-runtime";
import { prisma } from "~/db.server";
import { metricsRegister } from "~/metrics.server";

export async function loader({ request }: LoaderFunctionArgs) {
Expand All @@ -13,14 +12,9 @@ export async function loader({ request }: LoaderFunctionArgs) {
}
}

// We need to remove empty lines from the prisma metrics, grafana doesn't like them
const prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
const coreMetrics = await metricsRegister.metrics();

// Order matters, core metrics end with `# EOF`, prisma metrics don't
const metrics = prismaMetrics + coreMetrics;

return new Response(metrics, {
return new Response(coreMetrics, {
headers: {
"Content-Type": metricsRegister.contentType,
},
Comment on lines 15 to 20

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Prisma pool/metric observability removed with no replacement

Dropping the metrics preview feature removes $metrics.json() / $metrics.prometheus(), which is why configurePrismaMetrics and the /metrics route composition were deleted. The consequence is that all connection-pool visibility (db.pool.connections.*, db.client.queries.*) disappears from both the OTel meter and the Prometheus scrape endpoint, right as the pool implementation changes from the Rust engine to node-postgres. Since pool sizing/idle behaviour is also being changed in this PR, it would be worth re-adding equivalent gauges from the pg.Pool instance (totalCount, idleCount, waitingCount) so regressions are observable.

(Refers to lines 15-21)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Expand Down
Loading