Add cluster support to multiply outbound write throughput

[stilliard]

Summary

Under load, the final checks.create call lags by minutes while the preceding GET lookups stay fast. Octokit's throttling plugin serializes writes at maxConcurrent: 1, minTime: 1000 (~1 write/sec), keyed per installation and queued in-memory per process. A busy installation backs up behind that gate. (Confirmed via logs + an LB experiment: pausing one node drained the backlog; no GitHub secondary-rate 403s involved.)

Key Changes

Add a small cluster.js entrypoint. The primary forks WEB_CONCURRENCY workers (default 4), each running the existing Probot app unchanged. Each worker gets its own in-memory write queue, so aggregate write throughput scales with worker count. The queue is timer-gated (idle-waiting), so this helps even on single-core droplets — it's process count, not core count, that matters here.

• start → node cluster.js; start:single keeps the prior single-process path.
• dev / dev-debug pinned to start:single so local smee dev doesn't fan out events across workers.
• Crashed workers respawn; a burst of fast (startup) crashes makes the primary exit so the service manager can restart with backoff.
• SIGTERM/SIGINT shuts workers down cleanly without respawn.
• Worker mirrors probot run ./index.js exactly, so the default app's / and /probot routes are preserved.

Redis is intentionally left off — wiring redisConfig would share one queue fleet-wide and re-impose 1 write/sec globally per installation (worse for throughput).

Testing

1. Ensure WEBHOOK_PROXY_URL is unset in production (smee would cause fan-out there too).
2. Set WEB_CONCURRENCY if you want something other than 4 (start conservative).
3. Confirm the systemd unit has Restart=on-failure + RestartSec so the fast-crash exit self-heals with backoff.

After deploy on one droplet:

• ps -ef | grep cluster.js → 1 primary + N workers.
• No duplicate processing: journalctl -u <svc> | grep "Request received" | grep -oE "Context: [a-f0-9-]+" | sort | uniq -c → all counts 1.
• Watch the "Complete and sending back" → "Check response status 201" gap shrink vs. the unchanged droplet; watch for any secondary rate limit 403s (expected: none).

🤖 Generated with Claude Code

[Copilot]

⚠️ Not ready to approve

The worker startup uses probot.run() with an incorrect argv shape and WEB_CONCURRENCY parsing can result in starting zero workers, both of which can break production startup/processing.

Pull request overview

Adds a Node.js cluster-based entrypoint so the Probot app can run as multiple worker processes, increasing aggregate GitHub write throughput by giving each worker its own in-memory Octokit write queue.

Changes:

• Switch npm start to launch a new cluster.js primary process (with start:single preserved for the old single-process behavior).
• Add cluster.js to fork WEB_CONCURRENCY workers and respawn workers on crash (with “fast-crash” protection).
• Pin local dev scripts to the single-process path to avoid webhook fan-out during development.

File summaries

File	Description
package.json	Updates scripts to use clustered start by default while keeping a single-process option for dev.
cluster.js	New clustered entrypoint that forks workers and runs the existing Probot app in each worker.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 2

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

⚠️ Not ready to approve

The worker bootstrap calls require('probot').run() with a CLI-style argv array, which is incompatible with Probot v12’s run() API and can prevent workers from starting correctly.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 1

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.