Your software factory.
It never clocks out
Bugs, forum posts, Slack, alerts or a prompt start work; it lands on your board as a tested change, ready to ship.
Set it up once. It keeps shipping
What a normal day looks like, three months after setup.
Gateway p99 back to normal after the 08:00 spike.
Rate limiter is still the old in-memory one...
1 reply Last reply 9:16 AMRate limiter is still the old in-memory one, so it resets on every deploy and customers on the Scale plan keep hitting 429s at 08:00 UTC. @ForgeCode rewrite it on Redis, keep the public headers, per-tenant overrides from the plans table, and load test it before you open a PR.
On it. Opened mission LRK-410 on larkspur-cloud-1 with 5 issues. I will report back in this thread when a PR is ready.
| Issue | Status | Priority | Runs | Updated |
|---|---|---|---|---|
| LRK-412Token bucket store on Redis with per-key TTL | In Progress | High | Running | 09:16 |
| LRK-413Middleware: 429 with Retry-After and RateLimit-* headers | In Progress | High | Running | 09:16 |
| LRK-414Per-tenant overrides read from the plans table | In Progress | Medium | Running | 09:16 |
| LRK-415Load test: 12k rps burst, p99 under 40 ms | To Do | Medium | Blocked by 412, 413 | 09:16 |
| LRK-416Docs: rate limits page and SDK retry examples | To Do | Low | Idle | 09:16 |
| LRK-412Token bucket store | Done |
| LRK-413Middleware: 429 + headers | In Progress |
| LRK-414Per-tenant overrides | In Progress |
| LRK-415Load test 12k rps | To Do |
| LRK-416Docs and SDK retries | To Do |
Gateway p99 back to normal after the 08:00 spike.
Rate limiter is still the old in-memory one...
2 replies Last reply 6:32 PMRate limiter is still the old in-memory one... rewrite it on Redis, keep the public headers, per-tenant overrides, load test before you open a PR.
PR #1187 is ready for review. 4 of 5 issues done; LRK-416 (docs) is open in review for you.
- Redis token bucket with per-key TTL, 0 allocations on the hot path
- 429 responses carry Retry-After (integer seconds) and RateLimit-Limit / Remaining / Reset
- Tenant overrides read from
plans.rate_limit_rps, cached 30s - Load test: 12,400 rps burst for 5 min, p99 31 ms, 0 dropped
| Title | Trigger | Last run | Next run |
|---|---|---|---|
| Production alert: open incident, find root cause, open PR | alert sentry prod | 02:10 | on alert |
| Nudge stalled issues: rerun from checkpoint after 6h idle | watch: issue idle 6h | 03:35 | on stall |
| Remind reviewer when a PR waits more than 8h | watch: PR waiting 8h | 07:00 | on wait |
| GitHub bug report: triage, reproduce, open issue | github issues | Mon 16:48 | on event |
| Forum post tagged bug: reproduce and reply | forum discourse | Mon 11:02 | on post |
| Slack message in #api-platform | slack @ForgeCode | 09:14 | on message |
| Nightly flaky test sweep, escalate if above 1% | schedule 02:00 | 02:00 | Thu 02:00 |
NullPointer in billing/invoice.rs:212, 38 events in 4 min. LRK-419 opened.02:33 RCA: refunded invoices have no customer since PR #1174 dropped the join. PR #1188 opened; merge policy
merge when gates pass for hotfix/*: merged 02:41.02:44 posted for Dev Okonkwo (owns billing): what changed, why, and how to verify.
| Issue | Status | Priority | Runs | Updated |
|---|---|---|---|---|
| LRK-412Token bucket store on Redis with per-key TTL | Done | High | Idle | 11:02 |
| LRK-413Middleware: 429 with Retry-After and RateLimit-* headers | Done | High | Idle | 12:48 |
| LRK-414Per-tenant overrides read from the plans table | Done | Medium | Idle | 15:20 |
| LRK-415Load test: 12k rps burst, p99 under 40 ms | Done | Medium | Idle | 18:19 |
| LRK-416Docs: rate limits page and SDK retry examples | In Review | Low | Idle | 18:31 |
| LRK-419Incident: NullPointer in billing/invoice.rs (alert). PR #1188 merged by factory | Done | High | Idle | 02:41 |
- Tests: 212 passed
- Clippy: 0 warnings
- Load test: p99 31 ms
- Security scan: 0 findings
- Staging smoke: 18 of 18
- Reviewer agent: 2 fixed
Done Public API rate limiter rebuilt on Redis. Load tested at 12,400 rps, p99 31 ms.
Fixed Billing NullPointer from PR #1174. Hotfix merged under policy in 31 min.
Decide PR #1187 needs Priya's review to ship the rate limiter to Scale customers.
Better Flaky tests down to 0.6% from 1.4%. Review nudges now fire after 4h.
Next Thursday 02:00: Postgres 16 upgrade mission starts, merge policy ask me.
A coding harness makes one developer faster.
A factory raises organisational velocity
Your factory, your terms
Run factories on ForgeCode's cloud or bring your own cloud (BYOC), and bring your own model keys.
What teams run on a factory
Whole engineering roles and long-running programs, not single tasks.