The Cron That Ran Twice: How One Scaling Decision Duplicated a Month of Invoices

Project Overview

A routine capacity change turned every nightly batch job into two. It took thirteen hours to show up, and a timestamp to prove.

What the client saw

Customer accounts showing the same payment twice. Single lessons carrying two identical invoices, seconds apart, sharing an invoice number.

The reports arrived clustered around one date, and the first hypothesis was reasonable: something about that day's rescheduled lessons was triggering a billing edge case. The team went looking for the bug in the invoicing logic.

It wasn't there. The invoicing logic was correct. It had simply been executed twice.

The setup that made it possible

The legacy platform ran its scheduled work the way most long-lived PHP applications do — a console application invoked by system cron, on one machine. Nightly, at fixed times, single-threaded by construction:

flowchart LR
    accTitle: Legacy single-runner cron flow
    accDescr: System cron starts one console process, which writes scheduled work to the database.
    A[System cron] --> B[Console process]
    B --> C[(Database)]

Concurrency was never a design concern, because concurrency was structurally impossible. One host, one process, one run.

The modernized system moved those jobs into the application itself, registered as in-process scheduled tasks — the idiomatic pattern in a modern Node framework, and convenient: the jobs live with the code they call, share dependency injection, and deploy as one unit.

Then the API was scaled to two replicas for availability. Standard, sensible infrastructure practice.

flowchart TD
    accTitle: Replicated scheduler duplication flow
    accDescr: Two API replicas run the same scheduled job against one shared database, producing duplicate invoices and payments.
    subgraph Cluster["Production cluster"]
        R1[API replica 1<br/>scheduler active]
        R2[API replica 2<br/>scheduler active]
    end
    R1 -->|23:00 job fires| DB[(Shared database)]
    R2 -->|23:00 job fires| DB
    DB --> X[Two invoices per lesson<br/>Two payments per cycle]

    style X fill:#5a1e1e,color:#fff

Both replicas ran the full application. Both, therefore, ran the scheduler. Every nightly job now fired twice, simultaneously, against one shared database.

Why the existing guards didn't help

The jobs weren't naive. Each one checked before it wrote — does a payment already exist for this cycle? does this lesson already have an invoice?

That check is sufficient under sequential execution. It is worthless under concurrent execution, because it is a read-then-write with no lock between the two steps:

sequenceDiagram
    accTitle: Concurrent invoice creation race
    accDescr: Two replicas check for an invoice before either writes, then both insert an invoice for the same lesson.
    participant A as Replica 1
    participant DB as Database
    participant B as Replica 2

    A->>DB: Invoice exists for this lesson?
    B->>DB: Invoice exists for this lesson?
    DB-->>A: No
    DB-->>B: No
    A->>DB: INSERT invoice
    B->>DB: INSERT invoice
    Note over DB: Two invoices, same lesson

Both replicas asked the question before either wrote the answer. Both got "no." Both proceeded.

No database constraint caught it, because none existed — under the old single-runner model, none had ever been needed. The invoice numbering had the same flaw: read the last number, add one. Two readers, one number, twice.

Notably, the legacy code carried the same check-then-insert pattern. It had simply never been exposed, because its runtime made concurrency impossible. The vulnerability was inherited; only the execution model changed.

Proving it

The decisive evidence was chronological. The team lined up the deployment timestamp against the first duplicated record of each job type, and the pattern was unambiguous: every job's first duplicate appeared at that job's first scheduled run after the replica change — roughly thirteen hours later for the earliest one.

A broadened scan confirmed the scope was far larger than the handful of reported accounts, and spread across every location the jobs touched. The reported cases were the ones a staff member happened to notice.

The remediation

Immediate: reduce to a single replica, trading availability for correctness until the underlying issue is closed. Uncomfortable, and correct.

Structural, in layers:

• Job-level locking so only one instance of a scheduled job can run at a time — a durable lock table, a database advisory lock, or an external scheduler configured as a single runner.
• Database constraints making duplicates impossible rather than merely unlikely. A read check is an optimization; a uniqueness guarantee belongs in the schema.
• Transactional idempotency — lock the target row, re-check inside the transaction, then write.
• Guard the manual triggers. Admin endpoints that ran these jobs on demand could collide with the scheduler in exactly the same way.

Then cleanup: identify duplicated records, verify each set, and reverse the extras without disturbing the legitimate ones.

The transferable lesson

Legacy systems are full of assumptions that were true for so long that nobody wrote them down. "Only one copy of this runs at a time" was true here for a decade — enforced not by any code, but by the deployment topology.

Modernization changes deployment topology. That is often the point: horizontal scaling, higher availability, container orchestration. But every assumption the old topology silently guaranteed now needs an explicit owner.

When migrating scheduled or batch work, the question worth asking early is: what happens if this runs twice, at the same time, right now? If the honest answer is "it depends on there only ever being one of us," that is a design gap — not a hypothetical — and horizontal scaling will find it.

Ideally before the invoices go out.

Planning a move from single-host legacy infrastructure to a scaled deployment? We help teams find the assumptions that don't survive the transition. Get in touch.