Why No AI Result Gets Stuck Half-Finished

A model call succeeds. The result lands in one table. The status row in another table does not update, because the database connection was briefly saturated at exactly the wrong moment. The record is now "in flight" by status, "complete" by result, and "charged" by billing — three systems holding three different beliefs about the same study, none of them wrong on its own terms, and no component whose job it is to notice.

That is the shape of the reliability problem in AI inference, and it is why it does not look like a model problem. HeartSciences ran inference across multiple AI model providers for every incoming study on their AI-ECG platform, and the pipeline behaved under normal conditions. Under abnormal ones — transient network error, provider timeout, partial response — it did not fail cleanly. ML LABS rebuilt the inference layer of that platform's cloud backend around a single design property that replaces every reliability adjective anyone might want to use: every record reaches a definite terminal state, and that state is queryable.

That guarantee is worth more than a metric, because it is a proof rather than a measurement. A study is never "probably fine" or "still processing since Tuesday". It is in a state a query can name — and reconciliation, which used to require a person, becomes something a query does.

The State Contradiction

The cascading failures traced back to one architectural gap: the system had no recovery semantics for model inference. The chain runs like this.

• A model returns a valid result, but the record's status update fails
• The record looks stuck even though the work is done
• A retry re-invokes the model, generating a second charge
• Two conflicting results now exist for the same input, with no authority between them

The Cascade

Take a schematic trace — illustrative, not a client incident. A record enters processing and the first provider returns a result in seconds. The orchestrator writes the result but the status row fails to update. The record is in flight by status, complete by result, charged by billing. The scheduled retry job sees a stale "Processing" status, concludes the record never completed, and dispatches a second call to the same provider. A new charge fires. A second result lands beside the first. Days later a finance reconciliation flags the duplicate, an operator opens the record, and the two results disagree on a borderline classification — and the operator now owns a clinical decision that was never theirs to make.

Every step in that trace is individually reasonable. That is what makes the class hard: no component is buggy, and the system is still wrong. Hidden technical debt in machine learning systems (NeurIPS, 2015) named this a decade ago — the model is a small box inside a large system, and the failures live in the plumbing around it. A survey of case studies on deploying machine learning (ACM Computing Surveys, 2022) reaches the same place from the other end: the recurring deployment failures are integration and lifecycle failures, not modeling failures.

Three Diagnostic Signals

• Records sitting in non-terminal status beyond the longest expected inference time. If the slowest model call completes well inside a minute, a record still "Processing" many minutes later is a state failure, not a slow call. Count them by hour and you have a leading indicator that the recovery path is broken.
• A non-zero duplicate-result rate per input. Two stored results for one record — even when they agree — mean the system retried after a call that had already succeeded. Any duplicate rate above zero is a proxy for the underlying race.
• Retry-to-success ratio drifting above one. When successful retry attempts approach or exceed the count of records that actually needed retrying, the pipeline is paying for work it has already done. This is the cleanest single signal that billing has not been decoupled from processing.

A pipeline showing all three does not have a model quality problem. It has an integration problem in the gap between the model call and the durable state update, and no amount of provider tuning will close it.

Decouple Billing From Processing

The core fix decoupled billing from processing. Every model call used to generate a charge whether the result was ultimately used or discarded. ML LABS redesigned billing to be tracked at the record level, so the system knows whether a record has already been charged before any model call executes, and a retry that detects an existing charge skips the billing event entirely. Each inference request also carries an idempotency key, so a retry after a provider timeout cannot produce a second billable result for the same study — the same discipline that the usage-based billing system depends on downstream, where a charge that should not exist becomes an invoice a customer disputes.

graph TD
    A1["Record enters<br/>processing"]
    B1["Check billing<br/>status"]
    C1{"Already<br/>charged?"}
    D1["Return existing<br/>result"]
    E1["Process and<br/>record charge"]
    F1["Write result<br/>atomically"]

    A1 --> B1
    B1 --> C1
    C1 -->|"Yes"| D1
    C1 -->|"No"| E1
    E1 --> F1

    style A1 fill:#1a1a2e,stroke:#0f3460,color:#fff
    style B1 fill:#1a1a2e,stroke:#ffd700,color:#fff
    style C1 fill:#1a1a2e,stroke:#ffd700,color:#fff
    style D1 fill:#1a1a2e,stroke:#16c79a,color:#fff
    style E1 fill:#1a1a2e,stroke:#0f3460,color:#fff
    style F1 fill:#1a1a2e,stroke:#16c79a,color:#fff

Retry behavior was unified across providers at the same time. Each provider had been left to its own retry assumptions, which meant the pipeline's failure semantics were the union of several vendors' opinions about what a timeout means. One shared processing layer now holds the error handling, with safeguards against concurrent retries racing each other into duplicate states — and because it is one layer, adding a provider does not add a new failure dialect.

The most expensive assumption in model inference is that a failed status update means the work was not done. The model may well have completed — and retrying on that assumption creates a second charge and a state contradiction that only a person can unwind.

Terminal States Are The Guarantee

The synchronous architecture was the root fragility: a slow model call blocked the processing thread, and a mid-chain failure left records in ambiguous states with no recovery path. ML LABS replaced it with an async architecture that gives durability (work survives process crashes), isolation (a slow provider does not block new work), and observability (processing health is directly measurable rather than inferred).

On top of that sits the guarantee. Records that exhaust retries route to investigation rather than disappearing. Every record reaches a terminal status whether processing succeeded, failed, or was never possible — and every terminal state is queryable, so the question "what is the state of this study, and why" is answered by a query rather than by reading logs. That is a design property, not a performance claim, and it is the one an engineer should want: it holds by construction rather than by measurement.

Two supporting decisions make it stick. Failure simulation is built into the platform itself, so every failure mode can be triggered on demand in any environment through the real pipeline rather than through mocks — which is how the ambiguous case (model succeeds, status update fails, record appears stuck) became a test that runs in CI on every commit instead of a bug that only production can find. And every ingestion path — web upload, clinic file share, EHR integration — flows through identical processing logic, so a reliability guarantee proven on one path is not quietly absent on another. The regulatory frame demands exactly this posture: the guiding principles for good machine learning practice in medical devices (FDA, 2021) treat deployment monitoring as a design input, not as an operational afterthought.

When This Is Overkill

This level of reliability engineering adds real infrastructure complexity, and there are systems that should not pay for it. Internal experimentation pipelines where reprocessing is free. Stateless, cheap model calls with no billing consequence. Batch jobs that tolerate duplicates by design. In those cases the terminal-state machinery is ceremony, and the honest recommendation is to skip it.

The investment earns out when model calls cost money, when results feed stateful clinical or financial workflows, and when a duplicate has a consequence someone will eventually have to explain. The test is not how sophisticated the model is. It is whether a wrong state can turn into a wrong bill, a wrong record, or a wrong decision — and in a clinical system, all three are reachable from the same missed status update. What makes it worth defending is that the same strategies used to shed load and contain overload (AWS Builders Library, 2024) in any distributed system apply here — the difference is that the unit of work is a patient's study.

First Steps

1. Instrument the three signals before changing anything. Non-terminal records past the longest expected inference time, duplicate results per input, and retry-to-success ratio. If you cannot produce those numbers today, that is the finding.
2. Decouple billing from processing, highest-cost model first. The system must check charge status before any model call executes, not after it returns.
3. Simulate the ambiguous failure. Force a model success with a failed status update and watch what the system does. If the answer requires a person, the recovery path does not exist yet.

Make Every End State Queryable

The bar for a production inference pipeline is not "it retries". It is that every record has a definite end state, that the end state is queryable, and that no retry can ever produce a second charge or a second result. If an operator has to open a record to find out what happened to it, the pipeline is not reliable — it is being manually reconciled by someone whose job that is not, and that person is the recovery mechanism.

That bar is also the difference between a system that has an owner and one that merely has a builder. A production AI system without an owner degrades: retries drift, provider behavior changes underneath the code, and the failures that matter are the ones nobody is watching for. The AI Risk Management Framework (NIST, 2023) puts that plainly by making manage a continuous function rather than a launch gate — risk that is measured once is risk that is only known once. Holding that bar over time is the job managed AI operations exists to do — an accountable owner on a defined scope of live systems, watching the signals above and the ones the system has not needed yet. Reliability in AI inference was never a model property. It is a property of the state machine wrapped around the model, and it is built, owned, and defended there.

References

1. Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
2. Paleyes, A., Urma, R.-G., & Lawrence, N. D. Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Computing Surveys, 2022.
3. U.S. Food and Drug Administration, Health Canada, and MHRA. Good Machine Learning Practice for Medical Device Development: Guiding Principles. Regulatory Reference, 2021.
4. Amazon Web Services. Using Load Shedding to Avoid Overload. AWS Builders Library, 2024.
5. National Institute of Standards and Technology. AI Risk Management Framework (AI RMF 1.0). NIST, 2023.