Your Pipeline Can Report Success While Its Data Falls Behind

Consider an ingestion job whose upstream API adds a new record format. The client rejects records it cannot parse, returns them to a queue and reports success because the requeue operation succeeded. The dashboard stays green while the dataset falls behind.

An unlimited retry policy makes that failure persistent. Repeating the same request cannot teach the parser a new format. A permanent incompatibility needs a distinct error state and a repair path.

A Retry Loop Can Take Down Its Neighbours

If several feeds share one provider quota, retries from the broken feed can consume capacity that healthy feeds need. Per-source budgets and a shared limiter keep the failure from spreading.

Once the parser is fixed, replay the backlog gradually. Compare source counts with accepted counts, check for duplicates and watch the provider’s rate-limit responses. A deployment succeeding tells you little about whether missing records have actually arrived.

Expose gaps to the assistant

An assistant can turn an incomplete dataset into a confident explanation. Ask why activity fell and it may propose a business cause before checking whether collection failed. Retrieval should include coverage and freshness metadata, so the assistant can flag gaps and decline an unsupported comparison.

Collection quality needs its own acceptance criteria. Define which sources must arrive, how late they may be and what reconciliation proves completeness. An application can retrieve the correct document from an incomplete index and still answer the business question incorrectly.

Bound retries and reconcile the result

The retry path needs a limit and a way to recover the missing records.

Every retry loop has a ceiling, and items that hit it go to a dead-letter queue that a person looks at. Include the source, failure category and last attempt so the owner can investigate.

Keep the specific error category and enough redacted detail to diagnose it. “Unable to parse” needs the field, format version and a trace reference, without copying confidential payloads into logs.

Alarms should watch volume and age as well as errors. A sharp drop against a comparable period deserves investigation even when every job reports success. Set queue-age thresholds from the delivery promise and account for planned quiet periods.

Each source gets its own budget for rate-limited providers, so one failing feed cannot starve the others.

Keep a history of business events

Numbers often need a business-event history to explain them. In the Bilibili talk, Zara Zhang describes an analyst whose agent collects business actions from authorised chats, documents and meeting notes. A paused campaign can then be considered alongside a metric change. It is a lead to investigate, rather than proof of causation.

Caching creates another freshness boundary. A figure can have been correct when written and become misleading later if its source changes. Give each reported value a source timestamp and a freshness status, and make stale values visibly stale.

Two Measurements Need an Owner

Add two measurements to a pipeline that feeds reports or assistants: the age of the oldest pending item and the number of records accepted compared with an expected baseline. Choose thresholds with the feed’s owner, test an alert and assign someone to respond. Otherwise the new chart can stay unnoticed beside the old green dashboard.