Integration Bugs Live Between Systems

The order service accepts an order correctly. The inventory service updates stock correctly. The integration still sells an unavailable item because the two services disagree about when availability becomes authoritative. Each component can pass its own tests while the workflow fails between them.

Integration work is therefore about agreements across boundaries: identity, timing, meaning, ownership and recovery. A successful connection proves that systems can exchange messages. It does not prove that the combined process does what the business expects.

Two correct systems can still create one incorrect workflow.

Contracts include meaning and time

An API schema can state that quantity is an integer. It may leave unresolved whether the value means physical stock, sellable stock or available stock after reservations. A consumer can parse the response perfectly and still make the wrong decision.

Document the semantics that affect behavior. Identify the authority for each fact, when it becomes effective and what a missing value means. If a field represents a snapshot, expose enough information to distinguish an old snapshot from a current one.

Time matters because systems observe changes at different moments. A price can be valid from midnight in a business timezone while an integration records a UTC receipt timestamp. Those are different facts. Replacing one with the other can move a change across a day boundary.

Use timezone-aware instants for events that occur at a moment, and explicit business dates where the domain means a calendar day. Avoid pretending that adding midnight to a date resolves every ambiguity. The contract needs to explain which interpretation the workflow uses.

A timeout is an uncertain outcome

When a request times out, the caller knows it did not receive a timely response. It may not know whether the receiver completed the operation. Retrying without considering that uncertainty can create duplicate orders, repeated notifications or conflicting updates.

HTTP Semantics defines idempotent methods and discusses retry considerations. Those protocol semantics are useful, but an application still needs to understand the actual endpoint and the business operation it performs. A method name alone cannot repair an incorrectly implemented side effect.

For a consequential operation, define how the caller identifies the intended action across attempts. A stable operation identifier can help the receiver recognize repetition when the contract supports it. The identifier must represent the same intent, rather than being regenerated for every retry.

If the outcome remains uncertain, provide reconciliation. The caller may need to query by operation ID or inspect the destination before issuing another change. Reporting “failed” without distinguishing “not applied” from “unknown” can lead an operator to repeat work that already happened.

Duplicate delivery is an ordinary case

A message can be delivered again after a retry, a process restart or a lost acknowledgment. Consumers should have a defined response to repetition. That response may be to ignore an already applied event, replace a record with the same desired state or flag an inconsistent duplicate for investigation.

The choice depends on the operation. Setting a product description to a specified value differs from incrementing stock by a quantity. Repeating the first may converge on the same state; repeating the second can accumulate an incorrect result.

Keep the deduplication boundary aligned with the business action. A transport message ID may change while the intended order remains the same. Conversely, two legitimate actions may contain identical payloads. Deduplicating only by content can incorrectly discard real work.

Tests should include repeated delivery and restarts around the acknowledgment boundary. A happy-path test that sends one message once does not exercise the behavior most likely to become confusing during recovery.

Partial updates need a visible state

A workflow may update several systems without a transaction that covers all of them. The catalogue update succeeds, the marketplace update fails and the reporting export has not started. Calling the entire operation simply “failed” hides the state needed to recover.

Represent progress at the level required for repair. Record which steps are confirmed, which are pending and which have uncertain outcomes. The system can then resume or compensate deliberately instead of repeating everything blindly.

Recovery starts with knowing which effects already happened.

Compensation is not always a perfect inverse. Cancelling a reservation may release stock, but it does not erase a notification already sent to a customer. A refund does not make a payment disappear from history. Design recovery around the domain’s real operations rather than assuming every action has a clean undo.

For some workflows, a staged publication model is simpler: prepare and validate a complete artifact, then make it authoritative through a narrow final step. This reduces the number of partial states exposed to consumers, though it may not be available in every external system.

Make order assumptions explicit

Messages may arrive later than expected or in a different order. If an older product update overwrites a newer one, every individual request can succeed while the final state regresses. The integration needs a policy for versioning and conflict resolution.

A source revision can help when the producer supplies a trustworthy ordering within the relevant scope. A wall-clock timestamp may be insufficient if clocks differ or if several updates legitimately share a timestamp. Do not introduce a total order where the domain only provides a partial one without considering the consequences.

Specify the scope of an ordering guarantee. Events for one product may need to be applied in sequence while independent products can proceed concurrently. Requiring global order can create unnecessary coordination; ignoring per-product order can corrupt meaning.

The complexity essay explains why these agreements matter architecturally. Every ordering requirement creates coordination work. Make the necessary ones explicit and avoid inventing extra ones merely because the implementation uses one shared queue.

Observe the workflow across boundaries

Logs from one service rarely explain the entire integration. Use a correlation identifier that can connect relevant operations, while keeping business identifiers and sensitive data appropriately protected. A useful trace shows where work entered, which dependency was called and what outcome was confirmed.

Record failure categories that support action. Authentication failure, contract rejection, timeout and semantic validation failure require different responses. A generic exception count can alert the team but provides little guidance about recovery.

Measure the end-to-end outcome as well as component availability. A service can be healthy while the integration backlog grows or while accepted updates never become visible at the destination. The operational question is whether the promised workflow is completing within its expected conditions.

Keep observability proportional. A small file-based integration may need a clear run manifest and reconciliation report more than a complex tracing platform. The evidence should follow the workflow’s failure modes rather than the popularity of a monitoring tool.

Ownership of an incident should cross the same boundaries as the workflow. If each team can close its investigation after proving that its own service returned the documented response, the combined failure may have no owner. Name a workflow owner who can coordinate the evidence and decide when the end-to-end promise has been restored. This is an operational responsibility, not a requirement that one team own every implementation.

Test disagreement deliberately

Contract tests can check expected request and response shapes. Integration tests should also exercise conflicting assumptions: stale revisions, unknown identifiers, duplicate operations, delayed responses and partial completion. These cases reveal behavior that isolated unit tests cannot establish.

Use a controlled test environment or safe fixtures for consequential operations. A simulated timeout after the receiver commits an update is particularly informative because it reproduces the uncertainty a caller must handle in production.

Document what the test double cannot represent. A mock may be useful for a known response shape while failing to model real rate limits or asynchronous processing. Periodic checks against the actual integration boundary can expose drift that local tests miss.

Before declaring an integration complete, explain how it behaves when the systems disagree. Which fact wins? Which work pauses? Who can resolve the exception? The answer should be visible in code, contracts and operational evidence. That is the work that turns two individually correct systems into a dependable combined workflow.

Sources

1. HTTP semantics — checked 2026-10-14