Duplicate Detection

Overview

What it means in plain English

Duplicate detection is the process of identifying records, events, requests, or transactions that may represent the same intended financial action. It compares identifiers, idempotency keys, references, amounts, parties, timing, and payload fingerprints. Detection does not automatically prove duplication, so the system should distinguish exact repeats, probable duplicates, legitimate retries, installments, recurring payments, and separate transactions with similar details. The decision evidence should remain reviewable and auditable.

How it works and why it matters

Duplicate Detection protects payment systems from repeated submission, delivery, or processing. Potential duplicates can appear when a customer clicks twice, a client retries after a timeout, a webhook is redelivered, or a batch file is imported more than once.

Strong detection uses stable unique identifiers when available. A provider transaction ID, order reference, event ID, or idempotency key is more reliable than comparing amount and timestamp alone. Payload hashes can support detection but must account for fields that legitimately change between attempts.

The detector should classify the suspected duplication layer. A Duplicate Event may be the same notification delivered twice, while a Duplicate Transaction can be two actual financial instructions. These conditions require different responses.

Similarity rules need a review path because false positives can suppress legitimate recurring or split payments. The system should record matching features, confidence, prior record, and final disposition rather than returning a hidden duplicate flag.

Detection is one part of Duplicate Prevention . Prevention uses idempotency, uniqueness constraints, state checks, and atomic writes to stop repeated effects. Monitoring should measure suspected duplicates, confirmed duplicates, suppressed repeats, and missed duplicate incidents.

Detection rules should be tested with known legitimate patterns such as subscriptions, split shipments, partial captures, repeated invoices, and same-amount transfers between the same parties. A model tuned only on obvious double clicks can over-block valid business activity. Teams should review false-positive and false-negative samples, adjust thresholds by workflow, and retain an explicit reason for every automatic duplicate decision.

High-risk matches can be held before value moves, while lower-confidence matches can be flagged without blocking. The response should be proportional to both confidence and financial consequence.

Worked example

Two payment records have the same chain, transaction hash, log index and transfer details, suggesting they describe the same asset movement. A checker flags them before posting a second accounting entry. Separately, two distinct hashes with the same amount may be entirely legitimate.

Practical checks and common confusion

Detection finds possible duplicates; suppression decides what to ignore. Design a key appropriate to the asset type, including log index for transactions with multiple token transfers. Preserve original observations and audit actions.

Learn the related concepts

Duplicate Suppression Payment Monitoring Rule