Use this tamper-evidence test plan
Seed a payment with a known event ID. Compare its amount, actor, time, and state in the source service, queue, audit store, and reconciliation output. Then try to update, delete, or delay one event in a safe environment.
| Item | Check or owner | Evidence |
|---|---|---|
| Insert | Can a payment event be omitted? | Compare ledger sequence |
| Update | Can old events be changed? | Verify hash or protected copy |
| Delete | Can an admin erase evidence? | Test restricted role |
| Delay | Can logs arrive out of order? | Compare source event IDs |
Test and retest
Check whether an independent copy or alert exposes the change. If the same admin can change the payment and erase the log, the trail is weak. Keep source and observed records from the test, plus who reviewed the alert.
Worked synthetic case
Synthetic case: A staff admin changes a payout from pending to paid. The app event is logged, but the same admin can delete it from the audit store. The record exists until someone needs it most.
Seed an event ID and compare amount, actor, time, and status across source, queue, audit store, and reconciliation report. Try an authorized status change, a privileged edit, deletion, and delayed delivery. The pass condition is that unauthorized change is blocked or exposed by an independent copy and alert.
Give the alert to a person outside the payment-changing role. Test absence and delay as well as altered content. A hash chain can detect edits only when its anchor and key are protected from the same actor. Keep the test trace and alert response.
More log copies improve detection but increase sensitive-data exposure. Log IDs and state changes rather than full account data where possible, and limit access to the protected copy.
Distinguish tampering from missing events
Hashing rows detects a changed row only when a trusted checkpoint preserves the expected value. It does not show that an action was never logged. For a synthetic payout approval, compare business actions to audit events by request ID. Then delete one event in a test environment and verify the off-system sequence checkpoint detects the gap. Separately, submit a sensitive action while the log sink is unavailable. The action should either fail under a written policy or create a durable local queue item that later reaches the sink. Record the exact gap, alert time, and repair. Never claim tamper evidence from a hash that the same database owner can rewrite with the rows.
Run the gap check after a restore from backup too. A restored log store can look valid inside itself while omitting events that were exported after the backup was taken.
Use a sequence that has a known end
Synthetic fixture: stream PAY-A has committed events 101, 102, and 103. A separate protected checkpoint holds the stream ID, last sequence 103, and hash of that last event. Delete event 102 in the test copy. Then remove event 103 and recompute every remaining hash. The first test checks a gap; the second checks truncation at the tail. Both must differ from the checkpoint. A random event ID helps match records but does not tell a checker which numbers or actions are missing.
Checkpoints must describe committed business actions. A database sequence can skip numbers after rollback, so a missing allocated number alone is not proof of a missing payment. Match the business action ID and its required audit event. Keep the checkpoint outside the payment administrator’s write scope and show who can replace it. If that administrator can alter both copies, the drill has not tested an independent control.
Measure the delivery window and restore path
Set a synthetic delivery target of 60 seconds, then delay event 103 for 90 seconds. At 60 seconds the check should flag overdue delivery; after replay it should close the gap without creating a second event. These times are test settings, not a legal retention or alert rule. Record source commit time, queue acceptance, sink write, alert, and repair.
Restore the audit store from a backup ending at event 102 while the checkpoint still ends at 103. Recovery must fetch and verify the later event before declaring the stream complete. Distinguish altered content, delayed content, and a source action that emitted no event. The OWASP logging guide covers testing and protecting logs; the numbered fixture is a test design for that goal.