Fixing DynamoDB TransactionConflictException: When Transactions Collide

The 2 AM Wake-Up Call

Your phone buzzes. It is 2 AM, and the monitoring dashboard is a sea of red. A sudden traffic spike just triggered a wave of 500 errors in your order processing service. When you dive into the CloudWatch logs, one specific error appears thousands of times:

ClientError: An error occurred (TransactionConflictException) when calling the TransactWriteItems operation: Transaction is ongoing for the item

This happens because TransactWriteItems and TransactGetItems require exclusive access to items during the operation. If Transaction A is still being processed and Transaction B attempts to touch the same partition key, AWS rejects Transaction B immediately. It doesn't queue the request; it simply kills it.

Why Your Transactions Are Failing

In our recent production incident, we were updating a 'Stock' item and creating an 'Order' record in one atomic step. During a flash sale, 800+ concurrent Lambda executions tried to decrement the same 'Stock' item at the exact same millisecond.

DynamoDB transactions are ACID compliant, but they aren't built for high-contention on a single row. Think of it as a narrow hallway. If one person is walking through, everyone else has to wait outside. If 100 people try to shove through at once, DynamoDB starts turning people away to prevent a jam.

The Solution: A Three-Pronged Approach

Increasing your provisioned throughput won't fix this. You need to change how your application handles the "traffic jam."

1. Smart Retries with Jitter

The fastest fix is to catch the exception and try again. But if 100 failed requests all retry at exactly 100ms, they will just collide again. This is the "thundering herd" problem. You must use exponential backoff combined with jitter (randomized delay).

Here is a robust implementation using Python and Boto3:

import boto3
import time
import random
from botocore.exceptions import ClientError

dynamodb = boto3.client('dynamodb')

def update_stock_with_retry(item_id, quantity, max_retries=5):
    for attempt in range(max_retries):
        try:
            return dynamodb.transact_write_items(
                TransactItems=[
                    {
                        'Update': {
                            'TableName': 'Inventory',
                            'Key': {'ID': {'S': item_id}},
                            'UpdateExpression': 'SET stock = stock - :q',
                            'ExpressionAttributeValues': {':q': {'N': str(quantity)}},
                            'ConditionExpression': 'stock >= :q'
                        }
                    }
                ]
            )
        except ClientError as e:
            if e.response['Error']['Code'] == 'TransactionConflictException':
                # Exponential backoff: 50ms, 100ms, 200ms... 
                # We add random jitter to spread the load
                wait_time = ((2 ** attempt) * 0.05) + random.uniform(0, 0.02)
                print(f"Collision on {item_id}. Retrying in {wait_time:.3f}s...")
                time.sleep(wait_time)
            else:
                raise e
    raise Exception("Transaction failed after maximum retry attempts.")

2. Shrink Your Transaction Scope

Every item you add to a TransactWriteItems call increases the mathematical odds of a conflict. Does your 'LastLogin' timestamp really need to be updated in the same transaction as a financial 'Purchase'? Probably not. Move non-critical updates to a separate UpdateItem call. Standard updates use optimistic locking, which handles high concurrency much more gracefully than a full transaction.

3. Use Distributed Counters

If you are hitting a bottleneck on a single global counter, transactions are the wrong tool. Instead, use a sharding pattern. Split your counter into 10 or 20 different rows (e.g., counter_shard_1, counter_shard_2). Your application picks a random shard to update. When you need the total count, simply sum all the shards. This spreads the "heat" across multiple partition keys.

Verifying the Fix

Once you deploy the backoff logic, keep a close eye on your metrics. You want to see your error rate drop while maintaining data integrity.

- **CloudWatch Metrics:** Track `TransactionCheckFailed`. After implementing jitter, this number might still be above zero, but your application-level 500 errors should vanish as retries succeed.
- **Latency Monitoring:** Watch your Lambda `Duration`. Retries add milliseconds to your execution time. If your latency spikes above 500ms, you may need to increase your shard count.
- **Stress Testing:** Run a quick load test. We used a simple script to fire 50 concurrent requests at one item, and our success rate went from 12% to 100% after adding jittered retries.

Key Takeaways

DynamoDB transactions are powerful, but they aren't a direct replacement for traditional SQL transactions. They come with strict concurrency limits.

- **Limit Hot Keys:** If an item is updated more than 5-10 times per second, transactions will likely struggle.
- **Check SDK Settings:** Many AWS SDKs have default retries, but they often don't retry `TransactionConflictException`. Always verify your specific SDK's behavior.
- **Enforce Idempotency:** Use the `ClientRequestToken` parameter. This ensures that if a retry succeeds but the network times out, you don't accidentally charge a customer twice.