Stopping the Throttle: Fixing Kinesis ProvisionedThroughputExceededException

The Error Message

You’re pushing data to AWS Kinesis and suddenly your logs fill up with this error:

ProvisionedThroughputExceededException: Rate exceeded for shard shardId-000000000000 in stream <stream-name> under account <account-id>.

This isn't just a warning; it means Kinesis is dropping your data because you've hit a hard performance ceiling.

What is happening?

Think of each shard in your stream as a dedicated pipe with fixed dimensions. AWS enforces two strict speed limits on these pipes:

• Ingestion bandwidth: 1 MB per second.
• Write frequency: 1,000 records per second.

Exceed either of these, and AWS will throttle your requests. This usually happens because your total traffic has outgrown your shard count, or you have a "hot shard." A hot shard occurs when your partition keys aren't distributed well, forcing one shard to do all the heavy lifting while others sit idle.

Step 1: Identify the Hot Shard

Scaling out is expensive, so check your data distribution first. If you use a partition key like Region_ID and your US-EAST-1 region is ten times busier than others, that specific shard will choke even if the rest of your stream is empty.

Open the CloudWatch console and dive into these specific metrics:

• IncomingBytes and IncomingRecords (filtered by ShardId).
• WriteProvisionedThroughputExceeded.

Look for outliers. If one shard is peaking at 1,000 records/sec while the others are at 50, you have a partition key problem, not a capacity problem.

Step 2: Redesign Your Partition Keys

High cardinality is your best friend here. You want keys with thousands of unique values to ensure data spreads evenly across all available shards. Instead of using a broad category like Store_Location, try a UUID or a composite key like User_ID + Timestamp.

If you need to verify how your keys will distribute, hashing is a reliable shortcut. For instance, generating a SHA-256 hash of your data ensures a near-perfect random distribution. You can use the Hash Generator at ToolCraft to test how different inputs produce the unique strings needed for balanced shards.

Step 3: Add Exponential Backoff and Retries

Traffic is rarely a flat line; it comes in bursts. Your application must be resilient enough to pause and try again when it hits a temporary wall. While the AWS SDK handles some retries, writing a custom loop gives you more control over the "jitter" and delay.

Here is a Python example using boto3 that implements a basic backoff strategy:

import boto3
import time
import random
from botocore.exceptions import ClientError

kinesis = boto3.client('kinesis', region_name='us-east-1')

def put_record_with_retry(data, partition_key):
    attempt = 0
    max_attempts = 5

    while attempt < max_attempts:
        try:
            return kinesis.put_record(
                StreamName='my-data-stream',
                Data=data,
                PartitionKey=partition_key
            )
        except ClientError as e:
            if e.response['Error']['Code'] == 'ProvisionedThroughputExceededException':
                attempt += 1
                # Exponential backoff with a bit of random jitter
                delay = (2 ** attempt) + random.uniform(0, 1)
                print(f"Throttled. Retry {attempt}/{max_attempts} in {delay:.2f}s...")
                time.sleep(delay)
            else:
                raise e
    raise Exception("Failed after max retries")

Step 4: Scale the Stream (Resharding)

Sometimes you simply need a bigger pipe. If CloudWatch shows that all your shards are consistently hovering near 800-900 records per second, it’s time to reshard. This process adds capacity without taking your stream offline.

Use the AWS CLI to bump your shard count instantly:

aws kinesis update-shard-count \
    --stream-name my-data-stream \
    --target-shard-count 10 \
    --scaling-type UNIFORM_SCALING

Keep in mind that doubling your shard count doubles your hourly cost. Monitor your usage to ensure you aren't over-provisioning.

How to Verify the Fix

Don't just walk away once the code is deployed. Monitor these three areas to ensure the errors are gone:

• CloudWatch Trends: The WriteProvisionedThroughputExceeded count should drop to near zero.
• Consumer Health: Check your Lambda or KCL lag. If producers are retrying too often, your consumers might run out of data to process.
• Error Logs: Run a quick grep or Insight query on your logs. The ProvisionedThroughputExceededException should only appear during massive, unexpected spikes.

Expert Tips

• Batch your writes: Stop using put_record. Use put_records (plural) to send up to 500 records in one request. This cuts down on HTTP overhead significantly.
• Try the Kinesis Producer Library (KPL): For Java users, the KPL is a powerhouse. It handles complex batching and retries in the background, making your producer much more efficient.
• Consider On-Demand Mode: If your traffic is a rollercoaster and you hate manual scaling, switch to On-Demand mode. You pay a premium per GB, but AWS handles the shard management for you automatically.