Fixing AWS Lambda's [Errno 28]: How to Handle the 512MB /tmp Limit

The Problem

Imagine you are processing a 700MB CSV file from an S3 bucket. Your Lambda function has 2GB of RAM, so you expect it to breeze through the task. Instead, the execution halts abruptly with a disk space error. This happens because, by default, AWS Lambda provides only 512 MB of ephemeral storage in the /tmp directory.

[Errno 28] No space left on device: '/tmp/large_file.csv'

If your code downloads a file larger than 512MB, or if multiple concurrent processes fill that space, the environment hits a hard limit. Your function doesn't just slow down—it crashes immediately.

Solution 1: Scale Ephemeral Storage

The fastest fix is to increase the allocated disk space. AWS allows you to scale /tmp up to 10,240 MB (10 GB). While the first 512MB is free, keep in mind that additional storage costs roughly $0.00003090 per GB-second in most regions.

Using the AWS Console

• Navigate to the Lambda console and open your specific function.
• Select the Configuration tab.
• Choose General configuration from the left menu.
• Click Edit and locate the Ephemeral storage setting.
• Update the value (e.g., to 2048 MB) and hit Save.

Using the AWS CLI

For those automating infrastructure via CI/CD, use this command to update your settings instantly:

aws lambda update-function-configuration \
    --function-name MyProcessingFunction \
    --ephemeral-storage '{"Size": 2048}'

Solution 2: Stream Data to Bypass the Disk

Writing to /tmp is often unnecessary. You can save money and improve performance by streaming data directly into memory. This approach avoids the disk bottleneck entirely by processing the file in chunks.

Example: Streaming S3 data in Python (Boto3)

Instead of using download_file(), use get_object() to access a streaming body. This keeps your disk usage at zero.

import boto3
import csv

s3 = boto3.client('s3')

def lambda_handler(event, context):
    bucket = "my-data-bucket"
    key = "large_file.csv"
    
    response = s3.get_object(Bucket=bucket, Key=key)
    
    # Process the stream line-by-line
    lines = (line.decode('utf-8') for line in response['Body'].iter_lines())
    reader = csv.DictReader(lines)
    
    for row in reader:
        # Your logic here
        pass

Solution 3: Manage Warm Container Cleanup

AWS often reuses Lambda execution environments to reduce cold starts. If your code writes a 200MB file to /tmp and fails to delete it, the next execution starts with that 200MB already gone. After three runs, you hit the 512MB limit.

Always wrap your file operations in a try...finally block. This ensures that files are deleted even if the script encounters an error mid-way.

import os

file_path = "/tmp/temp_data.tmp"
try:
    # Perform your heavy lifting
    pass
finally:
    if os.path.exists(file_path):
        os.remove(file_path)

Solution 4: Mount Amazon EFS for Massive Workloads

Sometimes 10GB isn't enough. If you are handling multi-terabyte datasets or need a shared workspace for multiple functions, Amazon Elastic File System (EFS) is the right tool. This requires your Lambda to run inside a VPC.

• Create an EFS File System and an Access Point.
• Update your VPC security groups to allow NFS traffic (Port 2049).
• Mount the EFS to a local path like /mnt/storage in the Lambda configuration.

How to Verify Your Storage Limit

Not sure if your changes took effect? You can print the actual available disk space to your CloudWatch logs using the shutil library.

import shutil

def lambda_handler(event, context):
    total, used, free = shutil.disk_usage("/tmp")
    print(f"Total capacity: {total // (2**20)} MiB")
    print(f"Current usage: {used // (2**20)} MiB")

Quick Tips

• Disk vs. RAM: Don't confuse /tmp space with function memory. If you load a 1GB file into a variable, you need to increase your Memory setting, not ephemeral storage.
• Isolation: Each concurrent Lambda execution gets its own dedicated /tmp. One function cannot see or fill the disk of another.
• Cost: Streaming data is almost always cheaper than paying for extra ephemeral storage.