TL;DR: The Quick Fix
Python throws the TypeError: cannot pickle '<lambda>' object because the multiprocessing module needs to "package" (serialize) your code to send it to other CPU cores. The standard pickle library simply doesn't know how to handle anonymous functions like lambdas or nested logic.
The solution: Move your logic out of the lambda and into a named function at the top level of your script. This gives Python a clear "map" to find the function in a new process.
import multiprocessing
# ❌ THIS FAILS
# with multiprocessing.Pool(4) as p:
# p.map(lambda x: x * 10, [1, 2, 3])
# ✅ THIS WORKS
def multiply_by_ten(x):
return x * 10
if __name__ == "__main__":
with multiprocessing.Pool(processes=4) as p:
print(p.map(multiply_by_ten, [1, 2, 3]))
Why Serialization Fails
When you start a Pool, Python spawns fresh worker processes. These workers don't share memory with your main script. To get your function over to them, Python "pickles" it—converting the function into a byte stream that can be rebuilt on the other side.
Pickle works by looking up objects by their specific name. Since a lambda is anonymous, it has no unique name for the worker process to import. You will run into the exact same wall with:
- **Nested functions:** Functions defined inside another function.
- **Instance methods:** Methods attached to a class (especially on Windows/macOS).
- **System resources:** Open file handles, database connections, or network sockets.
OS differences matter here. Linux often defaults to fork, which copies the entire memory space and might let you get away with unpicklable objects. However, macOS (since Python 3.8) and Windows use spawn, which starts a clean slate and enforces strict pickling rules.
Three Ways to Fix It
1. Use Top-Level Functions
This is the cleanest approach. By placing your worker function at the module level, you ensure that every worker process can find and import it by name. It’s reliable, readable, and doesn't require extra libraries.
# logic.py
def heavy_lifting(data):
return sum(data) / len(data)
# main.py
from multiprocessing import Pool
from logic import heavy_lifting
if __name__ == "__main__":
items = [[1, 2], [3, 4], [5, 6]]
with Pool() as pool:
results = pool.map(heavy_lifting, items)
2. Swap Pickle for Pathos
If your project relies heavily on lambdas or complex class structures, try the pathos library. It uses dill instead of pickle. dill is much smarter; it can serialize almost anything, including closures and lambdas, by capturing the actual bytecode.
Install it via terminal:
pip install pathos
Then, swap your import:
from pathos.multiprocessing import ProcessingPool as Pool
if __name__ == "__main__":
# This works perfectly with pathos
with Pool(4) as p:
result = p.map(lambda x: x ** 2, [10, 20, 30])
print(result)
3. Convert Methods to Static Methods
Trying to pickle self.my_method often fails because Python tries to pickle the entire object instance (self). If your object holds a database connection or a 500MB data cache, the process will crash. Instead, use a @staticmethod to decouple the logic from the instance state.
class DataProcessor:
@staticmethod
def clean_string(text):
return text.strip().lower()
def run(self, data_list):
with multiprocessing.Pool() as p:
# We pass the static method, avoiding the 'self' pickling trap
return p.map(self.clean_string, data_list)
How to Verify Your Fix
Don't assume it works just because it passes on your local Linux machine. Follow these three steps to ensure portability:
- **Test for 'spawn' compatibility:** Add `multiprocessing.set_start_method('spawn', force=True)` at the very top of your script. If it works here, it will work on any OS.
- **The `__main__` Guard:** Always wrap your entry point in `if __name__ == "__main__":`. Without this, workers might try to spawn their own workers, leading to a CPU-crushing recursion loop.
- **Small Batch Run:** Run a test with 2 workers and 10 items. If there is a serialization mismatch, Python will report it within the first few seconds.
Further Reading
- [Python Docs: Picklable Objects](https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled)
- [Understanding Spawn vs. Fork](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods)