multiprocessing: True Parallelism

Reviewed & published by Brayan K

The multiprocessing module runs your code in separate OS processes — each with its own Python interpreter and its own GIL — so CPU-heavy work can run on multiple cores at the same time.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Where threads take turns under the GIL, processes run genuinely in parallel. This is the tool you reach for when you need raw computational throughput.

What You'll Learn in This Lesson

• Launching work with multiprocessing.Process
• Why processes sidestep the GIL for CPU-bound work
• The all-important if __name__ == '__main__' guard
• Fanning out tasks with Pool.map
• Returning data through a Queue
• The modern ProcessPoolExecutor API

1 Process, the Guard, and Returning Data

A Process works much like a Thread — give it a target and args, then start() and join(). But because processes don't share memory, you can't just write to a shared dict. Instead you pass results back through a Queue. Notice the if __name__ == "__main__": guard — it's mandatory.

from multiprocessing import Process, Queue

def square(n, q):
    q.put((n, n * n))            # send the result back

if __name__ == "__main__":       # required guard!
    q = Queue()
    procs = [Process(target=square, args=(n, q)) for n in [2, 3, 4]]
    for p in procs:
        p.start()
    for p in procs:
        p.join()

    results = sorted(q.get() for _ in procs)
    print(results)
    # [(2, 4), (3, 9), (4, 16)]

from multiprocessing import Process, Queue

def square(n, q):
    q.put((n, n * n))

if __name__ == "__main__":
    q = Queue()
    procs = [Process(target=square, args=(n, q)) for n in [2, 3, 4]]
    for p in procs:
        p.start()
    for p in procs:
        p.join()

    # sort so output is deterministic regardless of finish order
    results = sorted(q.get() for _ in procs)
    print(results)

# ✅ Expected output:
# [(2, 4), (3, 9), (4, 16)]

2 Pool.map: Fan Out CPU Work

A Pool manages a fixed set of worker processes. pool.map(func, iterable) splits the iterable across the workers, runs them in parallel on different cores, and returns the results in order. This is the cleanest way to parallelize CPU-bound work:

from multiprocessing import Pool

def heavy(n):
    total = 0
    for i in range(n * 1000):    # a little real CPU work
        total += i
    return total

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(heavy, [1, 2, 3, 4])
    print(results)
    # [499500, 1999000, 4498500, 7998000]

Because each heavy call runs in its own process with its own GIL, four cores can grind away simultaneously — something threads simply cannot do for this kind of pure-Python loop.

from multiprocessing import Pool

def heavy(n):
    total = 0
    for i in range(n * 1000):
        total += i
    return total

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(heavy, [1, 2, 3, 4])
    print(results)

# ✅ Expected output:
# [499500, 1999000, 4498500, 7998000]

3 The Modern API: ProcessPoolExecutor

concurrent.futures.ProcessPoolExecutor is the modern, unified interface. The best part: it shares the exact same API as ThreadPoolExecutor from the previous lesson. Swap one class name and your code goes from threads to processes:

from concurrent.futures import ProcessPoolExecutor

def cube(n):
    return n ** 3

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=3) as pool:
        results = list(pool.map(cube, range(1, 6)))
    print(results)
    # [1, 8, 27, 64, 125]

from concurrent.futures import ProcessPoolExecutor

def cube(n):
    return n ** 3

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=3) as pool:
        results = list(pool.map(cube, range(1, 6)))
    print(results)

# Expected output:
# [1, 8, 27, 64, 125]

🎯 Your Turn: Fill in the Blanks

Replace each ___ so a Pool doubles every number. Remember the guard and which method maps a function over the list.

from multiprocessing import Pool

def double(n):
    return n * 2

if ___ == "__main__":            # hint: the special module name
    with Pool(processes=2) as pool:
        results = pool.___(double, [10, 20, 30])   # hint: maps func over list
    print(results)

# Expected output:
# [20, 40, 60]

⚠️ Common Errors

❌ Omitting the __main__ guard

# Top-level code that spawns processes — recursion bomb on Win/macOS
with Pool() as pool:
    pool.map(work, data)

✅ Wrap it in the guard

if __name__ == "__main__":
    with Pool() as pool:
        pool.map(work, data)

❌ Expecting a shared variable to update

total = 0
def add(n):
    global total
    total += n        # changes the CHILD's copy, not yours

✅ Return the value and combine results

with Pool() as pool:
    total = sum(pool.map(lambda n: n, data))

❌ Using processes for tiny, fast tasks

If each task is trivial, process startup and pickling overhead can make multiprocessing slower than a plain loop. Reserve it for genuinely heavy, CPU-bound work.

🎯 Mini Challenge: Count Vowels in Parallel

Use a ProcessPoolExecutor to count the vowels in each word, build a word→count dictionary, and print the total number of vowels.

from concurrent.futures import ProcessPoolExecutor

def count_vowels(word):
    return (word, sum(1 for c in word if c in "aeiou"))

if __name__ == "__main__":
    words = ["banana", "apple", "sky", "queue"]
    with ProcessPoolExecutor(max_workers=2) as pool:
        counts = dict(pool.map(count_vowels, words))
    print(counts)
    print("Total vowels:", sum(counts.values()))

# Expected output:
# {'banana': 3, 'apple': 2, 'sky': 0, 'queue': 4}
# Total vowels: 9

📚 Keep Learning

• • multiprocessing — Process-based parallelism
• • concurrent.futures.ProcessPoolExecutor
• • multiprocessing — Programming guidelines

Frequently Asked Questions

Lesson complete — you can use every CPU core!

You can launch a Process, return data through a Queue, fan work out with Pool.map, and use the modern ProcessPoolExecutor. You understand why processes sidestep the GIL for CPU-bound work — and you never forget the if __name__ == "__main__": guard.

Practice quiz

Continue this course

• Previous: Threading & the GIL
• Next: Thread-Safe Queues (queue) — Coordinate producers and consumers safely with queue.Queue
• Quick reference: Python cheat sheet

Frequently asked questions

How does multiprocessing get around the GIL?

Each process is a separate Python interpreter with its own memory and its own GIL. Because they're independent OS processes, the operating system can schedule them on different CPU cores simultaneously. So while threads share one GIL and take turns, processes truly run at the same time — which is exactly what CPU-bound work needs.

Why is the 'if __name__ == "__main__":' guard required?

On Windows and macOS, new processes are created by importing your script fresh. Without the guard, that re-import would launch the process-spawning code again, leading to infinite recursion of new processes. Putting the launch code inside 'if __name__ == "__main__":' ensures it only runs in the original program, not in the freshly imported children.

When should I use multiprocessing vs threading?

Use multiprocessing for CPU-bound work — heavy math, image processing, parsing large files — because it uses multiple cores. Use threading for I/O-bound work — network calls, disk reads — where you're mostly waiting. A rough rule: if your task pegs a CPU core, go multiprocessing; if it mostly waits, go threading.

What's the cost of using processes?

Processes are heavier than threads. Starting them takes time, and because they don't share memory, any data you send to them or get back must be pickled (serialized) and copied. For tiny, fast tasks the overhead can outweigh the benefit. Multiprocessing pays off when each task does enough real work to dwarf that startup and copying cost.