Concurrency in Python: Threads vs Processes

Computers run tasks in parallel using:

• CPU cores → true parallelism (multiple chefs)
• Scheduling → switching between tasks (one chef, many pots)
• I/O waits → idle time you can use productively (waiting for water to boil)

A thread is a lightweight unit of execution within a single process.

import threading
import time

def task(name):
    # Each thread announces when it starts
    print(f"{name} starting")
    
    # Simulate I/O wait (like downloading a file)
    # During this sleep, other threads can run!
    time.sleep(0.5)
    
    print(f"{name} done")

# Create 3 thread objects
# Each thread will run the 'task' function with a different name
threads = [
    threading.Thread(target=task, args=(f"Thread-{i}",))
    for i in range(3)
]

# Start all threads (they begin running)

...

The GIL (Global Interpreter Lock) is a mutex that protects access to Python objects.

This means:

A process is a full Python interpreter with its own memory.

Basic multiprocessing:

import multiprocessing
import time

def cpu_task(name):
    print(f"{name} starting")
    
    # CPU-intensive work: calculate sum of squares
    # This keeps the CPU busy (not waiting for I/O)
    total = sum(i * i for i in range(10_000_000))
    
    print(f"{name} done: {total}")

# IMPORTANT: This guard is required on Windows!
# Without it, the script would spawn infinite processes
if __name__ == "__main__":
    # Create 2 process objects
    processes = [
        multiprocessing.Process(tar
...

Many real systems use both:

• Processes for CPU-heavy pipelines
• Threads for network/database operations
• AsyncIO for massive lightweight tasks

This is how production-grade scrapers, ML preprocessors, and automation bots work.

Python uses cooperative + preemptive scheduling for threads.

Here's what really happens:

This creates:

• artificial bottlenecks for CPU tasks
• no bottlenecks for I/O (because threads release the GIL while waiting)

This explains why:

• ✔ 100 threads downloading files works great
• ❌ 100 threads crunching numbers does NOT

Let's say you need to download 10,000 images.

Why?

Because threads wait most of the time, so Python overlaps waits.

Common I/O sources:

• HTTP requests
• reading/writing files
• database queries
• cloud APIs
• waiting for user input
• network sockets
• sensor data

Threads shine because I/O releases the GIL.

CPU-bound tasks include:

• encryption
• compression
• ML preprocessing
• hashing
• physics simulation
• number crunching
• image/video filtering
• audio processing

Processes use true multi-core hardware.

The next sections cover professional-level concurrency patterns:

A race condition happens when:

• Two or more threads access shared data
• AND at least one thread modifies it
• AND execution order determines the final result

Example — unsafe counter:

import threading

counter = 0

def increment():
    global counter
    for _ in range(100_000):
        counter += 1

threads = [threading.Thread(target=increment) for _ in range(10)]
[t.start() for t in threads]
[t.join() for t in threads]

print(counter)   # NEVER equals 1,000,000

Locks ensure only ONE thread accesses critical code at once.

import threading

lock = threading.Lock()
counter = 0

def increment():
    global counter
    for _ in range(100_000):
        with lock:
            counter += 1

threads = [threading.Thread(target=increment) for _ in range(10)]
[t.start() for t in threads]
[t.join() for t in threads]

print(counter)  # Now correctly equals 1,000,000

Advanced topics covered:

You now understand advanced concurrency:

You're operating at professional backend engineer level now.