Python Multithreading

Multithreading lets a Python program run multiple sequences of instructions, called threads, that share the same memory space within a single process. It’s the standard tool for keeping a program responsive while it waits on slow operations — network requests, file reads, database queries — without writing complicated manual scheduling code. This lesson covers how Python’s threading module works, why the Global Interpreter Lock changes what threading is actually good for, and how to avoid the subtle bugs (race conditions, deadlocks) that come with shared state.

Overview / How Multithreading Works

A process is an independent running program with its own memory space. A thread is a unit of execution inside a process. All threads within a process share the same memory — the same global variables, the same objects, the same open file handles. That shared memory is both the appeal of threading (no need to copy data between threads) and its biggest danger (two threads can corrupt shared data if they modify it at the same time without coordination).

In CPython, the reference implementation most people run, every thread you create with the threading module is a real operating-system thread (a POSIX thread on Linux/macOS, a Windows thread on Windows). However, CPython also has a mechanism called the Global Interpreter Lock (GIL): a single lock that ensures only one thread executes Python bytecode at any given instant, even on a multi-core machine. This means pure-Python threads do not give you true parallel computation for CPU-bound work — running four threads that each crunch numbers in a tight Python loop will not run noticeably faster than running them one after another, because they’re still taking turns holding the GIL.

So why use threading at all? Because the GIL is released whenever a thread performs a blocking I/O operation — reading a file, waiting on a socket, sleeping, querying a database. While one thread is blocked waiting for the operating system, the GIL is free, and another thread can run Python code. This makes threading extremely effective for I/O-bound workloads: downloading many URLs, handling many simultaneous client connections, reading several files at once. For CPU-bound workloads (heavy math, image processing, parsing large data), you generally want the multiprocessing module instead, which sidesteps the GIL by using separate processes with separate interpreters, or a library that releases the GIL internally (like NumPy).

Syntax

The core building block is the threading.Thread class. You typically either pass it a callable to run, or subclass it and override run().

thread = threading.Thread(target=some_function, args=(arg1, arg2), kwargs={}, daemon=False)
thread.start()   # begin execution in a new OS thread
thread.join()    # block the calling thread until this thread finishes

Parameter / Method	Meaning
target	The callable the thread will run when started
args / kwargs	Positional and keyword arguments passed to target
daemon	If True, the thread is killed automatically when the main program exits; if False (default), the program waits for it
start()	Schedules the thread to begin running; can only be called once
join(timeout=None)	Blocks the caller until the thread finishes (or the timeout elapses)
is_alive()	Returns True while the thread is still running
threading.current_thread()	Returns the Thread object for the calling thread

For coordinating shared data, the module also provides Lock, RLock, Event, Condition, and Semaphore. The queue.Queue class is a thread-safe FIFO queue and is usually the simplest way to hand data between threads without managing locks by hand.

Examples

Example 1: Creating and joining threads

import threading
import time

def worker(name: str, delay: float) -> None:
    time.sleep(delay)
    print(f"Thread {name} finished after {delay}s")

threads = []
for i, delay in enumerate([0.3, 0.1, 0.2], start=1):
    t = threading.Thread(target=worker, args=(f"T{i}", delay))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print("All threads completed")

Output:

Thread T2 finished after 0.1s
Thread T3 finished after 0.2s
Thread T1 finished after 0.3s
All threads completed

All three threads start almost immediately, but each sleeps for a different amount of time before printing, so the thread with the shortest delay (T2, 0.1s) finishes first, even though it was started second. The final join() loop makes the main thread wait until every worker is done before printing the last line, so "All threads completed" is always last.

Example 2: A race condition (what goes wrong without synchronization)

import threading

counter = 0

def increment_unsafe() -> None:
    global counter
    for _ in range(100_000):
        counter += 1

threads = [threading.Thread(target=increment_unsafe) for _ in range(5)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(f"Expected 500000, got {counter}")

Output (one possible run — the exact number varies):

Expected 500000, got 487213

Five threads each increment counter 100,000 times, so the mathematically correct final value is 500,000. But counter += 1 is not a single atomic operation — it compiles to a read, an addition, and a write-back as three separate bytecode steps. The GIL can switch to another thread between those steps, so two threads can both read the same old value before either writes back the incremented result, silently losing an update. Run this program several times and you’ll get a different (usually lower than 500,000) number each time — that unpredictability is the signature of a race condition.

Example 3: Fixing it with a Lock

import threading

counter = 0
counter_lock = threading.Lock()

def increment_safe() -> None:
    global counter
    for _ in range(100_000):
        with counter_lock:
            counter += 1

threads = [threading.Thread(target=increment_safe) for _ in range(5)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(f"Final counter value: {counter}")

Output:

Final counter value: 500000

A threading.Lock guarantees that only one thread can be inside the with counter_lock: block at a time. Every other thread that tries to acquire the lock while it’s held simply waits. This turns the three-step read-modify-write sequence into an atomic operation from the perspective of other threads, so no update is ever lost, and the result is exactly 500,000 every single time you run it.

Example 4: Concurrent I/O with ThreadPoolExecutor

import time
from concurrent.futures import ThreadPoolExecutor

def fetch_page(url: str) -> str:
    time.sleep(0.2)  # simulates network latency
    return f"Fetched {url} ({len(url)} chars)"

urls = ["https://a.example", "https://b.example", "https://c.example"]

with ThreadPoolExecutor(max_workers=3) as executor:
    for result in executor.map(fetch_page, urls):
        print(result)

Output:

Fetched https://a.example (17 chars)
Fetched https://b.example (17 chars)
Fetched https://c.example (17 chars)

Instead of manually creating and joining Thread objects, ThreadPoolExecutor maintains a reusable pool of worker threads. All three "fetches" run concurrently (each sleeping 0.2s at roughly the same time, so the whole thing takes about 0.2s total instead of 0.6s sequential), and executor.map guarantees the results come back in the same order as the input urls, regardless of which one finished first internally.

How It Works Step by Step / Under the Hood

• Thread creation: Thread(...) just builds a Python object; nothing runs yet.
• start(): asks the operating system to create a real native thread and begin executing your target function inside it. The calling thread continues immediately without waiting.
• GIL acquisition: before executing any Python bytecode, a thread must hold the GIL. CPython switches which thread holds the GIL periodically (by default roughly every 5 milliseconds, controlled by sys.setswitchinterval()), or immediately whenever the running thread performs a blocking call that releases the GIL (I/O, time.sleep, some C-extension calls).
• Shared memory: because all threads live in the same process, they see the same objects. There’s no serialization or copying — a list mutated by one thread is instantly visible to another, which is convenient but exactly why unsynchronized writes cause race conditions.
• join(): blocks the calling thread, checking periodically whether the target thread has finished, and returns as soon as it has (or the timeout expires).
• Program exit: Python waits for all non-daemon threads to finish before the interpreter exits. Threads marked daemon=True are killed abruptly when the main thread ends, so daemon threads should only do work that’s safe to interrupt (e.g. a background heartbeat), never something like writing a file that must finish cleanly.

Common Mistakes

Mistake 1 — expecting threads to speed up CPU-bound code. Because of the GIL, launching several threads that each run a tight Python loop of arithmetic will not run faster than doing the work sequentially — the threads are still fighting over the same GIL. If your bottleneck is computation rather than waiting, reach for multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor instead, which use separate processes (and therefore separate GILs) to achieve real parallelism across CPU cores.

Mistake 2 — mutating shared state without a lock. As shown in Example 2, incrementing a shared counter (or appending to a shared list, or updating a shared dictionary) from multiple threads without a Lock can silently drop updates or corrupt data structures. The fix, shown in Example 3, is to wrap every read-modify-write of shared state in a with some_lock: block, or better, to avoid sharing mutable state altogether by having each thread return its own result (as ThreadPoolExecutor encourages) or communicate through a thread-safe queue.Queue.

Best Practices

• Prefer concurrent.futures.ThreadPoolExecutor over raw Thread objects for most workloads — it manages the pool size, collects return values, and propagates exceptions for you.
• Use threading for I/O-bound work (network calls, file access, waiting on external processes); use multiprocessing or asyncio for CPU-bound work or very large numbers of concurrent I/O tasks.
• Always protect shared mutable state with a Lock, or avoid sharing it in the first place by passing data through a queue.Queue.
• Keep the code inside a lock’s with block as short as possible — holding a lock while doing slow work (like I/O) blocks every other thread waiting on it.
• Give threads descriptive names with the name= argument; it makes debugging and logging far easier.
• Use daemon=True only for background helper threads whose work is safe to abandon; never for threads that must finish cleanly (like ones writing to disk).
• Avoid busy-waiting (polling a flag in a loop with no sleep); use threading.Event or Condition to let a thread block efficiently until it’s signaled.

Practice Exercises

• Write a program that spawns 5 threads, each computing the sum of squares for a different range of 10,000 numbers, storing each thread’s result in a shared list protected by a Lock. Print the grand total after all threads finish.
• Take the race-condition example (Example 2) and fix it using a queue.Queue instead of a Lock: have each thread push its local increment count onto the queue, then sum the queue’s contents in the main thread. Confirm the result is always exactly 500,000.
• Use ThreadPoolExecutor to simulate downloading 10 files with random delays between 0.1 and 0.5 seconds (via time.sleep and random.uniform). Compare the total wall-clock time against running the same 10 "downloads" sequentially in a loop, and print both durations.

Summary

• Threads share memory within a process; the threading module lets you create, start, and join them.
• CPython’s GIL means only one thread runs Python bytecode at a time, so threading doesn’t parallelize CPU-bound work, but it does let I/O-bound work overlap efficiently.
• Unsynchronized access to shared data causes race conditions — protect shared state with threading.Lock or hand data between threads via queue.Queue.
• concurrent.futures.ThreadPoolExecutor is the recommended high-level API for most threaded workloads: it manages the pool, preserves result order with map(), and handles cleanup via the with statement.
• For CPU-bound parallelism, use multiprocessing instead of threads.