Skip to main content

Lib/concurrent/futures/process.py

Source:

cpython 3.14 @ ab2d84fe1023/Lib/concurrent/futures/process.py

process.py implements ProcessPoolExecutor. It is the most complex file in concurrent.futures because it must deal with process crashes, a multiprocessing call queue, a result queue, and a management thread that coordinates between the calling process and the worker pool.

Map​

LinesSymbolRole
1-100_WorkItem, _ResultItemPicklable call/result containers
101-250_process_worker, _process_chunkEntry point run in each worker process
251-480_ExecutorManagerThreadBackground thread in main process; drains result queue
481-740ProcessPoolExecutorPublic executor; submit, map, shutdown

Reading​

Worker entry point​

Each worker process runs _process_worker, which loops over a SimpleQueue, calling each _WorkItem and putting a _ResultItem back on the result queue.

# CPython: Lib/concurrent/futures/process.py:101 _process_worker
def _process_worker(call_queue, result_queue, initializer, initargs, max_tasks=None):
    if initializer is not None:
        try:
            initializer(*initargs)
        except BaseException:
            _base.LOGGER.critical('Exception in initializer:', exc_info=True)
            return
    while True:
        call_item = call_queue.get(block=True)
        if call_item is None:
            result_queue.put(os.getpid())
            return
        ...

A None sentinel signals the worker to shut down gracefully. Workers send their PID back before exiting so the manager thread can join them.

_ExecutorManagerThread​

The manager thread is started by the first submit() call. It runs a select-based loop over three file descriptors: the result queue reader, a wakeup pipe that fires when a worker dies unexpectedly, and a shutdown event.

# CPython: Lib/concurrent/futures/process.py:330 _ExecutorManagerThread.run
def run(self):
    while True:
        self.add_call_item_to_queue()
        result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
        if is_broken:
            self.terminate_broken(cause)
            return
        if result_item is not None:
            self.process_result_item(result_item)
            del result_item
        if self.is_shutting_down():
            self.flag_executor_dead_and_cleanup()
            return

When a worker crashes (is_broken=True), all pending futures receive BrokenProcessPool as their exception.

Pickling constraint​

All arguments and return values must be picklable because they cross process boundaries through multiprocessing.SimpleQueue. ThreadPoolExecutor has no such constraint.

gopy notes​

Not yet ported. A Go port would use os/exec for worker processes and encoding/gob or JSON for call/result serialization. The wakeup-pipe pattern maps naturally to a chan struct{} in Go.