Parallel processing

ParallelPipeline distributes the work between multiple processes. This can speed up processing massively in CPU-bound applications.

class morphocut.parallel.ParallelPipeline(num_workers=None, queue_size=2, multiprocessing_context=None, parent=None)[source]

Parallel processing of the stream in multiple processes.

Parameters:

num_workers (int, optional) – Number of worker processes. Default: Number of CPUs in the system.
queue_size (int, optional) – Upperbound limit on the number of items that can be placed in the input queue of each worker. If queue_size is less than or equal to zero, the queue size is infinite.
multiprocessing_context (optional) – Result of multiprocessing.get_context().
parent (Pipeline) – The parent pipeline.

Note

ParallelPipeline creates distinct copies of its nodes in each worker thread that are not accessible from the main thread.

Example

with Pipeline() as pipeline:
    # Regular sequential processing
    ...
    with ParallelPipeline():
        # Parallelized processing in this block,
        # work is distributed between all cores.
        ...

pipeline.run()