Master Python's Multiprocessing Pool: Apply Vs Map Vs Imap Vs Starmap Explained

Apply Function: Parallel Execution of Single Function

The Apply function is used to parallelize the execution of a single function across multiple processes. It takes two parameters: the target function to be executed and a tuple of arguments for that function. The function is executed asynchronously by the worker processes in the pool, and the result is returned as a plain object.

import multiprocessing

def square(x):

pool = multiprocessing.Pool()
result = pool.apply(square, (5,))

In this example, the Apply function is used to calculate the square of 5 using the square function. The result, 25, is stored in the result variable.

Map Function: Parallel Execution of Single Function with Multiple Inputs

The Map function extends the Apply function by allowing the execution of a single function on a sequence of inputs. It takes two parameters: the function to be applied and an iterable of inputs. The function is applied to each input in parallel, and the results are returned as a list.

import multiprocessing

def square(x):

numbers = [1, 2, 3, 4, 5]
pool = multiprocessing.Pool()
result = pool.map(square, numbers)

In this example, the Map function squares each number in the numbers list in parallel. The result is a list containing the squares of each number.

Imap Function: Iterative Fetching of Results from Parallel Execution

The Imap function is similar to Map, but it returns an iterator instead of a list. This allows for more efficient handling when working with large amounts of data or when the results need to be processed incrementally. The iterator can be used to fetch results as they become available.

import multiprocessing

def square(x):

numbers = [1, 2, 3, 4, 5]
pool = multiprocessing.Pool()
result_iter = pool.imap(square, numbers)

for result in result_iter:
    print(result)

In this example, the Imap function squares each number in the numbers list in parallel and returns an iterator. The for loop iterates over the iterator and prints each result as it becomes available.

Starmap Function: Parallel Execution of Function with Unpacked Inputs

The Starmap function is a variant of Map that accepts an iterable of tuples as input instead of a sequence of single values. The function is applied to each tuple, with each element of the tuple being passed as a separate argument to the function.

import multiprocessing

def sum_pair(x, y):
    return x + y

pairs = [(1, 2), (3, 4), (5, 6)]
pool = multiprocessing.Pool()
result = pool.starmap(sum_pair, pairs)

In this example, the Starmap function sums each pair in the pairs list in parallel. The result is a list containing the sums of each pair.

The Apply, Map, Imap, and Starmap functions offer varying levels of parallelism and flexibility for parallel programming tasks. Their choice depends on the specific requirements of the application.