Python: run functions in parallel with a multiprocessing wrapper function
One common way to run functions in parallel with Python is to use the multiprocessing module which is powerful, it has many options to configure and a lot of things to tweak.
But what about if we want just a very simple functionality like running a number of functions in parallel and nothing else?
We can use the following wrapper function for this:def run_parallel(*functions):
'''
Run functions in parallel
'''
from multiprocessing import Process
processes = []
for function in functions:
proc = Process(target=function)
proc.start()
processes.append(proc)
for proc in processes:
proc.join()
This function allows us to input as parameter any number of functions with or without their parameters and executed in parallel.
Example#!/usr/bin/env python3def task_a():
print('this is task a')def task_b():
print('this is task b')def hello(msg):
print('Hello: %s'%(msg))def run_parallel(*functions):
'''
Run functions in parallel
'''
from multiprocessing import Process
processes = []
for function in functions:
proc = Process(target=function)
proc.start()
processes.append(proc)
for proc in processes:
proc.join()if __name__ == '__main__':
run_parallel(task_a(),task_b(),hello(msg="konstantinos"))
Executing this will bring the following results:this is task a
this is task b
Hello: konstantinos
This is nice and simple, but how we will get results back from the functions?
To do this we need to use a shared variable like a list and append the results of the function in the list:#!/usr/bin/env python3
results = []
def task_a():
res = {}
res['task_a'] = 1 + 1
results.append(res)def task_b():
res = {}
res['task_b'] = 2 + 2
results.append(res)def task_c(name):
res = {}
res['task_c'] = "Name: %s"%(name)
results.append(res)def run_parallel(*functions):
'''
Run functions in parallel
'''
from multiprocessing import Process
processes = []
for function in functions:
proc = Process(target=function)
proc.start()
processes.append(proc)
for proc in processes:
proc.join()if __name__ == '__main__':run_parallel(task_a(),task_b(),task_c(name="konstantinos"))
print(results)
Results:[{'task_a': 2}, {'task_b': 4}, {'task_c': 'Name: konstantinos'}]
