Python: run functions in parallel with a multiprocessing wrapper function
One common way to runfunctions in parallel with Python is to use the multiprocessing module which is powerfull, it has many options to…
One common way to run functions in parallel with Python is to use the multiprocessing module which is powerful, it has many options to configure and a lot of things to tweak.
But what about if we want just a very simple functionality like running a number of functions in parallel and nothing else?
We can use the following wrapper function for this:def run_parallel(*functions):
'''
Run functions in parallel
'''
from multiprocessing import Process
processes = []
for function in functions:
proc = Process(target=function)
proc.start()
processes.append(proc)
for proc in processes:
proc.join()
This function allows us to input as parameter any number of functions with or without their parameters and executed in parallel.
Example#!/usr/bin/env python3def task_a():
print('this is task a')def task_b():
print('this is task b')def hello(msg):
print('Hello: %s'%(msg))def run_parallel(*functions):
'''
Run functions in parallel
'''
from multiprocessing import Process
processes = []
for function in functions:
proc = Process(target=function)
proc.start()
processes.append(proc)
for proc in processes:
proc.join()if __name__ == '__main__':
run_parallel(task_a(),task_b(),hello(msg="konstantinos"))
Executing this will bring the following results:this is task a
this is task b
Hello: konstantinos
This is nice and simple, but how we will get results back from the functions?
To do this we need to use a shared variable like a list and append the results of the function in the list:#!/usr/bin/env python3
results = []
def task_a():
res = {}
res['task_a'] = 1 + 1
results.append(res)def task_b():
res = {}
res['task_b'] = 2 + 2
results.append(res)def task_c(name):
res = {}
res['task_c'] = "Name: %s"%(name)
results.append(res)def run_parallel(*functions):
'''
Run functions in parallel
'''
from multiprocessing import Process
processes = []
for function in functions:
proc = Process(target=function)
proc.start()
processes.append(proc)
for proc in processes:
proc.join()if __name__ == '__main__':run_parallel(task_a(),task_b(),task_c(name="konstantinos"))
print(results)
Results:[{'task_a': 2}, {'task_b': 4}, {'task_c': 'Name: konstantinos'}]
