MultiProcessing Single Function (mpsf)

Process multiple data points at the same time.

Note

This is a personal project made by an amateur programmer and unlikely to be maintained outside of the interest of the author.

Quick use

from mpsf import smart_multi_proc

def my_func(x):
   print(x)

data = (i for i in range(10)) # dummy data as iterator
smart_multi_proc(my_func, data)

Set the number of my_func functions to be spawned:

Set worker count

from mpsf import smart_multi_proc

def my_func(x):
   print(x)

data = (i for i in range(10)) # dummy data as iterator
n_workers = 10
smart_multi_proc(my_func, data, n_processes=n_workers)

All data in the data iterator is not loaded in memory immedietly, but only loads a set amount of data points into a queue waiting for a worker to grap it. To control the maximum queue size (memory use) and the delay between each time the queue is filled up do:

Set queue size and delay

from mpsf import smart_multi_proc

def my_func(x):
   print(x)

data = (i for i in range(10)) # dummy data as iterator
n_workers = 10
max_queue_size = 1000
queue_fill_delay = 10 # time in seconds
smart_multi_proc(
   my_func,
   data,
   n_processes=n_workers,
   fill_pause=queue_fill_delay,
   max_queue_size = max_queue_size
)

In case workers can empty the queue faster than it can be filled up a 5 second delay is set before it asks again. The value can be changed:

Set request delay for workers

from mpsf import smart_multi_proc

def my_func(x):
   print(x)

data = (i for i in range(10)) # dummy data as iterator
n_workers = 10
max_queue_size = 1000
queue_fill_delay = 10 # time in seconds
request_delay = 1 # time in seconds
smart_multi_proc(
   my_func,
   data,
   n_processes=n_workers,
   request_delay=request_delay,
   fill_pause=queue_fill_delay,
   max_queue_size = max_queue_size
)