Cheatography
https://cheatography.com
A series of libraries with useful functions, classes and decorators which greatly improve simple computations if used wisely.
This is a draft cheat sheet. It is a work in progress and is not finished yet.
A word of advice
Before jumping into parallelization and code optimization, try to understand what you are trying to achieve, in comparison with what you've written. Is your code not clean enough? Perhaps untangling it and using of other libraries will help. Does your code make many read/write operations? Perhaps parallelizing in threads will speed it up. Does your code perform the same task over and over again, with minimal changes? Perhaps using parallelization in processes will aid you.
Perhaps, perhaps, perhaps... think about what you have just coded. |
Concurrent processes
Main benefit |
Parallelize CPU-bound tasks, independent from each other. |
Import class |
from concurrent.futures import ProcessPoolExecutor
|
Run parallel tasks |
with ProcessPoolExecutor() as executor:
results = executor.map(func, list)
|
Submit a specific task to a core |
with ProcessPoolExecutor() as executor:
results = executor.submit(func)
|
Option (number of cores) |
ProcessPoolExecutor(max_workers=10)
|
Extra function arguments can be Included before mapping the function in the executor. In order to do so, we need to create a partial version of the function with partial_func = functools.partial(func, a=a, b=b, ...)
.
Concurrent threads
Main benefit |
Parallelize I/O-bound tasks, independent from each other. |
Import class |
from concurrent.futures import ThreadPoolExecutor
|
Run parallel tasks |
with ThreadPoolExecutor() as executor:
results = executor.map(func, list)
|
Submit a specific task to a core |
with ThreadPoolExecutor() as executor:
results = executor.submit(func)
|
Option (number of cores) |
ThreadPoolExecutor(max_workers=10)
|
Extra function arguments can be Included before mapping the function in the executor. In order to do so, we need to create a partial version of the function with partial_func = functools.partial(func, a=a, b=b, ...)
.
|
|
Numba library
Main benefit |
Useful decorators which compile the function the first time it is used, speeding it up in subsequent runs. |
Importing decorators |
from numba import jit, njit
|
Using jit |
@jit
def function(...):
...
|
Using njit |
@njit
def function(...):
...
|
This module is very limited, in the sense that only basic and/or numpy operations and classes can speed up. Neither scipy nor pandas or networkx can be improved. If jit
is used, then those functions are treated as usual. If njit
is used, then the code will fail to compile as it won't know what to do with them.
|
