You’re processing a large amount of data with Python, the processing
seems easily parallelizable—and it’s sloooooooow.
The obvious next step is switch to some sort of multiprocessing, or
even start processing data on a cluster so you can use multiple
machines. Obvious, but often wrong: switching straight to
multiprocessing, and even more so to a cluster, can be a very
expensive choice in the long run.
In this article you’ll learn why, as we:
- Consider two different goals for performance: faster results and
reduced hardware costs.
- See how different approaches achieve those goals.
- Suggest a better order for many situations: performance
optimization first, only then trying parallelization.