Speeding up complex existing Python code
This was a very interesting project. The client had a ‘multi objective evolutionary algorithm’ implemented in Python. It tries to find an optimal solution to a complex problem by starting with some random solutions and evolving them over a number of generations. The existing Python code returned the desired results but took too long to run, so the client asked me to speed it up.
It did not have any tests. Both the algorithm and the code were too complicated to fully learn in the time available.
To allow me to make changes without breaking it I first wrote some code which captures the output of each run, compares it with the previous output and flags up any changes. This gave me the confidence to experiment with different versions of the code.
Using a profiler I identified the most time consuming areas. I then applied some of the techniques from my ‘Benchmarking, Profiling and Optimising Your Python Code’ video. This cut out about 90% of the time, i.e. it speeded up the process by a factor of 10.
By the end of the process I had a better, albeit not yet complete, understanding of the algorithm. A methodical application of small, focussed, incremental changes yielded the results the client needed, without fundamentally changing the code structure. Because the changes were relatively small, the new code was still very familiar to the client.