The GIL Detector

09 August 2011

Ever wonder if your flavor of python has a Global Interpreter Lock? The recipe gil_detector.py checks that. In this day and age CPython is still core-locked; though we are seeing improvements thanks to Antoine Pitrou and other great people of the Python community. Wanting to measure just how bad the problem is - my way of looking at it was through a number I call the "python effective core count".

Python Effective Core Count

How many cores does python see? If you bought a quad core, how many cores can each python process utilize? Measuring how long it takes to complete a given amount of work, W, then measuring how long it takes for python to run 2W using 2 threads, 3W on 3 threads, etc. The script gil_detector.py calculates: effective_cpus = amount_of_work / (time_to_finish / baseline) Where the baseline is the time_to_finish for 1 work unit. E.g. if it took the same amount of time to finish 4W (amount_of_work = 4) on 4 threads as it took 1W on 1 thread - python is utilizing 4 cores.

Results

I recommend reading the whole output to see the exact numbers.

Implementation	Effective Core Count
Jython 2.5.2	3.8/4 cores
IronPython 2.7	3.2/4 cores
PyPy-1.5	1.0/4 cores
Stackless Python 3.2	1.0/4 cores
CPython 3.2	1.0/4 cores
CPython 2.7	0.2/4 cores

Basically, Jython has the best multithreading with IronPython not too far behind. I know multiprocessing is really easy in python, but it's still harder. We have to solve this before 8 core CPU's become the standard at grandma's house. Those other languages (C/C++) utilize 3.9-4.0 cores of a quad core machine easily, why can't we? An honorable mention goes to PyPy which by far was the fastest to execute the benchmark (10x faster). PyPy is definitely the future of Python, hopefully they can save us all. A note about CPython 2.7 - yes that number is under 1 because adding threads on python-cpu-intensive tasks hurt performance badly on older versions of the GIL, Antoine fixed that on CPython 3.2. In my opinion the community should call this a bug and have a unittest in there yelling at us until we fix it, though it's easy for me to complain when I'm not the core-dev. Maybe I can join PyPy and see what I can do when I find some free time. Hopefully so. Edit - updated gil_detector.py as corrected at reddit.