Compiling Python from Source Code

Published 18 March 2023. Nil revisions.

Finding Version-Specific Source Code

The source code for all of the released Python versions can be found on the python source releases web-page on the python.org website. For example, if one wants use Python 3.9, follow the aforelinked web-page and select the latest 3.9.x version. Navigate to the section containing the files and look for the XZ compressed source tarball.

Preliminary Setup

Throughout this guide it is assumed that the Python executables will be stored in ~/dev/pyexes. Either create these directories now:

cd ~

mkdir dev

mkdir dev/pyexes

Or, choose an alternative directory and modify the subsequent commands accordingly.

Note, in order to avoid system issues, the executables stored in this directory should never be installed to the system (i.e. added to the system path) unless one understands the intended (and unintended) effects of doing so.

Download and Extract the Source Code

Create a directory for the specific Python version to be installed:

mkdir dev/pyexes/py39

Create a temporary directory to store all the files that are downloaded and created during the configuration and build process, and then change into it:

mkdir dev/pyexes/py39/tempfiles

cd dev/pyexes/py39/tempfiles

Download the source code for the particular Python version:

curl https://www.python.org/ftp/python/3.9.16/Python-3.9.16.tar.xz --output Python-3.9.16.tar.xz

Extract the .tar.xz file:

tar -xJvf Python-3.9.16.tar.xz

Configure

Configure the build process by running configure:

Python-3.9.16/configure --prefix=/home/<username>/dev/pyexes/py39 --with-pydebug --enable-optimizations --with-lto

More information on the configure flags for the build process can be found on the configure python documentation web-page on the python.org website. Alternatively, one can run Python-3.9.16/configure --help).

Of particular import is the --prefix option which lets one set the location where the build will be installed. If one doesn't set this, the default install location will be in /usr/local which will conflict with any system Python versions already installed. The path set here must be an absolute directory name (i.e. instead of ~ one must specify /home/<username>).

The --enable-optimizations and --with-lto flags enables Profile Guide Optimization (PGO) and Link Time Optimization (LTO) which is recommended for best performance (note that these features were introduced in Python version 3.6, so these flags will not work if compiling a Python version earlier than 3.6.

Debug Builds

Setting the --with-pydebug flag creates a "pydebug" build. It turns on various extra sanity checks which help catch common issues. It is recommended to always develop under a debug build of CPython (unless one is taking performance measurements). However, this may cause compatibility issues with some Python package wheels if using Python versions prior to 3.8. It is recommended to read the Python Builds and Compatible Package Wheels article to understand how the configuration and build process may influence which wheels can be used to install packages.

Build

Build the Python executable:

make -j32

Change the number of cores passed to the -j flag to match those available on the system. Alternatively, if the version of Make being used supports it, use -j with the number omitted and Make will not limit the number of simultaneous steps.

The build process (compilation) will take anywhere from 2-10 minutes depending on the computer's hardware. Once it has completed, scroll up and there should be a line stating that the Python build finished successfully!, below this exclamation, if any optional modules were not able to be added they will be listed (e.g. _bz2, _hashlib etc.). The reason these modules failed to be added is because they rely on certain development headers being supplied by the associated libraries (e.g. the zlib library for the compression module). If one requires the functionality offered by the optional modules that were not added, then install the associated development libraries for each module using the system's package manager. A web-search will have to be carried out to find the names of the exact packages to be installed. More information can be found under the build dependencies section of the setup and building web-page on the python.org website.

If one doesn't need to install any additional packages to attain some of the functionality offered by the optional modules that were not added, skip to the Install section.

Once the packages are installed, rerun the configure command, followed by make:

Python-3.9.16/configure --prefix=/home/<username>/dev/pyexes/py39 --with-pydebug --enable-optimizations --with-lto

make -j32

Install

After running make, a working build will be created in the directory from which the command was run. This means that Python can be run from this directory and the interpreter will realise where it is being run from and use the files found in this working directory. Hence, there is technically no need to install the built copy of Python. However, to keep things tidy, one can install (i.e. copy) the necessary files from the working build to the location specified by the --prefix flag earlier:

make install

Note that if the --prefix flag wasn't specified earlier when setting up the build configuration via the configure command, then one will have to run the above command with elevated privileges (e.g. sudo make install) as it will install to /usr/local by default which is a location that requires elevated privileges to write to.

After installing, there may be a warning that the pip3 and pip3.9 scripts were installed to ~/dev/pyexes/py39/bin which is not on the system path. This can be ignored if one does not intend to use this Python version for system-related actions.

Python has now been successfully installed. All that remains is to remove leftover files.

Cleaning Temporary Files

Navigate up and out of the tempfiles directory:

cd ..

Remove the tempfiles directory which contains all the files that are no longer needed (i.e. source code, configuration files and build files):

rm -rf tempfiles

Now only four directories should remain in the --prefix location specified earlier: bin, include, lib and share. The most important one is the python3.9 executable located in the bin directory.

Test the Created Executable

Test that the Python executable was created successfully by printing its version:

bin/python3.9 --version   # Python 3.9.16

Using Different Python Versions

To use different Python versions at any time, simply call the version-specific Python executable located in the respective bin folder. For example:

~/dev/pyexes/py39/bin/python3.9

However, if one has multiple Python versions, the use of virtual environments is recommended. See the Python Virtual Environments guide for more information.