Python Packaging
Demystified
Tutorial - EuroPython 2021 - 🌐 edition
by Bernat Gabor /
@gjbernat /
Bloomberg
https://bit.ly/pkg-eu-py-21
Who am I?
- software engineer at Bloomberg (data ingestion and quality control)
- OSS contributor gaborbernat@github - https://bernat.tech
- member of the Python Packaging Authority
- 😎 parent to two Yorkshire Terriers (Silky & Junior)
Photo by Bernat Gabor - All rights reserved
Maintainer (author *) of Python OSS projects
| Name | Release | Activity | Popularity |
|---|---|---|---|
| virtualenv * |
|
|
|
| tox * |
|
|
|
| build * |
|
|
|
| pipx |
|
|
|
| pytest-print * |
|
|
|
| tox-conda |
|
|
|
| attrs-strict |
|
|
|
| sphinx-argparse-cli * |
|
|
|
| retype |
|
|
|
| tox-ini-fmt * |
|
|
|
Chapter 1
Distributing Python code
Python code is easy to read (and write) 😁
Shipping and running it to other machines is not 😭
people's faces when they gaze into Python packaging the first time
The state of Python packaging
From Python Packaging Authority point of view
- Focus on Python only (think pypi.org and pip)
- not conda
- not your OS packaging system (DPKG/RPM/etc.)
A demo problem
Calculating 𝜋
𝜋 is the ratio of a circle's circumference to its diameter
Calculating 𝜋
The Gregory-Leibniz Series
\[\begin{aligned} \frac{\pi}{4} & = 1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \cdots \end{aligned} \]
\[\begin{aligned} \pi & = \left(\frac{1}{1} - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} -
\cdots \right) \cdot 4 \end{aligned} \]
Not the fastest way (requires 300 iteration for first two precision)
Calculating 𝜋
The Gregory-Leibniz Series
\[\begin{aligned} \pi & = \left(\frac{1}{1} - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} -
\cdots \right) \cdot 4 \end{aligned} \]
# pi_approx.py
from math import pi
def approximate_pi(iteration_count: int) -> float:
sign, result = 1, 0.0
for at in range(iteration_count):
result += sign / (2 * at + 1)
sign *= -1
return result * 4
if __name__ == "__main__":
approx_1, approx_2 = approximate_pi(300), approximate_pi(301)
print(f"approx 300 {approx_1} with diff {approx_1 - pi}")
print(f"approx 301 {approx_2} with diff {approx_2 - pi}")
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
Back to packaging
How to make it available on another machine?
There are two ways to use the code:from within Python (the developer way)
$ python
Python 3.9.4 (default, Apr 12 2021, 09:53:36)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
>>> from pi_approx import approximate_pi
>>> approximate_pi(300)
3.1382593295155914
from the command line (as an executable - the end-user way)
$ python ./pi_approx.py
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
How to make it available on another machine?
There are two ways to use the code:from within Python (the developer way) - library
$ python
Python 3.9.4 (default, Apr 12 2021, 09:53:36)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
>>> from pi_approx import approximate_pi
>>> approximate_pi(300)
3.1382593295155914
from the command line (as an executable - the end user way) - application
$ python ./pi_approx.py
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
How to make it available on another machine?
There are two ways to use the code:from within Python (the developer way) - library - expose the source within a Python interpreter
$ python
Python 3.9.4 (default, Apr 12 2021, 09:53:36)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
>>> from pi_approx import approximate_pi
>>> approximate_pi(300)
3.1382593295155914
from the command line (as an executable - the end user way) - application - expose the logic
$ python ./pi_approx.py
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
Application mode
Allow user to run the code - closed system
-
For example, a Python GUI, CLI or web application needs:
- A Python interpreter environment
- Dependencies - exact version - may be obfuscated
- The code - may be obfuscated
- Entry point - how to invoke the execution
Library mode
Allow user to interact with the code - open system
-
Needs:
- A Python interpreter environment
- Dependencies - compatible versions
- The code
How to package a library
What is a library?
-
code that can be
- imported and executed
- from within a python interpreter
Python interpreter types
-
global or system one (as installed by the operating system's package manager)
$ python3.9 -c 'import sys; print(sys.executable)' C:\Users\gabor\AppData\Local\Programs\Python\Python39\python.exe -
virtual environments - use system's Python executable and standard library, but isolate from other
libraries
-
virtual environment - via venv
$ python3.9 -m venv env $ .\env\Scripts\python.exe -c 'import sys; print(sys.executable)' C:\Users\gabor\git\env\Scripts\python.exe -
virtual environment - via virtualenv
$ python3.9 -m virtualenv env created virtual environment CPython3.9.4.final.0-64 in 475ms creator CPython3Windows(dest=C:\Users\gabor\git\env, clear=True, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\...\pypa\virtualenv) added seed packages: pip==21.1, setuptools==56.0.0, wheel==0.36.2 activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator $ .\env\Scripts\python.exe -c 'import sys; print(sys.executable)' C:\Users\gabor\git\env\Scripts\python.exe
-
virtual environment - via venv
Comparing virtual environment creators
| virtualenv | venv | |
|---|---|---|
| Installation | yes, third-party package | no, standard library |
| Update | anytime via pip | with the interpreter |
| Python version | 2.7 and 3.4 or later | 3.3 or later |
| Python flavour | PyPy, CPython | self-support for 3.3+ |
| Configurability | CLI + environment variables + per user config file | CLI |
| Extensibility | via plugins that can be installed alongside | by extending the base class |
| Performance | by using the cached seed mechanism <500ms | 2 seconds+ per invocation |
| API | rich: e.g., lookup executable, site-package dir | limited as per PEP-405 |
Performance difference
-
measured via
hyperfine
on Windows 10 with M2 disk (on UNIX around 2.5 faster)
-
venv
$ hyperfine 'python3.9 -m venv env --clear' Benchmark #1: python3.9 -m venv env --clear 0 Time (mean ± σ): 5.232 s ± 0.173 s [User: 3.0 ms, System: 9.0 ms] 0 Range (min … max): 4.990 s … 5.520 s 10 runs -
virtualenv
$ hyperfine 'python3.9 -m virtualenv env --clear' Benchmark #1: python3.9 -m virtualenv env --clear 0 Time (mean ± σ): 672.2 ms ± 26.9 ms [User: 4.4 ms, System: 8.5 ms] 1 Range (min … max): 645.4 ms … 740.1 ms 10 runs
Use virtualenv when can, fall back to venv
pyflow - a fresh take on virtual environments
$ pyflow init
Please enter the Python version for this project: (eg: 3.8)
3.9
Created `pyproject.toml`
🐍 Setting up Python...
$ pyflow install rich
Found lockfile
⬇ Installing colorama 0.4.4 ...
⬇ Installing rich 10.1.0 ...
⬇ Installing pygments 2.9.0 ...
Added a console script: pygmentize
⬇ Installing typing_extensions 3.10.0.0 ...
⬇ Installing commonmark 0.9.1 ...
Added a console script: cmark
Installation complete
$ pyflow init
ls .
__pypackages__ pyflow.lock pyproject.toml
$ tree .
__pypackages__
└── 3.9
├── bin
│ ├── cmark
│ └── pygmentize
└── lib
├── colorama
│ ├── __init__.py
│ ├── ansi.py
│ ├── ansitowin32.py
...
The library discovery system
-
How does Python knows if a library is available (import-able)?
-
It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
$ virtualenv env $ .\env\Scripts\python.exe Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import setuptools >>> setuptools <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'> -
setuptools here is a module - a dictionary of name to python objects groupped under a given name
>>> dir(setuptools)[-3:] ['string_types', 'version', 'windows_support']
The library discovery system
-
How does Python knows if a library is available (import-able)?
-
It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
$ virtualenv env $ .\env\Scripts\python.exe Python 3.9.4 (tags/v3.9.4:1f2e308, Apr 6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import setuptools >>> setuptools <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'> -
setuptools here is a module - a dictionary of name to python objects groupped under a given name
>>> dir(setuptools)[-3:] ['string_types', 'version', 'windows_support'] >>> setuptools.setup <function setup at 0x0000018C1A9660D0>
The library discovery system
-
How does Python knows if a library is available (import-able)?
-
It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
$ virtualenv env $ .\env\Scripts\python.exe Python 3.9.4 (tags/v3.9.4:1f2e308, Apr 6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import setuptools >>> setuptools <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'> -
setuptools here is a module - a dictionary of name to python objects groupped under a given name
>>> dir(setuptools)[-3:] ['string_types', 'version', 'windows_support'] >>> setuptools.setup <function setup at 0x0000018C1A9660D0> >>> setuptools.string_types (<class 'str'>,)
The library discovery system
-
How does Python knows if a library is available (import-able)?
-
It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
$ virtualenv env $ .\env\Scripts\python.exe Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import setuptools >>> setuptools <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'> -
setuptools here is a module - a dictionary of name to python objects groupped under a given name
>>> dir(setuptools)[-3:] ['string_types', 'version', 'windows_support'] >>> setuptools.setup <function setup at 0x0000018C1A9660D0> >>> setuptools.string_types (<class 'str'>,) >>> setuptools.version <module 'setuptools.version' from 'C:\\Users\\gabor\\AppData\\...\\Python39\\lib\\site-packages\\setuptools\\version.py'>
from imports
A from import is just syntactic sugar for a module import and then an assigment
>>> from setuptools import setup
>>> import setuptools
>>> setup = setuptools.setup
>>> del locals()["setuptools"]
not all modules are equal
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> import os
>>> os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
not all modules are equal
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
builtin module
>>> import os
>>> os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
not all modules are equal
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> import os
>>> os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
standard library module
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
not all modules are equal
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> import os
>>> os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
standard library dynamic load module
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
not all modules are equal
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> import os
>>> os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
global site-package module -> python -m pip install
>>> import pep517
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
not all modules are equal
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> import os
>>> os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
user site-package module -> python -m pip install --user
the import system - importers
>>> import sys
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for importer in sys.meta_path:
... print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>
the import system - built-in
>>> import sys
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for importer in sys.meta_path:
... print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>
the import system - frozen
freeze tool, pyinstaller, etc.
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for importer in sys.meta_path:
... print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>
the import system - pathfinder
>>> import os
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for importer in sys.meta_path:
... print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>
the import system - pathfinder
>>> import os
<class '_frozen_importlib_external.PathFinder'>
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
the import system - pathfinder
>>> import os
<class '_frozen_importlib_external.PathFinder'>
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
the import system - pathfinder
>>> import os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
<class '_frozen_importlib_external.PathFinder'>
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9 standard library directory
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
the import system - pathfinder
>>> import _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
<class '_frozen_importlib_external.PathFinder'>
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extensions
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
the import system - pathfinder
>>> import pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
<class '_frozen_importlib_external.PathFinder'>
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages user site-package
/opt/python3.9/lib/python3.9/site-packages
the import system - pathfinder
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
<class '_frozen_importlib_external.PathFinder'>
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages global site-package
What is a virtual environment? - PEP-405
-
isolated Python environment
- access to its own site-package
-
shares access to host
- built-ins
- standard library
- optionally isolated from the system site-package
System python paths
>>> import sys
>>> sys.executable
/opt/python3.9/bin/python
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9 standard library
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extension
/Users/bernat/.local/lib/python3.9/site-packages user site-package
/opt/python3.9/lib/python3.9/site-packages global site-package
virtual environment python paths - no global access
>>> import sys
>>> sys.executable
/Users/bernat/env/bin/python
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9 standard library
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extension
/Users/bernat/env/lib/python3.9/site-package OWN site-package
virtual environment python paths - global access
>>> import sys
>>> sys.executable
/Users/bernat/env/bin/python
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for path in sys.path:
... print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9 standard library
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extension
/Users/bernat/env/lib/python3.9/site-package OWN site-package
/Users/bernat/.local/lib/python3.9/site-packages user site-package
/opt/python3.9/lib/python3.9/site-packages global site-package
site-packages == purelib + platlib
Python defines two folders as python library locations, often set to same
Python 3.9.4 (default, Apr 6 2021, 00:00:00)
[GCC 11.0.1 20210324 (Red Hat 11.0.1-0)] on linux
>>> import sys, sysconfig
>>> sysconfig.get_path('platlib')
'/env/lib64/python3.9/site-packages'
>>> sysconfig.get_path('purelib')
'/env/lib/python3.9/site-packages'
>>> print('\n'.join(sys.path))
/usr/lib64/python39.zip
/usr/lib64/python3.9
/usr/lib64/python3.9/lib-dynload
/env/lib64/python3.9/site-packages
/env/lib/python3.9/site-packages
What is an installed library made of?
-
the code itself - a module or package
❯ lsd C:\Users\gabor\git\env\lib\site-packages\setuptools --tree setuptools <- package ├── __init__.py <- module ... ├── build_meta.py ├── cli-32.exe<- resource file ... └── windows_support.py -
entry points - console/GUI generated alongside the Python executable
❯ lsd --tree .\env\Scripts Scripts ├── pip-3.9.exe ├── pip.exe ├── pip3.9.exe └── pip3.exe -
metadata declaring the package by/for the installer - mostly as defined by
PEP-427
❯ lsd C:\Users\gabor\git\env\lib\site-packages\setuptools-56.0.0.dist-info --tree setuptools-56.0.0.dist-info ├── dependency_links.txt ├── entry_points.txt ├── INSTALLER ├── LICENSE ├── METADATA ├── RECORD ├── top_level.txt └── WHEEL
What would it mean to be able to run our pi logic?
$ python
Python 3.9.4 (default, Apr 12 2021, 09:53:36)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
>>> from pi_approx import approximate_pi
>>> approximate_pi(300)
3.1382593295155914
# pi_approx.py
from math import pi
def approximate_pi(iteration_count: int) -> float:
sign, result = 1, 0.0
for at in range(iteration_count):
result += sign / (2 * at + 1)
sign *= -1
return result * 4
❯ lsd .\env\Lib\site-packages\pi_approx* --tree
pi_approx.py
pi_approx-1.0.dist-info
├── direct_url.json
├── INSTALLER
├── LICENSE
├── METADATA
├── RECORD
├── REQUESTED
└── WHEEL
What is a python project made of?
project
└── pi_approx.py business logic file(s)
What is a python project made of?
project
├── pi_approx.py
└── test_pi_approx.py test file(s)
What is a python project made of?
project
├── pi_approx.py
├── test_pi_approx.py
├── LICENSE project management file(s)
├── README.md
└── .github
└── workflows
└── check.yaml
What is a python project made of?
project
├── pi_approx.py
├── test_pi_approx.py
├── LICENSE
├── README.md
├── .github
│ └── workflows
│ └── check.yaml
└── pyproject.toml project packaging file(s)
What is a python project made of?
-
layout on our development machine
project ├── pi_approx.py ├── test_pi_approx.py ├── LICENSE ├── README.md ├── .github │ └── workflows │ └── check.yaml └── pyproject.toml -
layout on their machine - the run environment
❯ lsd .\env\Lib\site-packages\pi_approx* --tree pi_approx.py pi_approx-1.0.dist-info ├── direct_url.json ├── INSTALLER ├── LICENSE ├── METADATA ├── RECORD ├── REQUESTED └── WHEEL
What is a python project made of?
-
layout on our development machine
project ├── pi_approx.py ├── test_pi_approx.py ├── LICENSE ├── README.md ├── .github │ └── workflows │ └── check.yaml └── pyproject.toml -
layout on their machine - the run environment
❯ lsd .\env\Lib\site-packages\pi_approx* --tree pi_approx.py pi_approx-1.0.dist-info ├── direct_url.json ├── INSTALLER ├── LICENSE ├── METADATA ├── RECORD ├── REQUESTED └── WHEEL
Shipping code to another machine
Photo by Bernat Gabor - All rights reserved
Package types - (sdist)
source distribution - resembles what we have on the developer machine
- all the files needed to install (and test) a package from raw source
-
source tree minus
- project management files
- maintainer files
- has business logic, packaging, tests
project
├── pi_approx.py
├── test_pi_approx.py
├── LICENSE
├── README.md
├── .github
│ └── workflows
│ └── check.yaml
└── pyproject.toml
Package types - wheel
resembles what we want on the target machine
- the installed binary files with some metadata
-
source tree minus
- project management files
- maintainer files
- tests
- packaging files
❯ lsd .\env\Lib\site-packages\pi_approx* --tree
pi_approx.py
pi_approx-1.0.dist-info
├── direct_url.json
├── INSTALLER
├── LICENSE
├── METADATA
├── RECORD
├── REQUESTED
└── WHEEL
sdist or wheel
- We always install wheel - if we have a sdist, we first build a wheel from it
Breaking down shipping a library
Generate a package
Consume a package
Build a package
buckle up for a bit of history
Photo by Bernat Gabor - All rights reserved
History of packaging
-
2000 - Python 1.6 - distutils - setup.py
introduced:
- used as invocation interface
- configuration as Python code
- 2004 - setuptools quickly became de facto standard
- 2014 - wheels - PEP-427
-
2015 - flit - declarative over dynamic
(avoid arbitrary code run)
- easier to understand
- harder to get wrong
- 2018 - poetry - single tool to rule them all
How does a build used to work
navigate to the developer source tree and invoke the following command(s)
python setup.py sdist # build a source distribution
python setup.py bdist_wheel # build a wheel
Problem 1 - build dependencies
-
setuptools pulled in from the builder Python
python setup.py sdist python setup.py install -
cryptic error if build dependencies are missing
File "setup_build.py", line 99, in run from Cython.Build import cythonize ImportError: No module named Cython.Build - or even worse, no error - incorrect version
- or even worse - user can hijack setuptools and try to inject their own code
Problem 1 - build dependencies
-
Idea: declarative build environment provision
python setup.py bdist_wheel -
create temporary folder
-
create an isolated Python environment
python -m venv our_build_env -
install build dependencies
our_build_env/bin/python -m pip dep1 dep2
-
create an isolated Python environment
-
generate a wheel that we can simply install afterwards
our_build_env/bin/python setup.py bdist_wheel
Problem 1 - build dependencies
- PEP-518
-
specify deps via pyproject.toml - must be at the root of your project
[build-system] requires = [ "setuptools >= 40.8.0", "wheel >= 0.30.0", "cython >= 0.29.4", ] -
TOML - Tom's Obvious, Minimal Language.
- standardized
- not overly complex
- does not conflict with other files out there - setup.cfg
Problem 1 - build dependencies
- PEP-518
- build back-end - performs the package generation in an isolated environment
-
build front-end - prepares the isolated environment, and invokes the back-end
- build - unified CLI for back-ends
- poetry, hatch, trampolim, pdm
- pip - PEP-518 support with version 18.0
- tox - PEP-518 support with version 3.3.0
Problem 2 - diversity - better user interface
-
PEP-518 involves still calling to setup.py
our_build_env/bin/python setup.py bdist_wheel - setup.py allows arbitrary python code to run - cannot change due to backward compatibility
- flit implementation still needed generating setup.py and adhering to those limitations
Problem 2 - diversity - better user interface
- PEP-517
-
So instead of:
our_build_env/bin/python setup.py bdist_wheel -
Allow build back-ends to expose invocation end-points:
[build-system] requires = ["flit"] build-backend = "flit.api:main" -
And the build front-end will now do:
import flit.api backend = flit.api.main backend.build_wheel() # build wheel via backend.build_sdist() # build source distribution via
Problem 2 - diversity - better user interface
How to build a package?
- Pick your build back-end
-
Use pyproject.toml to specify your
- build dependencies
- build back-end interface
- Follow the docs of your back-end on how to configure it
Installing tools - pipx
-
Install pipx by installing pipx-in-pipx
python -m pip install pipx-in-pipx - Make pipx path available on your PATH - ~/.local/bin
-
Install your tools via pipx
pipx install build pipx install flit
Building a package with flit
Let's package our 𝜋 approximator with flit
Speed it up - cython
Adding Cython speed-up
- Move it to setuptools
- Add cython
Cython speed-up
❯ .\venv\Scripts\python.exe -m timeit -s 'from pi_approx import approximate_pi' 'approximate_pi(1_000_000)'
2 loops, best of 5: 105 msec per loop
❯ .\venv\Scripts\python.exe -m timeit -s 'from pi_approx_cy import approximate_pi' 'approximate_pi(1_000_000)'
5 loops, best of 5: 68.4 msec per loop
Upload a package
-
Historically it was (deprecated)
python setup.py upload -
Today we use twine
twine upload dist/* -
Though some tools do abstract this away:
flit publish poetry publish
Discover and download a package
- For human discovery, visit PyPI
-
Machines use the Simple Repository API ~ PEP-503,
e.g., build
pip download 'build<0.3' --no-deps -d wheels
Install a package
- pip
- Programmatic API - installer
Photo by Bernat Gabor - All rights reserved
Testing a library
-
create virtual environment
virtualenv venv -
build (in a PEP-517/PEP-518 isolated environment) and install the package
pyproject-build --sdist . .\venv\Scripts\pip install .\dist\pi_approx-1.0.0.tar.gz # or the shorter form # pip install . -
install test dependencies
.\venv\Scripts\pip.exe install 'pytest>6' -
run test
.\venv\Scripts\pytest tests
tox
# tox.ini
[tox]
envlist =
py39
isolated_build = true
[testenv]
deps =
pytest>6
commands =
pytest tests
tox
[tox]
envlist =
py39
py38
flake8
isolated_build = true
[testenv]
deps =
pytest>6
commands =
pytest tests
[testenv:flake8]
deps =
flake8==3.9.1
commands =
flake8 pi_approx.py tests
Best practices
- Always publish both a wheel and a sdist
- Ship the test suite with your sdist
Creating an application
Photo by Bernat Gabor - All rights reserved
Application vs. library
-
Library
- runs with latest compatible dependencies
- test with latest compatible dependencies
- need to expose all modules/resources as they are
- needs to be wheel installed
-
Application
- runs with given set of dependencies
- test with given set of dependencies
- need to expose only the entry points
- ideally just copy and execute
Application as a library
- Do the same as we learned in previous chapter
-
Downsides
- Users need to understand virtual environments
- Entry-points must be defined separately
- Running them might require path mangling
- Dependencies are not pinned to a given version
- Is still usable though if you can solve these in otherways - e.g., build docker images
zipapp
- PEP-441 - Improving Python ZIP Application Support
- Package python logic into a ZIP archive that can be executed by passing it to an interpreter
-
For example:
curl https://bootstrap.pypa.io/virtualenv.pyz -o virtualenv.pyz python3.9 virtualenv.pyz --help - provides a way to organize python files into a ZIP file and define a unified entry point for the ZIP file
zipapp package pi approximator
py -m zipapp .\pi-approx-cy -m "pi_approx:run"
def run() -> None:
parser = argparse.ArgumentParser(description="Calculate pi approximation.")
parser.add_argument("it", metavar="N", type=int, help="iteration count")
ns = parser.parse_args()
result = approximate_pi(ns.it)
print(f"result is {result}")
pex
-
Create self-contained applications - UNIX only (no Windows support)
pex -c virtualenv -o virtualenv.pex virtualenv python virtualenv.pex --help - Python 2/3 support
-
Automatically discovers dependencies and packages
0 1980-01-01 00:00 .bootstrap/ 0 1980-01-01 00:00 .bootstrap/pex/ 401 1980-01-01 00:00 .bootstrap/pex/__init__.py 235 1980-01-01 00:00 .bootstrap/pex/__main__.py 483 1980-01-01 00:00 .bootstrap/pex/attrs.py ... 0 1980-01-01 00:00 .deps/ 0 1980-01-01 00:00 .deps/appdirs-1.4.4-py2.py3-none-any.whl/ 24720 1980-01-01 00:00 .deps/appdirs-1.4.4-py2.py3-none-any.whl/appdirs.py -
Marks created package executable
# uses #!/usr/bin/env python virtualenv.pex --help - As entry-point, can select either main or console entry points
- Note C-extension dependencies make the generated package not compatible with other platforms
shiv
- Python 3.6+ support only but
-
Create self-contained applications - UNIX + Windows support
shiv -c virtualenv -o virtualenv.shiv virtualenv python virtualenv.shiv --help -
Automatically discovers dependencies and packages
Archive: ./virtualenv.shiv Length Date Time Name --------- ---------- ----- ---- 24720 04/05/2021 22:37 site-packages/appdirs.py 8 04/05/2021 22:37 site-packages/appdirs-1.4.4.dist-info/top_level.txt ... 65 04/05/2021 22:37 site-packages/virtualenv/version.py 2878 04/05/2021 22:37 _bootstrap/environment.py 2063 04/05/2021 22:37 _bootstrap/filelock.py 1762 04/05/2021 22:37 _bootstrap/interpreter.py 8728 04/05/2021 22:37 _bootstrap/__init__.py 377 04/05/2021 22:37 environment.json 65 04/05/2021 22:37 __main__.py - Note C-extension dependencies make the generated package not compatible with other platforms
pyinstaller
- Packages not just the entry point, app and dependencies, but also the interpreter
-
generates a single binary executable
❯ pyinstaller.exe -F .\pi_approx.py ❯ .\dist\pi_approx.exe 10000000 result is 3.1415925535897915
use pyinstaller to generate an executable for pi approximator
zipapp with dynamic dependency
virtualenv - case study
- some dependencies are OS specific
- some dependencies are Python version specific
install_requires =
appdirs>=1.4.3,<2
distlib>=0.3.0,<1
filelock>=3.0.0,<4
six>=1.9.0,<2 # keep it >=1.9.0 as it may cause problems on LTS platforms
importlib-metadata>=0.12,<2;python_version<"3.8"
importlib-resources>=1.0;python_version<"3.7"
pathlib2>=2.3.3,<3;python_version < '3.4' and sys.platform != 'win32'
zipapp with dynamic dependencies
| 3.9 | 3.8 | 3.7 | 3.6 | 3.5 | 3.4 | 2.7 | |
|---|---|---|---|---|---|---|---|
| appdirs | 1.4.4 | 1.4.4 | 1.4.4 | 1.4.4 | 1.4.4 | 1.4.4 | 1.4.4 |
| distlib | 0.3.1 | 0.3.1 | 0.3.1 | 0.3.1 | 0.3.1 | 0.3.1 | 0.3.1 |
| filelock | 3.0.12 | 3.0.12 | 3.0.12 | 3.0.12 | 3.0.12 | 3.0.12 | 3.0.12 |
| six | 1.15.0 | 1.15.0 | 1.15.0 | 1.15.0 | 1.15.0 | 1.15.0 | 1.15.0 |
| importlib-metadata | 1.7.0 | 1.7.0 | 1.7.0 | 1.1.3 | 1.1.3 | ||
| importlib-resources | 2.0.1 | 2.0.1 | 1.0.2 | 1.0.2 | |||
| pathlib2 | 2.3.5 | 2.3.5 |
zipapp content structure
.
├── __main__.py
├── virtualenv
│ └ ...
│── virtualenv-20.0.25.dist-info
│ ├── entry_points.txt
│ ├── LICENSE
│ ├── METADATA
│ ├── RECORD
│ ├── top_level.txt
│ ├── WHEEL
│ └── zip-safe
├── __virtualenv__
│ ├── appdirs-1.4.4-py2.py3-none-any
│ ├── configparser-4.0.2-py2.py3-none-any
│ ├── contextlib2-0.6.0.post1-py2.py3-none-any
│ ├── distlib-0.3.0-py3-none-any
│ ├── filelock-3.0.12-py3-none-any
│ ├── importlib_metadata-1.1.3-py2.py3-none-any
│ ├── importlib_metadata-1.6.1-py2.py3-none-any
│ ├── importlib_resources-1.0.2-py2.py3-none-any
│ ├── importlib_resources-2.0.1-py2.py3-none-any
│ ├── pathlib2-2.3.5-py2.py3-none-any
│ ├── scandir-1.10.0-cp38-cp38-macosx_10_14_x86_64
│ ├── six-1.15.0-py2.py3-none-any
│ ├── typing-3.7.4.1-py2-none-any
│ ├── typing-3.7.4.1-py3-none-any
│ ├── zipp-1.2.0-py2.py3-none-any
│ └── zipp-3.1.0-py3-none-any
├── distributions.json
└── modules.json
zipapp content structure
.
├── __main__.py
├── virtualenv
│ └ ...
│── virtualenv-20.0.25.dist-info
│ ├── entry_points.txt
│ ├── LICENSE
│ ├── METADATA
│ ├── RECORD
│ ├── top_level.txt
│ ├── WHEEL
│ └── zip-safe
├── __virtualenv__
│ ├── appdirs-1.4.4-py2.py3-none-any
│ ├── configparser-4.0.2-py2.py3-none-any
│ ├── contextlib2-0.6.0.post1-py2.py3-none-any
│ ├── distlib-0.3.0-py3-none-any
│ ├── filelock-3.0.12-py3-none-any
│ ├── importlib_metadata-1.1.3-py2.py3-none-any
│ ├── importlib_metadata-1.6.1-py2.py3-none-any
│ ├── importlib_resources-1.0.2-py2.py3-none-any
│ ├── importlib_resources-2.0.1-py2.py3-none-any
│ ├── pathlib2-2.3.5-py2.py3-none-any
│ ├── scandir-1.10.0-cp38-cp38-macosx_10_14_x86_64
│ ├── six-1.15.0-py2.py3-none-any
│ ├── typing-3.7.4.1-py2-none-any
│ ├── typing-3.7.4.1-py3-none-any
│ ├── zipp-1.2.0-py2.py3-none-any
│ └── zipp-3.1.0-py3-none-any
├── distributions.json
└── modules.json
Use import hooks within zipapp
- Python 3: PEP-451 - A ModuleSpec Type for the Import System
- Python 2: PEP-302 - New import hooks