Python Packaging

Demystified

Tutorial - EuroPython 2021 - 🌐 edition

by Bernat Gabor / @gjbernat / Bloomberg
https://bit.ly/pkg-eu-py-21

Who am I?

  • software engineer at Bloomberg (data ingestion and quality control)
  • OSS contributor gaborbernat@github - https://bernat.tech
  • member of the Python Packaging Authority
  • 😎 parent to two Yorkshire Terriers (Silky & Junior)

Photo by Bernat Gabor - All rights reserved

Maintainer (author *) of Python OSS projects

Name Release Activity Popularity
virtualenv * PyPI GitHub last commit PyPI - Downloads
tox * PyPI GitHub last commit PyPI - Downloads
build * PyPI GitHub last commit PyPI - Downloads
pipx PyPI GitHub last commit PyPI - Downloads
pytest-print * PyPI GitHub last commit PyPI - Downloads
tox-conda PyPI GitHub last commit PyPI - Downloads
attrs-strict PyPI GitHub last commit PyPI - Downloads
sphinx-argparse-cli * PyPI GitHub last commit PyPI - Downloads
retype PyPI GitHub last commit PyPI - Downloads
tox-ini-fmt * PyPI GitHub last commit PyPI - Downloads

Chapter 1

Distributing Python code

Python code is easy to read (and write) 😁

Shipping and running it to other machines is not 😭

people's faces when they gaze into Python packaging the first time

why
Photo by Marcus Cramer / Unsplash

The state of Python packaging

From Python Packaging Authority point of view
  • Focus on Python only (think pypi.org and pip)
  • not conda
  • not your OS packaging system (DPKG/RPM/etc.)

A demo problem

Calculating 𝜋

𝜋 is the ratio of a circle's circumference to its diameter

why
Photo by Kjoonlee on Wikipedia / Public Domain

Calculating 𝜋

The Gregory-Leibniz Series

\[\begin{aligned} \frac{\pi}{4} & = 1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \cdots \end{aligned} \]
\[\begin{aligned} \pi & = \left(\frac{1}{1} - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \cdots \right) \cdot 4 \end{aligned} \]

Not the fastest way (requires 300 iteration for first two precision)

Calculating 𝜋

The Gregory-Leibniz Series

\[\begin{aligned} \pi & = \left(\frac{1}{1} - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \cdots \right) \cdot 4 \end{aligned} \]

            # pi_approx.py
            from math import pi

            def approximate_pi(iteration_count: int) -> float:
                sign, result = 1, 0.0
                for at in range(iteration_count):
                    result += sign / (2 * at + 1)
                    sign *= -1
                return result * 4

            if __name__ == "__main__":
                approx_1, approx_2 = approximate_pi(300), approximate_pi(301)
                print(f"approx 300 {approx_1} with diff {approx_1 - pi}")
                print(f"approx 301 {approx_2} with diff {approx_2 - pi}")
            
                                    
                      approx 300 3.1382593295155914 with diff -0.0033333240742017267
                      approx 301 3.1449149035588526 with diff 0.0033222499690594987
                
            

Back to packaging

How to make it available on another machine?

There are two ways to use the code:

from within Python (the developer way)


                        $ python
                        Python 3.9.4 (default, Apr 12 2021, 09:53:36)
                        [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
                        >>> from pi_approx import approximate_pi
                        >>> approximate_pi(300)
                        3.1382593295155914
                        

from the command line (as an executable - the end-user way)


                        $  python ./pi_approx.py
                        approx 300 3.1382593295155914 with diff -0.0033333240742017267
                        approx 301 3.1449149035588526 with diff 0.0033222499690594987
                        

How to make it available on another machine?

There are two ways to use the code:

from within Python (the developer way) - library


                        $ python
                        Python 3.9.4 (default, Apr 12 2021, 09:53:36)
                        [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
                        >>> from pi_approx import approximate_pi
                        >>> approximate_pi(300)
                        3.1382593295155914
                        

from the command line (as an executable - the end user way) - application


                        $  python ./pi_approx.py
                        approx 300 3.1382593295155914 with diff -0.0033333240742017267
                        approx 301 3.1449149035588526 with diff 0.0033222499690594987
                        

How to make it available on another machine?

There are two ways to use the code:

from within Python (the developer way) - library - expose the source within a Python interpreter


                        $ python
                        Python 3.9.4 (default, Apr 12 2021, 09:53:36)
                        [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
                        >>> from pi_approx import approximate_pi
                        >>> approximate_pi(300)
                        3.1382593295155914
                        

from the command line (as an executable - the end user way) - application - expose the logic


                $  python ./pi_approx.py
                approx 300 3.1382593295155914 with diff -0.0033333240742017267
                approx 301 3.1449149035588526 with diff 0.0033222499690594987
                

Application mode

Allow user to run the code - closed system

    For example, a Python GUI, CLI or web application needs:
  • A Python interpreter environment
  • Dependencies - exact version - may be obfuscated
  • The code - may be obfuscated
  • Entry point - how to invoke the execution

Library mode

Allow user to interact with the code - open system

    Needs:
  • A Python interpreter environment
  • Dependencies - compatible versions
  • The code

How to package a library

What is a library?

    code that can be
  • imported and executed
  • from within a python interpreter

Python interpreter types

  • global or system one (as installed by the operating system's package manager)
    
                    $ python3.9 -c 'import sys; print(sys.executable)'
                    C:\Users\gabor\AppData\Local\Programs\Python\Python39\python.exe
                    
  • virtual environments - use system's Python executable and standard library, but isolate from other libraries
    • virtual environment - via venv
      
                        $ python3.9 -m venv env
        
                        $ .\env\Scripts\python.exe -c 'import sys; print(sys.executable)'
                        C:\Users\gabor\git\env\Scripts\python.exe
                       
    • virtual environment - via virtualenv
      
                        $ python3.9 -m virtualenv env
                        created virtual environment CPython3.9.4.final.0-64 in 475ms
                        creator CPython3Windows(dest=C:\Users\gabor\git\env, clear=True, no_vcs_ignore=False, global=False)
                        seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\...\pypa\virtualenv)
                          added seed packages: pip==21.1, setuptools==56.0.0, wheel==0.36.2
                        activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
        
                        $ .\env\Scripts\python.exe -c 'import sys; print(sys.executable)'
                        C:\Users\gabor\git\env\Scripts\python.exe
                       

Comparing virtual environment creators

virtualenv venv
Installation yes, third-party package no, standard library
Update anytime via pip with the interpreter
Python version 2.7 and 3.4 or later 3.3 or later
Python flavour PyPy, CPython self-support for 3.3+
Configurability CLI + environment variables + per user config file CLI
Extensibility via plugins that can be installed alongside by extending the base class
Performance by using the cached seed mechanism <500ms 2 seconds+ per invocation
API rich: e.g., lookup executable, site-package dir limited as per PEP-405

Performance difference

    measured via hyperfine on Windows 10 with M2 disk (on UNIX around 2.5 faster)
  • venv
    
                    $ hyperfine 'python3.9 -m venv env --clear'
                    Benchmark #1: python3.9 -m venv env --clear
                    0  Time (mean ± σ):      5.232 s ±  0.173 s    [User: 3.0 ms, System: 9.0 ms]
                    0  Range (min … max):    4.990 s …  5.520 s    10 runs
                   
  • virtualenv
    
                    $ hyperfine 'python3.9 -m virtualenv env --clear'
                    Benchmark #1: python3.9 -m virtualenv env --clear
                    0  Time (mean ± σ):     672.2 ms ±  26.9 ms    [User: 4.4 ms, System: 8.5 ms]
                    1  Range (min … max):   645.4 ms … 740.1 ms    10 runs
                   

Use virtualenv when can, fall back to venv

pyflow - a fresh take on virtual environments


          $ pyflow init
          Please enter the Python version for this project: (eg: 3.8)
          3.9
          Created `pyproject.toml`
          🐍 Setting up Python...
          

          $ pyflow install rich
          Found lockfile
          ⬇ Installing colorama 0.4.4 ...
          ⬇ Installing rich 10.1.0 ...
          ⬇ Installing pygments 2.9.0 ...
          Added a console script: pygmentize
          ⬇ Installing typing_extensions 3.10.0.0 ...
          ⬇ Installing commonmark 0.9.1 ...
          Added a console script: cmark
          Installation complete
          

          $ pyflow init
          ls .
          __pypackages__  pyflow.lock  pyproject.toml
          $ tree .
          __pypackages__
          └── 3.9
            ├── bin
            │  ├── cmark
            │  └── pygmentize
            └── lib
                ├── colorama
                │  ├── __init__.py
                │  ├── ansi.py
                │  ├── ansitowin32.py
          ...
          

The library discovery system

    How does Python knows if a library is available (import-able)?
  • It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
    
                    $ virtualenv env
                    $ .\env\Scripts\python.exe
                    Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
                    Type "help", "copyright", "credits" or "license" for more information.
                    >>> import setuptools
                    >>> setuptools
                    <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'>
                   
  • setuptools here is a module - a dictionary of name to python objects groupped under a given name
    >>> dir(setuptools)[-3:]
    ['string_types', 'version', 'windows_support']
    
    
    
    
    
    
    
    
                   

The library discovery system

    How does Python knows if a library is available (import-able)?
  • It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
    
                    $ virtualenv env
                    $ .\env\Scripts\python.exe
                    Python 3.9.4 (tags/v3.9.4:1f2e308, Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
                    Type "help", "copyright", "credits" or "license" for more information.
                    >>> import setuptools
                    >>> setuptools
                    <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'>
                   
  • setuptools here is a module - a dictionary of name to python objects groupped under a given name
    >>> dir(setuptools)[-3:]
    ['string_types', 'version', 'windows_support']
    
    >>> setuptools.setup
    <function setup at 0x0000018C1A9660D0>
    
    
    
    
    
    
    

The library discovery system

    How does Python knows if a library is available (import-able)?
  • It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
    
                    $ virtualenv env
                    $ .\env\Scripts\python.exe
                    Python 3.9.4 (tags/v3.9.4:1f2e308, Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
                    Type "help", "copyright", "credits" or "license" for more information.
                    >>> import setuptools
                    >>> setuptools
                    <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'>
                   
  • setuptools here is a module - a dictionary of name to python objects groupped under a given name
    >>> dir(setuptools)[-3:]
    ['string_types', 'version', 'windows_support']
    
    >>> setuptools.setup
    <function setup at 0x0000018C1A9660D0>
    
    >>> setuptools.string_types
    (<class 'str'>,)
    
    
    
    

The library discovery system

    How does Python knows if a library is available (import-able)?
  • It doesn't - "it's easier to ask for forgiveness than permission" - try to import and raise if fails
    
                    $ virtualenv env
                    $ .\env\Scripts\python.exe
                    Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
                    Type "help", "copyright", "credits" or "license" for more information.
                    >>> import setuptools
                    >>> setuptools
                    <module 'setuptools' from 'C:\\Users\\gabor\\git\\env\\lib\\site-packages\\setuptools\\__init__.py'>
                   
  • setuptools here is a module - a dictionary of name to python objects groupped under a given name
    >>> dir(setuptools)[-3:]
    ['string_types', 'version', 'windows_support']
    
    >>> setuptools.setup
    <function setup at 0x0000018C1A9660D0>
    
    >>> setuptools.string_types
    (<class 'str'>,)
    
    >>> setuptools.version
    <module 'setuptools.version' from 'C:\\Users\\gabor\\AppData\\...\\Python39\\lib\\site-packages\\setuptools\\version.py'>

from imports

A from import is just syntactic sugar for a module import and then an assigment


          >>> from setuptools import setup
          

            >>> import setuptools
            >>> setup = setuptools.setup
            >>> del locals()["setuptools"]
            

not all modules are equal

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys >>> sys <module 'sys' (built-in)>
>>> import os >>> os <module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv >>> _csv <module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml >>> pytoml <module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517 >>> pep517 <module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>

not all modules are equal

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
  <module 'sys' (built-in)>
builtin module
>>> import os >>> os <module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv >>> _csv <module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml >>> pytoml <module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517 >>> pep517 <module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>

not all modules are equal

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys >>> sys <module 'sys' (built-in)>
>>> import os >>> os <module 'os' from '/opt/python3.9/lib/python3.9/os.py'> standard library module
>>> import _csv >>> _csv <module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml >>> pytoml <module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517 >>> pep517 <module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>

not all modules are equal

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys >>> sys <module 'sys' (built-in)>
>>> import os >>> os <module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv >>> _csv <module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'> standard library dynamic load module
>>> import pytoml >>> pytoml <module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517 >>> pep517 <module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>

not all modules are equal

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys >>> sys <module 'sys' (built-in)>
>>> import os >>> os <module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv >>> _csv <module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml >>> pytoml <module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'> global site-package module -> python -m pip install
>>> import pep517 >>> pep517 <module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>

not all modules are equal

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys >>> sys <module 'sys' (built-in)>
>>> import os >>> os <module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
>>> import _csv >>> _csv <module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
>>> import pytoml >>> pytoml <module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
>>> import pep517 >>> pep517 <module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'> user site-package module -> python -m pip install --user

the import system - importers

                  
>>> import sys

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
                    
>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

                  

the import system - built-in

                  
>>> import sys

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
                    
>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

                  

the import system - frozen

freeze tool, pyinstaller, etc.

                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
                    
>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

                  

the import system - pathfinder

                  
>>> import os

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
                    
>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>
 

the import system - pathfinder

                  
>>> import os

<class '_frozen_importlib_external.PathFinder'>

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
                    
>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
 

the import system - pathfinder

                  
>>> import os

<class '_frozen_importlib_external.PathFinder'>

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
                    
>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
 

the import system - pathfinder

                  
>>> import os
<module 'os' from '/opt/python3.9/lib/python3.9/os.py'>
<class '_frozen_importlib_external.PathFinder'>

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9 standard library directory
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
 

the import system - pathfinder

                  
>>> import _csv
<module '_csv' from '/opt/python3.9/lib/python3.9/lib-dynload/_csv.cpython-39dm-darwin.so'>
<class '_frozen_importlib_external.PathFinder'>

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extensions
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages
 

the import system - pathfinder

                  
>>> import pytoml
<module 'pytoml' from '/opt/python3.9/lib/python3.9/site-packages/pytoml/__init__.py'>
<class '_frozen_importlib_external.PathFinder'>

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages user site-package
/opt/python3.9/lib/python3.9/site-packages
 

the import system - pathfinder

                  
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.9/site-packages/pep517/__init__.py'>
<class '_frozen_importlib_external.PathFinder'>
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip
/opt/python3.9/lib/python3.9
/opt/python3.9/lib/python3.9/lib-dynload
/Users/bernat/.local/lib/python3.9/site-packages
/opt/python3.9/lib/python3.9/site-packages global site-package

What is a virtual environment? - PEP-405

    isolated Python environment
  • access to its own site-package
  • shares access to host
    • built-ins
    • standard library
  • optionally isolated from the system site-package

System python paths

                  
>>> import sys
>>> sys.executable
/opt/python3.9/bin/python

                  
                  Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9 standard library
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extension

/Users/bernat/.local/lib/python3.9/site-packages user site-package
/opt/python3.9/lib/python3.9/site-packages global site-package
 

virtual environment python paths - no global access

                  
>>> import sys
>>> sys.executable
/Users/bernat/env/bin/python

                  
                  Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9 standard library
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extension
/Users/bernat/env/lib/python3.9/site-package OWN site-package


 

virtual environment python paths - global access

                  
>>> import sys
>>> sys.executable
/Users/bernat/env/bin/python

                  
                  
Python 3.9.4 (default, Apr 10 2021, 15:31:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.9/lib/python39.zip standard library zipped
/opt/python3.9/lib/python3.9 standard library
/opt/python3.9/lib/python3.9/lib-dynload standard library c-extension
/Users/bernat/env/lib/python3.9/site-package OWN site-package
/Users/bernat/.local/lib/python3.9/site-packages user site-package
/opt/python3.9/lib/python3.9/site-packages global site-package
 

site-packages == purelib + platlib

Python defines two folders as python library locations, often set to same
              
                Python 3.9.4 (default, Apr  6 2021, 00:00:00)
                [GCC 11.0.1 20210324 (Red Hat 11.0.1-0)] on linux

                >>> import sys, sysconfig

                >>> sysconfig.get_path('platlib')
                '/env/lib64/python3.9/site-packages'

                >>> sysconfig.get_path('purelib')
                '/env/lib/python3.9/site-packages'

                >>> print('\n'.join(sys.path))
                /usr/lib64/python39.zip
                /usr/lib64/python3.9
                /usr/lib64/python3.9/lib-dynload
                /env/lib64/python3.9/site-packages
                /env/lib/python3.9/site-packages
              
          

What is an installed library made of?

  • the code itself - a module or package
    
                      ❯ lsd C:\Users\gabor\git\env\lib\site-packages\setuptools --tree
                       setuptools <- package
                      ├──  __init__.py <- module
                      ...
                      ├──  build_meta.py
                      ├──  cli-32.exe<- resource file
                      ...
                      └──  windows_support.py
                  
  • entry points - console/GUI generated alongside the Python executable
                  
                    ❯ lsd --tree .\env\Scripts
                     Scripts
                    ├──  pip-3.9.exe
                    ├──  pip.exe
                    ├──  pip3.9.exe
                    └──  pip3.exe
                  
  • metadata declaring the package by/for the installer - mostly as defined by PEP-427
                    
                      ❯ lsd C:\Users\gabor\git\env\lib\site-packages\setuptools-56.0.0.dist-info --tree
                       setuptools-56.0.0.dist-info
                      ├──  dependency_links.txt
                      ├──  entry_points.txt
                      ├──  INSTALLER
                      ├──  LICENSE
                      ├──  METADATA
                      ├──  RECORD
                      ├──  top_level.txt
                      └──  WHEEL
                    
                

What would it mean to be able to run our pi logic?


            $ python
            Python 3.9.4 (default, Apr 12 2021, 09:53:36)
            [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
            >>> from pi_approx import approximate_pi
            >>> approximate_pi(300)
            3.1382593295155914
            

            # pi_approx.py
            from math import pi

            def approximate_pi(iteration_count: int) -> float:
                sign, result = 1, 0.0
                for at in range(iteration_count):
                    result += sign / (2 * at + 1)
                    sign *= -1
                return result * 4
            
                
                  ❯ lsd .\env\Lib\site-packages\pi_approx* --tree
                   pi_approx.py
                   pi_approx-1.0.dist-info
                  ├──  direct_url.json
                  ├──  INSTALLER
                  ├──  LICENSE
                  ├──  METADATA
                  ├──  RECORD
                  ├──  REQUESTED
                  └──  WHEEL
                
                
What is a python project made of?
                 project
└──  pi_approx.py business logic file(s)







                
            
What is a python project made of?
                 project
├──  pi_approx.py
└──  test_pi_approx.py test file(s)






                
            
What is a python project made of?
                 project
├──  pi_approx.py
├──  test_pi_approx.py
├──  LICENSE project management file(s)
├──  README.md
└── .github
    └──  workflows
        └──  check.yaml

                
            
What is a python project made of?
                 project
├──  pi_approx.py
├──  test_pi_approx.py
├──  LICENSE
├──  README.md
├── .github
│   └──  workflows
│       └──  check.yaml
└──  pyproject.toml project packaging file(s)
                
            
What is a python project made of?
  • layout on our development machine
                     project
    ├──  pi_approx.py
    ├──  test_pi_approx.py
    ├──  LICENSE
    ├──  README.md
    ├── .github
    │   └──  workflows
    │       └──  check.yaml
    └──  pyproject.toml
                    
                
  • layout on their machine - the run environment
                    
                      ❯ lsd .\env\Lib\site-packages\pi_approx* --tree
                       pi_approx.py
                       pi_approx-1.0.dist-info
                      ├──  direct_url.json
                      ├──  INSTALLER
                      ├──  LICENSE
                      ├──  METADATA
                      ├──  RECORD
                      ├──  REQUESTED
                      └──  WHEEL
                    
                    
What is a python project made of?
  • layout on our development machine
                     project
    ├──  pi_approx.py
    ├──  test_pi_approx.py
    ├──  LICENSE
    ├──  README.md
    ├── .github
    │   └──  workflows
    │       └──  check.yaml
    └──  pyproject.toml
                    
                
  • layout on their machine - the run environment
                    
                      ❯ lsd .\env\Lib\site-packages\pi_approx* --tree
                       pi_approx.py
                       pi_approx-1.0.dist-info
                      ├──  direct_url.json
                      ├──  INSTALLER
                      ├──  LICENSE
                      ├──  METADATA
                      ├──  RECORD
                      ├──  REQUESTED
                      └──  WHEEL
                    
                    
Shipping code to another machine

Photo by Bernat Gabor - All rights reserved

Package types - (sdist)

source distribution - resembles what we have on the developer machine

  • all the files needed to install (and test) a package from raw source
  • source tree minus
    • project management files
    • maintainer files
  • has business logic, packaging, tests
               project
├──  pi_approx.py
├──  test_pi_approx.py
├──  LICENSE
├──  README.md
├── .github
│   └──  workflows
│       └──  check.yaml
└──  pyproject.toml
              
          

Package types - wheel

resembles what we want on the target machine

  • the installed binary files with some metadata
  • source tree minus
    • project management files
    • maintainer files
    • tests
    • packaging files
            
              ❯ lsd .\env\Lib\site-packages\pi_approx* --tree
               pi_approx.py
               pi_approx-1.0.dist-info
              ├──  direct_url.json
              ├──  INSTALLER
              ├──  LICENSE
              ├──  METADATA
              ├──  RECORD
              ├──  REQUESTED
              └──  WHEEL
            
            

sdist or wheel

  • We always install wheel - if we have a sdist, we first build a wheel from it

Breaking down shipping a library

Generate a package
Consume a package

Build a package

buckle up for a bit of history

Photo by Bernat Gabor - All rights reserved

History of packaging
  • 2000 - Python 1.6 - distutils - setup.py introduced:
    • used as invocation interface
    • configuration as Python code
  • 2004 - setuptools quickly became de facto standard
  • 2014 - wheels - PEP-427
  • 2015 - flit - declarative over dynamic (avoid arbitrary code run)
    • easier to understand
    • harder to get wrong
  • 2018 - poetry - single tool to rule them all
How does a build used to work

navigate to the developer source tree and invoke the following command(s)


              python setup.py sdist  # build a source distribution
              python setup.py bdist_wheel  # build a wheel
            
Problem 1 - build dependencies
  • setuptools pulled in from the builder Python
    
                        python setup.py sdist
                        python setup.py install
                    
                
  • cryptic error if build dependencies are missing
                            
                                      File "setup_build.py", line 99, in run
                                            from Cython.Build import cythonize
                                        ImportError: No module named Cython.Build
                            
                        
  • or even worse, no error - incorrect version
  • or even worse - user can hijack setuptools and try to inject their own code
Problem 1 - build dependencies
  • Idea: declarative build environment provision
                    
                        python setup.py bdist_wheel
                    
                    
  • create temporary folder
    • create an isolated Python environment
                          
                              python -m venv our_build_env
                          
                          
    • install build dependencies
                          
                              our_build_env/bin/python -m pip dep1 dep2
                          
                          
  • generate a wheel that we can simply install afterwards
                    
                        our_build_env/bin/python setup.py bdist_wheel
                    
                    
Problem 1 - build dependencies
  • PEP-518
  • specify deps via pyproject.toml - must be at the root of your project
                    
                        [build-system]
                        requires = [
                            "setuptools >= 40.8.0",
                            "wheel >= 0.30.0",
                            "cython >= 0.29.4",
                        ]
                    
                    
  • TOML - Tom's Obvious, Minimal Language.
    • standardized
    • not overly complex
    • does not conflict with other files out there - setup.cfg
Problem 1 - build dependencies
  • PEP-518
  • build back-end - performs the package generation in an isolated environment
  • build front-end - prepares the isolated environment, and invokes the back-end
    • build - unified CLI for back-ends
    • poetry, hatch, trampolim, pdm
    • pip - PEP-518 support with version 18.0
    • tox - PEP-518 support with version 3.3.0
Problem 2 - diversity - better user interface
  • PEP-518 involves still calling to setup.py
                    
                        our_build_env/bin/python setup.py bdist_wheel
                    
                
  • setup.py allows arbitrary python code to run - cannot change due to backward compatibility
  • flit implementation still needed generating setup.py and adhering to those limitations
Problem 2 - diversity - better user interface
  • PEP-517
  • So instead of:
                            
                                our_build_env/bin/python setup.py bdist_wheel
                            
                        
  • Allow build back-ends to expose invocation end-points:
    
                  [build-system]
                  requires = ["flit"]
                  build-backend = "flit.api:main"
                  
  • And the build front-end will now do:
                            
                                import flit.api
                                backend = flit.api.main
    
                                backend.build_wheel()  # build wheel via
                                backend.build_sdist()  # build source distribution via
                            
                        
Problem 2 - diversity - better user interface
  • PEP-517 back-end support
    • setuptools - setuptools.build_meta - version 40.8+
    • flit - flit.buildapi
    • trampolim - trampolim
    • poetry - poetry.masonry.api - version 0.12.11
  • PEP-517 front-end support
    • build
    • pip 19.0
    • tox 3.3.0

How to build a package?

  • Pick your build back-end
  • Use pyproject.toml to specify your
    • build dependencies
    • build back-end interface
  • Follow the docs of your back-end on how to configure it

Installing tools - pipx

  1. Install pipx by installing pipx-in-pipx
    
                  python -m pip install pipx-in-pipx
                  
  2. Make pipx path available on your PATH - ~/.local/bin
  3. Install your tools via pipx
    
                  pipx install build
                  pipx install flit
                  

Building a package with flit

Let's package our 𝜋 approximator with flit

Speed it up - cython

Adding Cython speed-up

  1. Move it to setuptools
  2. Add cython

Cython speed-up


          ❯ .\venv\Scripts\python.exe -m timeit -s 'from pi_approx import approximate_pi' 'approximate_pi(1_000_000)'
          2 loops, best of 5: 105 msec per loop

          ❯ .\venv\Scripts\python.exe -m timeit -s 'from pi_approx_cy import approximate_pi' 'approximate_pi(1_000_000)'
          5 loops, best of 5: 68.4 msec per loop
          

Upload a package

  • Historically it was (deprecated)
    
                  python setup.py upload
                  
  • Today we use twine
    
                  twine upload dist/*
                  
  • Though some tools do abstract this away:
    
                  flit publish
                  poetry publish
                  

Discover and download a package

Install a package

Photo by Bernat Gabor - All rights reserved

Testing a library

  1. create virtual environment
    
                  virtualenv venv
                  
  2. build (in a PEP-517/PEP-518 isolated environment) and install the package
    
                  pyproject-build --sdist .
                  .\venv\Scripts\pip install .\dist\pi_approx-1.0.0.tar.gz
    
                  # or the shorter form
                  # pip install .
                  
  3. install test dependencies
    
                    .\venv\Scripts\pip.exe install 'pytest>6'
                  
  4. run test
    
                  .\venv\Scripts\pytest tests
                  

tox


            # tox.ini
            [tox]
            envlist =
                py39
            isolated_build = true

            [testenv]
            deps =
                pytest>6
            commands =
                pytest tests
          

tox


            [tox]
            envlist =
                py39
                py38
                flake8
            isolated_build = true

            [testenv]
            deps =
                pytest>6
            commands =
                pytest tests

            [testenv:flake8]
            deps =
                flake8==3.9.1
            commands = 
                flake8 pi_approx.py tests

          

Best practices

  • Always publish both a wheel and a sdist
  • Ship the test suite with your sdist

Creating an application

Photo by Bernat Gabor - All rights reserved

Application vs. library

  • Library
    • runs with latest compatible dependencies
    • test with latest compatible dependencies
    • need to expose all modules/resources as they are
    • needs to be wheel installed
  • Application
    • runs with given set of dependencies
    • test with given set of dependencies
    • need to expose only the entry points
    • ideally just copy and execute

Application as a library

  • Do the same as we learned in previous chapter
  • Downsides
    • Users need to understand virtual environments
    • Entry-points must be defined separately
    • Running them might require path mangling
    • Dependencies are not pinned to a given version
  • Is still usable though if you can solve these in otherways - e.g., build docker images

zipapp

  • PEP-441 - Improving Python ZIP Application Support
  • Package python logic into a ZIP archive that can be executed by passing it to an interpreter
  • For example:
    
                  curl https://bootstrap.pypa.io/virtualenv.pyz -o virtualenv.pyz
                  python3.9 virtualenv.pyz --help
                  
  • provides a way to organize python files into a ZIP file and define a unified entry point for the ZIP file

zipapp package pi approximator


          py -m zipapp .\pi-approx-cy -m "pi_approx:run"
          

          def run() -> None:
              parser = argparse.ArgumentParser(description="Calculate pi approximation.")
              parser.add_argument("it", metavar="N", type=int, help="iteration count")
              ns = parser.parse_args()
              result = approximate_pi(ns.it)
              print(f"result is {result}")
          

pex

  • Create self-contained applications - UNIX only (no Windows support)
    
                    pex -c virtualenv -o virtualenv.pex virtualenv
                    python virtualenv.pex --help
                    
  • Python 2/3 support
  • Automatically discovers dependencies and packages
    
                0      1980-01-01 00:00   .bootstrap/
                0      1980-01-01 00:00   .bootstrap/pex/
                401    1980-01-01 00:00   .bootstrap/pex/__init__.py
                235    1980-01-01 00:00   .bootstrap/pex/__main__.py
                483    1980-01-01 00:00   .bootstrap/pex/attrs.py
                ...
                0      1980-01-01 00:00   .deps/
                0      1980-01-01 00:00   .deps/appdirs-1.4.4-py2.py3-none-any.whl/
                24720  1980-01-01 00:00   .deps/appdirs-1.4.4-py2.py3-none-any.whl/appdirs.py
              
  • Marks created package executable
    
                    # uses #!/usr/bin/env python
                    virtualenv.pex --help
                  
  • As entry-point, can select either main or console entry points
  • Note C-extension dependencies make the generated package not compatible with other platforms

shiv

  • Python 3.6+ support only but
  • Create self-contained applications - UNIX + Windows support
    
                    shiv -c virtualenv -o virtualenv.shiv virtualenv
                    python virtualenv.shiv --help
                    
  • Automatically discovers dependencies and packages
    
                  Archive:  ./virtualenv.shiv
                  Length      Date    Time    Name
                  ---------  ---------- -----   ----
                      24720  04/05/2021 22:37   site-packages/appdirs.py
                      8      04/05/2021 22:37   site-packages/appdirs-1.4.4.dist-info/top_level.txt
                      ...
                      65     04/05/2021 22:37   site-packages/virtualenv/version.py
                      2878   04/05/2021 22:37   _bootstrap/environment.py
                      2063   04/05/2021 22:37   _bootstrap/filelock.py
                      1762   04/05/2021 22:37   _bootstrap/interpreter.py
                      8728   04/05/2021 22:37   _bootstrap/__init__.py
                      377    04/05/2021 22:37   environment.json
                      65     04/05/2021 22:37   __main__.py
                  
  • Note C-extension dependencies make the generated package not compatible with other platforms

pyinstaller

  • Packages not just the entry point, app and dependencies, but also the interpreter
  • generates a single binary executable
    
                    ❯ pyinstaller.exe -F .\pi_approx.py 
                    ❯ .\dist\pi_approx.exe 10000000
                    result is 3.1415925535897915
                  

use pyinstaller to generate an executable for pi approximator

zipapp with dynamic dependency

virtualenv - case study

  • some dependencies are OS specific
  • some dependencies are Python version specific
                
                 install_requires =
                    appdirs>=1.4.3,<2
                    distlib>=0.3.0,<1
                    filelock>=3.0.0,<4
                    six>=1.9.0,<2   # keep it >=1.9.0 as it may cause problems on LTS platforms
                    importlib-metadata>=0.12,<2;python_version<"3.8"
                    importlib-resources>=1.0;python_version<"3.7"
                    pathlib2>=2.3.3,<3;python_version < '3.4' and sys.platform != 'win32'
                
            

zipapp with dynamic dependencies

3.9 3.8 3.7 3.6 3.5 3.4 2.7
appdirs 1.4.4 1.4.4 1.4.4 1.4.4 1.4.4 1.4.4 1.4.4
distlib 0.3.1 0.3.1 0.3.1 0.3.1 0.3.1 0.3.1 0.3.1
filelock 3.0.12 3.0.12 3.0.12 3.0.12 3.0.12 3.0.12 3.0.12
six 1.15.0 1.15.0 1.15.0 1.15.0 1.15.0 1.15.0 1.15.0
importlib-metadata 1.7.0 1.7.0 1.7.0 1.1.3 1.1.3
importlib-resources 2.0.1 2.0.1 1.0.2 1.0.2
pathlib2 2.3.5 2.3.5

zipapp content structure

                
                 .
                ├──  __main__.py
                ├──  virtualenv
                │   └  ...
                │──  virtualenv-20.0.25.dist-info
                │  ├──  entry_points.txt
                │  ├──  LICENSE
                │  ├──  METADATA
                │  ├──  RECORD
                │  ├──  top_level.txt
                │  ├──  WHEEL
                │  └──  zip-safe
                ├──  __virtualenv__
                │  ├──  appdirs-1.4.4-py2.py3-none-any
                │  ├──  configparser-4.0.2-py2.py3-none-any
                │  ├──  contextlib2-0.6.0.post1-py2.py3-none-any
                │  ├──  distlib-0.3.0-py3-none-any
                │  ├──  filelock-3.0.12-py3-none-any
                │  ├──  importlib_metadata-1.1.3-py2.py3-none-any
                │  ├──  importlib_metadata-1.6.1-py2.py3-none-any
                │  ├──  importlib_resources-1.0.2-py2.py3-none-any
                │  ├──  importlib_resources-2.0.1-py2.py3-none-any
                │  ├──  pathlib2-2.3.5-py2.py3-none-any
                │  ├──  scandir-1.10.0-cp38-cp38-macosx_10_14_x86_64
                │  ├──  six-1.15.0-py2.py3-none-any
                │  ├──  typing-3.7.4.1-py2-none-any
                │  ├──  typing-3.7.4.1-py3-none-any
                │  ├──  zipp-1.2.0-py2.py3-none-any
                │  └──  zipp-3.1.0-py3-none-any
                ├──  distributions.json
                └──  modules.json
                
            

zipapp content structure

                
                 .
                ├──  __main__.py
                ├──  virtualenv
                │   └  ...
                │──  virtualenv-20.0.25.dist-info
                │  ├──  entry_points.txt
                │  ├──  LICENSE
                │  ├──  METADATA
                │  ├──  RECORD
                │  ├──  top_level.txt
                │  ├──  WHEEL
                │  └──  zip-safe
                ├──  __virtualenv__
                │  ├──  appdirs-1.4.4-py2.py3-none-any
                │  ├──  configparser-4.0.2-py2.py3-none-any
                │  ├──  contextlib2-0.6.0.post1-py2.py3-none-any
                │  ├──  distlib-0.3.0-py3-none-any
                │  ├──  filelock-3.0.12-py3-none-any
                │  ├──  importlib_metadata-1.1.3-py2.py3-none-any
                │  ├──  importlib_metadata-1.6.1-py2.py3-none-any
                │  ├──  importlib_resources-1.0.2-py2.py3-none-any
                │  ├──  importlib_resources-2.0.1-py2.py3-none-any
                │  ├──  pathlib2-2.3.5-py2.py3-none-any
                │  ├──  scandir-1.10.0-cp38-cp38-macosx_10_14_x86_64
                │  ├──  six-1.15.0-py2.py3-none-any
                │  ├──  typing-3.7.4.1-py2-none-any
                │  ├──  typing-3.7.4.1-py3-none-any
                │  ├──  zipp-1.2.0-py2.py3-none-any
                │  └──  zipp-3.1.0-py3-none-any
                ├──  distributions.json
                └──  modules.json
                
            

Use import hooks within zipapp

🙏 thank you 🙏