Who am I?

• software engineer at Bloomberg (data ingestion and quality control)
• OSS contributor @ github/gaborbernat, blog bernat.tech
• member of the Python Packaging Authority (virtualenv, build, pipx, tox maintainer)
• 😎 parent to two Yorkshire Terriers

Copyright 2022 Bernat Gabor All rights reserved

What is an editable installation?

What is an editable installation?

• Let there be the following Python project

   .
  ├──  pyproject.toml
  └──  src
     └──  demo
        └──  __init__.py

• Project metadata in pyproject.toml (PEP-621 style)

  [build-system]
  build-backend = "hatchling.build"
  requires = ["hatchling>=0.14"]
  
  [project]
  name = "demo"
  version = "1.0"

• src/demo/__init__.py

  def say_hi(name: str) -> None:
    print(f"Hello {name}")

• How do we run it?

  $ python -c 'from demo import say_hi; say_hi("PyCon")'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named 'demo'

What is an editable installation?

To test the library, we need to install it (in an isolated virtual environment). For example:
1. create isolated virtual environment

   $ virtualenv venv --clear --no-wheel --no-setuptools --activators ""
   created virtual environment CPython3.10.4.final.0-64 in 88ms
     creator CPython3Posix(dest=/tmp/prj/venv, clear=True, no_vcs_ignore=False, global=False)
     seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/tmp/virtualenv)
       added seed packages: pip==22.0.4

2. install the project by using pip

   $ venv/bin/pip install .
   Processing /tmp/prj
     Installing build dependencies ... done
     Getting requirements to build wheel ... done
     Preparing metadata (pyproject.toml) ... done
   Building wheels for collected packages: demo
     Building wheel for demo (pyproject.toml) ... done
     Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029
     Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339
   Successfully built demo
   Installing collected packages: demo
   Successfully installed demo-1.0

3. run the test code

   $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
   Hello PyCon

What is an editable installation?

What will happen when I change the library code inside the projects source directory?
• change src/demo/__init__.py

  def say_hi(name: str) -> None:
    print(f"Hello {name}!")

• and run the test code again

  $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
  Hello PyCon

install the library

$ venv/bin/pip install .
Processing /tmp/prj
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: demo
  Building wheel for demo (pyproject.toml) ... done
  Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029
  Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339
Successfully built demo
Installing collected packages: demo
Successfully installed demo-1.0

run the test code

$ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
Hello PyCon!

What is an editable installation?

• Having to install the project after every code change slows down development
  
  

  Copyright of the above image owned by StreamClips

What is an editable installation?

Editable install ensures subsequent code changes take effect without a reinstallation
install the library in editable mode

$ venv/bin/pip install -e .
Processing /tmp/prj
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: demo
  Building wheel for demo (pyproject.toml) ... done
  Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029f9ae06fdd752a1e36c
  Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339bf57ad6563
Successfully built demo
Installing collected packages: demo
Successfully installed demo-1.0

run the test code

$ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
Hello PyCon

change src/demo/__init__.py

def say_hi(name: str) -> None:
  print(f"Hello {name}!")

run the test code

$ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
Hello PyCon!

Prior art

Copyright 2022 Bernat Gabor All rights reserved

Prior art

• Initially introduced with distutils and inherited by setuptools under the interface

  $ python setup.py develop

• Used by pip until version 21.3 (released on 2021-10-11):

  $ python -m pip install -e /path/to/project
  # under the hood calls - python setup.py develop with current working directory of /path/to/project

Prior art - Normal installation

 .
├──  pyproject.toml
└──  src
    └──  demo
      └──  __init__.py

$ venv/bin/pip install .
Processing /tmp/prj
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: demo
  Building wheel for demo (pyproject.toml) ... done
  Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029f9ae06fdd752a1e36c
  Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339bf57ad6563
Successfully built demo
Installing collected packages: demo
Successfully installed demo-1.0

$ zipinfo -1 demo-1.0-py2.py3-none-any.whl
demo/__init__.py
demo-1.0.dist-info/METADATA
demo-1.0.dist-info/WHEEL
demo-1.0.dist-info/entry_points.txt
demo-1.0.dist-info/RECORD

Prior art - Normal installation

 .
├──  pyproject.toml
└──  src
    └──  demo
      └──  __init__.py

$ venv/bin/pip install .
Processing /tmp/prj
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: demo
  Building wheel for demo (pyproject.toml) ... done
  Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029f9ae06fdd752a1e36c
  Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339bf57ad6563
Successfully built demo
Installing collected packages: demo
Successfully installed demo-1.0

$ zipinfo -1 demo-1.0-py2.py3-none-any.whl
demo/__init__.py
demo-1.0.dist-info/METADATA
demo-1.0.dist-info/WHEEL
demo-1.0.dist-info/entry_points.txt
demo-1.0.dist-info/RECORD

Prior art - Normal installation

$ zipinfo -1 demo-1.0-py2.py3-none-any.whl
demo/__init__.py
demo-1.0.dist-info/METADATA
demo-1.0.dist-info/WHEEL
demo-1.0.dist-info/entry_points.txt
demo-1.0.dist-info/RECORD

$ lsd --tree venv/lib/python3.10/site-packages/demo*
 venv/lib/python3.10/site-packages/demo
├──  __init__.py
└──  __pycache__
  └──  __init__.cpython-310.pyc
 venv/lib/python3.10/site-packages/demo-1.0.dist-info
├──  direct_url.json
├──  entry_points.txt
├──  INSTALLER
├──  METADATA
├──  RECORD
├──  REQUESTED
└──  WHEEL

Prior art - Normal installation

$ zipinfo -1 demo-1.0-py2.py3-none-any.whl
demo/__init__.py
demo-1.0.dist-info/METADATA
demo-1.0.dist-info/WHEEL
demo-1.0.dist-info/entry_points.txt
demo-1.0.dist-info/RECORD

$ lsd --tree venv/lib/python3.10/site-packages/demo*
 venv/lib/python3.10/site-packages/demo
├──  __init__.py
└──  __pycache__
  └──  __init__.cpython-310.pyc
 venv/lib/python3.10/site-packages/demo-1.0.dist-info
├──  direct_url.json
├──  entry_points.txt
├──  INSTALLER
├──  METADATA
├──  RECORD
├──  REQUESTED
└──  WHEEL

Prior art - Normal installation

$ zipinfo -1 demo-1.0-py2.py3-none-any.whl
demo/__init__.py
demo-1.0.dist-info/METADATA
demo-1.0.dist-info/WHEEL
demo-1.0.dist-info/entry_points.txt
demo-1.0.dist-info/RECORD

$ lsd --tree venv/lib/python3.10/site-packages/demo*
 venv/lib/python3.10/site-packages/demo
├──  __init__.py
└──  __pycache__
  └──  __init__.cpython-310.pyc
 venv/lib/python3.10/site-packages/demo-1.0.dist-info
├──  direct_url.json
├──  entry_points.txt
├──  INSTALLER
├──  METADATA
├──  RECORD
├──  REQUESTED
└──  WHEEL

Prior art - Editable installation

 .
├──  setup.py
└──  src
  └──  demo
      └──  __init__.py

from setuptools import setup
setup(
    name="demo",
    version="1.0",
    package_dir={"": "src"},
    packages=["demo"])

$ virtualenv venv --clear --no-wheel --pip 21.2 --activators ""
created virtual environment CPython3.10.4.final.0-64 in 101ms
added seed packages: pip==21.2, setuptools==62.1.0
$ venv/bin/pip install -e .
Obtaining file:///home/bernat/git/pep-660-662-2022/prj
Installing collected packages: demo
  Running setup.py develop for demo
Successfully installed demo-1.0

Prior art - Editable installation

 .
├──  setup.py
└──  src
  ├──  demo
  │  └──  __init__.py
  └──  demo.egg-info
      ├──  dependency_links.txt
      ├──  PKG-INFO
      ├──  SOURCES.txt
      └──  top_level.txt

$ cat venv/lib/python3.10/site-packages/easy-install.pth
/tmp/prj/src

$ venv/bin/python -m site
sys.path = [
    '/tmp/prj',
    '/usr/lib/python310.zip',
    '/usr/lib/python3.10',
    '/usr/lib/python3.10/lib-dynload',
    '/tmp/prj/venv/lib/python3.10/site-packages',
    '/tmp/prj/src',
]
USER_BASE: '/home/bernat/.local' (exists)
USER_SITE: '/home/bernat/.local/lib/python3.10/site-packages' (exists)
ENABLE_USER_SITE: False

Prior art - Editable installation

 .
├──  setup.py
└──  src
  ├──  demo
  │  └──  __init__.py
  └──  demo.egg-info
      ├──  dependency_links.txt
      ├──  PKG-INFO
      ├──  SOURCES.txt
      └──  top_level.txt

$ cat venv/lib/python3.10/site-packages/easy-install.pth
/tmp/prj/src

$ venv/bin/python -m site
sys.path = [
    '/tmp/prj',
    '/usr/lib/python310.zip',
    '/usr/lib/python3.10',
    '/usr/lib/python3.10/lib-dynload',
    '/tmp/prj/venv/lib/python3.10/site-packages',
    '/tmp/prj/src',
]
USER_BASE: '/home/bernat/.local' (exists)
USER_SITE: '/home/bernat/.local/lib/python3.10/site-packages' (exists)
ENABLE_USER_SITE: False

Is this problem solved? No!

• Requires setup.py file
• What an editable installation is and how it should work was never agreed upon
• Prior art is an implementation detail of a setuptools
• Why did we not do it before? We did try! (more than once)
  • 2015 - part of PEP-517 (standardization of the normal builds)
  • 2019 April - part of the PyCon US 2019 packaging summit
  • 2019 May - follow-up discussion to the decision at the summit
  • 2020 April - third try on editable installs
  • 2021 April - Editable installs for pyproject.toml based builds
  • 2021 May - Editable installs via virtual wheels

Drawbacks of the prior art

Copyright 2022 Bernat Gabor All rights reserved

Files to add to package not validated

from setuptools import setup

setup(
    name="demo",
    version="1.0",
    package_dir={"": "src"},
    packages=["demo"],
)

 .
├──  setup.py
└──  src
  ├──  demo
  │  └──  __init__.py  
  ├──  magic
  │  └──  __init__.py
  └──  demo.egg-info
      ├──  dependency_links.txt
      ├──  PKG-INFO
      ├──  SOURCES.txt
      └──  top_level.txt

• editable installation will pick up import magic
• normal installation will not pick package magic import magic

Files to exclude from package not validated

from setuptools import setup

setup(
    name="demo",
    version="1.0",
    package_dir={"": "src"},
    packages=["demo"],
    exclude=["demo.tests"],
)

 .
├──  setup.py
└──  src
  ├──  demo
  │  ├──   __init__.py
  │  └──   tests
  │     └──  __init__.py
  └──  demo.egg-info
      ├──  dependency_links.txt
      ├──  PKG-INFO
      ├──  SOURCES.txt
      └──  top_level.txt

• editable installation will allow you to run import demo.tests
• normal installation will fail import demo.tests

No support for generated code

• you get JSON/OpenAPI schema to a service - generate Python objects from it

  {
    "type": "object",
    "properties": {
      "firstName": {
        "type": "string",
        "description": "The person's first name."
      }
    }
  }

• for example, PyDantic objects via datamodel-code-generator

  $ datamodel-codegen --input schema.json
  # generated by datamodel-codegen:
  #   filename:  schema.json
  #   timestamp: 2022-04-17T15:16:32+00:00
  
  from __future__ import annotations
  
  from typing import Optional
  
  from pydantic import BaseModel, Field
  
  
  class Model(BaseModel):
      firstName: Optional[str] = Field(None, description="The person's first name.")

No support for C-extensions

• for example, a Cython powered project

   .
  ├──  setup.py
  └──  src
    └──  demo
       ├──   __init__.py
       └──   _demo.pyx  

• an editable build generates (compiles) native code at install time only

   .
  ├──  setup.py
  └──  src
    ├──  demo
    │  ├──   __init__.py
    │  ├──   _demo.pyx
    │  └──    _demo.cpython-310-x86_64-linux-gnu.so*
    └──  demo.egg-info
        ├──  dependency_links.txt
        ├──  PKG-INFO
        ├──  SOURCES.txt
        └──  top_level.txt

New approaches

Copyright 2022 Bernat Gabor All rights reserved

Life without a PTH file?

Symlinks - Folder

• Available on UNIX, Windows 7+ supports it (with a potential one-time setup)
• Symlink each file from within the source directory to site-packages at build time

  $ lsd -tree /tmp/prj
   . 
  ├──  pyproject.toml
  └──  src
      └──  demo
        └──  __init__.py

  
  

  $ lsd -alth venv/lib/python3.10/site-packages/demo
  drwxr-xr-x bernat bernat 131 B  Sun Apr 17 17:30:12 2022  .
  lrwxrwxrwx bernat bernat  46 B  Sun Apr 17 17:30:12 2022  demo ⇒ /tmp/prj/src/demo

  • No startup performance impact
  • File paths mirror normal installation
  • New files added are automatically included but does not allow exclusion

Symlinks - Files

• Symlink each file from within the source directory to site-packages at build time

  $ lsd -tree /tmp/prj
   . 
  ├──  pyproject.toml
  └──  src
      │──  demo
      │  └──  __init__.py
      └──  test_demo.py

  
  

  $ lsd -alth venv/lib/python3.10/site-packages/demo
  drwxr-xr-x bernat bernat  25 B Sun Apr 17 17:36:40 2022  .
  lrwxrwxrwx bernat bernat  58 B Sun Apr 17 17:36:14 2022  __init__.py ⇒ /tmp/prj/src/demo/__init__.py

  • No startup performance impact
  • File paths mirror normal installation
  • Allows file level inclusion/exclusion but new files require reinstall

Import hooks

• We're importing modules, not files, let's use that!
• Extend the import system to manifest the module at import time
• Fully dynamic (load packaging config at first import, and validate config/file system)!
• Brainstormed together with Paul Moore in 2020 - he released it as the editables library

Import hooks

• Generates a PTH file that handles system initialization

  .rw-r--r--  venv/lib/python3.10/site-packages/demo.pth
  .rw-r--r--  venv/lib/python3.10/site-packages/_editable_impl_demo.py

• Just load the import logic

  import _editable_impl_pyproject_fmt

• Register the editable import hook and define how modules map to source files

  from editables.redirector import RedirectingFinder as F
  F.install()  # install import editables import hook to sys.meta_path
  F.map_module('demo', '/tmp/prj/src/demo/__init__.py')   # register module name to source file

Import hooks

• addition of new files as it would check during import if exists
• exclusion of files as it could check during the import if excluded and skip
• generate code from JSONSpec/OpenAPI during the import
• recompile/build C-extension modules on import if the source file has been changed since last build

The Python build system

Copyright 2022 Bernat Gabor All rights reserved

The Python build system

pip wheel -w --no-deps /tmp/prj
pyproject-build -w .

In order to build a wheel, two distinct components work together:
• build backend (e.g., setuptools, hatchling)
  load project configuration and build the wheel file
• build frontend (pip, build)
  • creates an isolated build environment
  • installs the build dependencies as specified by the requires key in pyproject.toml

    [build-system]
    build-backend = "hatchling.build"
    requires = ["hatchling>=0.14"]

  • call the build backend via the interface defined in PEP-517
    • get runtime build dependencies and install them

      def get_requires_for_build_wheel(config_settings=None):
          return ["wheel"]

    • perform the wheel build

      def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
          ...  # e.g., for setuptools call python setup.py bdist_wheel

The Python build system

$ pyproject-build -w /tmp/virtualenv
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
* Getting dependencies for wheel...
* Installing packages in isolated environment... (wheel >= 0.37.1)
* Building wheel...
running bdist_wheel
running build
running build_py
...
Successfully built virtualenv-20.14.1-py2.py3-none-any.whl

The Python build system

$ pyproject-build -w /tmp/virtualenv
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
* Getting dependencies for wheel...
* Installing packages in isolated environment... (wheel >= 0.37.1)
* Building wheel...
running bdist_wheel
running build
running build_py
...
Successfully built virtualenv-20.14.1-py2.py3-none-any.whl

The Python build system

$ pyproject-build -w /tmp/virtualenv
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
* Getting dependencies for wheel...
* Installing packages in isolated environment... (wheel >= 0.37.1)
* Building wheel...
running bdist_wheel
running build
running build_py
...
Successfully built virtualenv-20.14.1-py2.py3-none-any.whl

The Python build system

$ pyproject-build -w /tmp/virtualenv
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
* Getting dependencies for wheel...
* Installing packages in isolated environment... (wheel >= 0.37.1)
* Building wheel...
running bdist_wheel
running build
running build_py
...
Successfully built virtualenv-20.14.1-py2.py3-none-any.whl

The Python build system

$ pyproject-build -w /tmp/virtualenv
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
* Getting dependencies for wheel...
* Installing packages in isolated environment... (wheel >= 0.37.1)
* Building wheel...
running bdist_wheel
running build
running build_py
...
Successfully built virtualenv-20.14.1-py2.py3-none-any.whl

The Python build system

$ pyproject-build -w /tmp/virtualenv
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
* Getting dependencies for wheel...
* Installing packages in isolated environment... (wheel >= 0.37.1)
* Building wheel...
running bdist_wheel
running build
running build_py
...
Successfully built virtualenv-20.14.1-py2.py3-none-any.whl

The new solution

Time to write a Python Enhancement Proposal

Copyright 2022 Bernat Gabor All rights reserved

Two proposals

• We could agree on what editable builds are
• We struggled to agree on:
  • How much of the complex problems we should solve?
  • Should the build backend or the build frontend do the heavy lifting?
• In the end, we created two proposals:
  • Backend does most of it - PEP-660 - traditional wheel based
  • Frontend does most of it - PEP-662 - virtual wheels

PEP-660 – Editable installs for pyproject.toml based builds (wheel based)

Extends the API between the build backend and frontend to build editable wheels:
• Introduce an editable wheel build environment

  def get_requires_for_build_editable(config_settings=None):
    ...

• Adds an endpoint that builds the editable wheel

  def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
    ...

PEP-660 – Editable installs for pyproject.toml based builds (wheel based)

An editable installation is made up of:
1. Build frontend creates an isolated build environment
2. Build frontend installs build-system.requires from pyproject.toml
3. The frontend asks the backend for dependencies needed to build an editable wheel

   def get_requires_for_build_editable(config_settings=None):
     ...

4. Frontend installs additional dependencies
5. Frontend asks the backend to build the editable wheel

   def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
     ...

6. Wheel installed into test environment via an installer per existing standards

PEP-660 – Editable installs for pyproject.toml based builds (wheel based)

• The editable effect is handled by the build backend via the files it puts in the wheel
  • PTH
  • import hook
• Backend may expose multiple ways (change via config_settings arg of build_editable)
• Backends can reuse/delegate editable logic via libraries (e.g., editables)
• Symlinks are not supported as of today
  • interested parties are invited to submit a PEP to extend the wheel standard
• No change for uninstallers - reusing the wheel standard takes care of it

PEP-662 – Editable installs via virtual wheels

• Interface extension between build backend and frontend is same as PEP-660
• Similar build process, but the produced wheel artifact must contain at root editables.json

  {
      "version": 1,
      "scheme": {
         "purelib": {"/src/tree/a.py": "tree/a.py"},
         "platlib": {},
         "data": {"/src/tree/py.typed": "tree/py.typed"},
         "headers": {},
         "scripts": {}
      }
   }

• The JSON file tells the frontend what files need to be mapped and where
• It's up to the frontend how it makes it happen, and should allow user to select:

  pip install -e /prj --mode=symlink|pth|hook

  • Symlinks
  • PTH
  • sitecustomize.py
  • import hooks

Choosing a winner

• Paul Moore is the BDFL for packaging PEPs
  
• one week cooldown period
• PEP-660 announced as winner on June 2021
  • Primary reason given was that PEP-662 left the frontends tasks too open

Lessons learned

Quick adoption by the community
• Build frontends support
  • pip support since version 21.3 (released on 2021-10-11)
• Build backend support
  • hatchling, pdm, enscons, flit, poetry
  • setuptools adoption stalled 😔 due to tech debt and not wanting a 1:1 migration

Lessons learned

Creative Commons Attribution-Share Alike 4.0 International

• IDE and static checkers cannot support the dynamic aspect of import hooks
  • would require generating some static metadata - best effort support

Questions?

• Find me after at the conference!

thank you

Copyright 2022 Bernat Gabor All rights reserved

https://www.bernat.tech

we're hiring

TechAtBloomberg.com/python

bloomberg.com/engineering

@TechAtBloomberg