How we standardized

editable installs

PEP-660 vs. PEP-662

by Bernat Gabor / @gjbernat / Bloomberg
https://gaborbernat.github.io/pep-660-662-2022

Who am I?

  • software engineer at Bloomberg (data ingestion and quality control)
  • OSS contributor @ github/gaborbernat, blog bernat.tech
  • member of the Python Packaging Authority (virtualenv, build, pipx, tox maintainer)
  • 😎 parent to two Yorkshire Terriers
Silky

Copyright 2022 Bernat Gabor All rights reserved

What is an editable installation?

Eager

Copyright 2022 Bernat Gabor All rights reserved

What is an editable installation?

  • Let there be the following Python project
    
                     .
                    ├──  pyproject.toml
                    └──  src
                       └──  demo
                          └──  __init__.py
                  
  • Project metadata in pyproject.toml (PEP-621 style)
    
                  [build-system]
                  build-backend = "hatchling.build"
                  requires = ["hatchling>=0.14"]
    
                  [project]
                  name = "demo"
                  version = "1.0"
                  
  • src/demo/__init__.py
    
                  def say_hi(name: str) -> None:
                    print(f"Hello {name}")
                  
  • How do we run it?
    
                  $ python -c 'from demo import say_hi; say_hi("PyCon")'
                  Traceback (most recent call last):
                    File "<string>", line 1, in <module>
                  ModuleNotFoundError: No module named 'demo'
                  

What is an editable installation?

    To test the library, we need to install it (in an isolated virtual environment). For example:
  1. create isolated virtual environment
    
                  $ virtualenv venv --clear --no-wheel --no-setuptools --activators ""
                  created virtual environment CPython3.10.4.final.0-64 in 88ms
                    creator CPython3Posix(dest=/tmp/prj/venv, clear=True, no_vcs_ignore=False, global=False)
                    seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/tmp/virtualenv)
                      added seed packages: pip==22.0.4
                  
  2. install the project by using pip
    
                    $ venv/bin/pip install .
                    Processing /tmp/prj
                      Installing build dependencies ... done
                      Getting requirements to build wheel ... done
                      Preparing metadata (pyproject.toml) ... done
                    Building wheels for collected packages: demo
                      Building wheel for demo (pyproject.toml) ... done
                      Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029
                      Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339
                    Successfully built demo
                    Installing collected packages: demo
                    Successfully installed demo-1.0
                    
  3. run the test code
    
                  $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
                  Hello PyCon
                  

What is an editable installation?

    What will happen when I change the library code inside the projects source directory?
  • change src/demo/__init__.py
    
                  def say_hi(name: str) -> None:
                    print(f"Hello {name}!")
                  
  • and run the test code again
    
                  $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
                  Hello PyCon
                  
  • install the library
    
                      $ venv/bin/pip install .
                      Processing /tmp/prj
                        Installing build dependencies ... done
                        Getting requirements to build wheel ... done
                        Preparing metadata (pyproject.toml) ... done
                      Building wheels for collected packages: demo
                        Building wheel for demo (pyproject.toml) ... done
                        Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029
                        Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339
                      Successfully built demo
                      Installing collected packages: demo
                      Successfully installed demo-1.0
                      
  • run the test code
    
                    $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
                    Hello PyCon!
                    

What is an editable installation?

  • Having to install the project after every code change slows down development

    Copyright of the above image owned by StreamClips

What is an editable installation?

    Editable install ensures subsequent code changes take effect without a reinstallation
  • install the library in editable mode
    
                      $ venv/bin/pip install -e .
                      Processing /tmp/prj
                        Installing build dependencies ... done
                        Getting requirements to build wheel ... done
                        Preparing metadata (pyproject.toml) ... done
                      Building wheels for collected packages: demo
                        Building wheel for demo (pyproject.toml) ... done
                        Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029f9ae06fdd752a1e36c
                        Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339bf57ad6563
                      Successfully built demo
                      Installing collected packages: demo
                      Successfully installed demo-1.0
                      
  • run the test code
    
                    $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
                    Hello PyCon
                    
  • change src/demo/__init__.py
    
                    def say_hi(name: str) -> None:
                      print(f"Hello {name}!")
                    
  • run the test code
    
                    $ venv/bin/python -c 'from demo import say_hi; say_hi("PyCon")'
                    Hello PyCon!
                    

Prior art

Curios

Copyright 2022 Bernat Gabor All rights reserved

Prior art

  • Initially introduced with distutils and inherited by setuptools under the interface
    
                  $ python setup.py develop
                  
  • Used by pip until version 21.3 (released on 2021-10-11):
    
                  $ python -m pip install -e /path/to/project
                  # under the hood calls - python setup.py develop with current working directory of /path/to/project
                  

Prior art - Normal installation

graph LR S[Source directory] W[Wheel] SP[site-packages] P[Python executable] S -->|build via copy| W W -->|install via extract| SP SP -.-|import demo| P

           .
          ├──  pyproject.toml
          └──  src
              └──  demo
                └──  __init__.py
          

          $ venv/bin/pip install .
          Processing /tmp/prj
            Installing build dependencies ... done
            Getting requirements to build wheel ... done
            Preparing metadata (pyproject.toml) ... done
          Building wheels for collected packages: demo
            Building wheel for demo (pyproject.toml) ... done
            Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029f9ae06fdd752a1e36c
            Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339bf57ad6563
          Successfully built demo
          Installing collected packages: demo
          Successfully installed demo-1.0
          


          $ zipinfo -1 demo-1.0-py2.py3-none-any.whl
          demo/__init__.py
          demo-1.0.dist-info/METADATA
          demo-1.0.dist-info/WHEEL
          demo-1.0.dist-info/entry_points.txt
          demo-1.0.dist-info/RECORD
          

Prior art - Normal installation

graph LR S[Source directory] W[Wheel] SP[site-packages] P[Python executable] S -->|build via copy| W W -->|install via extract| SP SP -.-|import demo| P

           .
          ├──  pyproject.toml
          └──  src
              └──  demo
                └──  __init__.py
          

          $ venv/bin/pip install .
          Processing /tmp/prj
            Installing build dependencies ... done
            Getting requirements to build wheel ... done
            Preparing metadata (pyproject.toml) ... done
          Building wheels for collected packages: demo
            Building wheel for demo (pyproject.toml) ... done
            Created wheel for demo: filename=demo-1.0-py2.py3-none-any.whl size=1089 sha256=4af9f4008bef1f029f9ae06fdd752a1e36c
            Stored in directory: /tmp/pip-ephem-wheel-cache-1t02m0nm/wheels/41/8a/ba/ffb94b3ccf1b5c11339bf57ad6563
          Successfully built demo
          Installing collected packages: demo
          Successfully installed demo-1.0
          


          $ zipinfo -1 demo-1.0-py2.py3-none-any.whl
          demo/__init__.py
          demo-1.0.dist-info/METADATA
          demo-1.0.dist-info/WHEEL
          demo-1.0.dist-info/entry_points.txt
          demo-1.0.dist-info/RECORD
          

Prior art - Normal installation

graph LR S[Source directory] W[Wheel] SP[site-packages] P[Python executable] S -->|build via copy| W W -->|install via extract| SP SP -.-|import demo| P

              $ zipinfo -1 demo-1.0-py2.py3-none-any.whl
              demo/__init__.py
              demo-1.0.dist-info/METADATA
              demo-1.0.dist-info/WHEEL
              demo-1.0.dist-info/entry_points.txt
              demo-1.0.dist-info/RECORD
              


            $ lsd --tree venv/lib/python3.10/site-packages/demo*
             venv/lib/python3.10/site-packages/demo
            ├──  __init__.py
            └──  __pycache__
              └──  __init__.cpython-310.pyc
             venv/lib/python3.10/site-packages/demo-1.0.dist-info
            ├──  direct_url.json
            ├──  entry_points.txt
            ├──  INSTALLER
            ├──  METADATA
            ├──  RECORD
            ├──  REQUESTED
            └──  WHEEL
            

Prior art - Normal installation

graph LR S[Source directory] W[Wheel] SP[site-packages] P[Python executable] S -->|build via copy| W W -->|install via extract| SP SP -.-|import demo| P

              $ zipinfo -1 demo-1.0-py2.py3-none-any.whl
              demo/__init__.py
              demo-1.0.dist-info/METADATA
              demo-1.0.dist-info/WHEEL
              demo-1.0.dist-info/entry_points.txt
              demo-1.0.dist-info/RECORD
              


          $ lsd --tree venv/lib/python3.10/site-packages/demo*
           venv/lib/python3.10/site-packages/demo
          ├──  __init__.py
          └──  __pycache__
            └──  __init__.cpython-310.pyc
           venv/lib/python3.10/site-packages/demo-1.0.dist-info
          ├──  direct_url.json
          ├──  entry_points.txt
          ├──  INSTALLER
          ├──  METADATA
          ├──  RECORD
          ├──  REQUESTED
          └──  WHEEL
            

Prior art - Normal installation

graph LR S[Source directory] W[Wheel] SP[site-packages] P[Python executable] S -->|build via copy| W W -->|install via extract| SP SP -.-|import demo| P


              $ zipinfo -1 demo-1.0-py2.py3-none-any.whl
              demo/__init__.py
              demo-1.0.dist-info/METADATA
              demo-1.0.dist-info/WHEEL
              demo-1.0.dist-info/entry_points.txt
              demo-1.0.dist-info/RECORD
              


            $ lsd --tree venv/lib/python3.10/site-packages/demo*
             venv/lib/python3.10/site-packages/demo
            ├──  __init__.py
            └──  __pycache__
              └──  __init__.cpython-310.pyc
             venv/lib/python3.10/site-packages/demo-1.0.dist-info
            ├──  direct_url.json
            ├──  entry_points.txt
            ├──  INSTALLER
            ├──  METADATA
            ├──  RECORD
            ├──  REQUESTED
            └──  WHEEL
            

Prior art - Editable installation

graph LR S[Source directory] P[Python executable] S -.-|import demo| P S -->|build metadata and install pth file to resolve imports| S

             .
            ├──  setup.py
            └──  src
              └──  demo
                  └──  __init__.py
            


          from setuptools import setup
          setup(
              name="demo",
              version="1.0",
              package_dir={"": "src"},
              packages=["demo"])
           


           $ virtualenv venv --clear --no-wheel --pip 21.2 --activators ""
           created virtual environment CPython3.10.4.final.0-64 in 101ms
           added seed packages: pip==21.2, setuptools==62.1.0
           $ venv/bin/pip install -e .
           Obtaining file:///home/bernat/git/pep-660-662-2022/prj
           Installing collected packages: demo
             Running setup.py develop for demo
           Successfully installed demo-1.0
           

Prior art - Editable installation

graph LR S[Source directory] P[Python executable] S -.-|import demo| P S -->|build metadata and install pth file to resolve imports| S

             .
            ├──  setup.py
            └──  src
              ├──  demo
              │  └──  __init__.py
              └──  demo.egg-info
                  ├──  dependency_links.txt
                  ├──  PKG-INFO
                  ├──  SOURCES.txt
                  └──  top_level.txt
            


            $ cat venv/lib/python3.10/site-packages/easy-install.pth
            /tmp/prj/src
            


          $ venv/bin/python -m site
          sys.path = [
              '/tmp/prj',
              '/usr/lib/python310.zip',
              '/usr/lib/python3.10',
              '/usr/lib/python3.10/lib-dynload',
              '/tmp/prj/venv/lib/python3.10/site-packages',
              '/tmp/prj/src',
          ]
          USER_BASE: '/home/bernat/.local' (exists)
          USER_SITE: '/home/bernat/.local/lib/python3.10/site-packages' (exists)
          ENABLE_USER_SITE: False
          

Prior art - Editable installation

graph LR S[Source directory] P[Python executable] S -.-|import demo| P S -->|build metadata and install pth file to resolve imports| S

             .
            ├──  setup.py
            └──  src
              ├──  demo
              │  └──  __init__.py
              └──  demo.egg-info
                  ├──  dependency_links.txt
                  ├──  PKG-INFO
                  ├──  SOURCES.txt
                  └──  top_level.txt
            


            $ cat venv/lib/python3.10/site-packages/easy-install.pth
            /tmp/prj/src
            


          $ venv/bin/python -m site
          sys.path = [
              '/tmp/prj',
              '/usr/lib/python310.zip',
              '/usr/lib/python3.10',
              '/usr/lib/python3.10/lib-dynload',
              '/tmp/prj/venv/lib/python3.10/site-packages',
              '/tmp/prj/src',
          ]
          USER_BASE: '/home/bernat/.local' (exists)
          USER_SITE: '/home/bernat/.local/lib/python3.10/site-packages' (exists)
          ENABLE_USER_SITE: False
          

Is this problem solved? No!

Drawbacks of the prior art

Not all sunshine

Copyright 2022 Bernat Gabor All rights reserved

Files to add to package not validated


          from setuptools import setup

          setup(
              name="demo",
              version="1.0",
              package_dir={"": "src"},
              packages=["demo"],
          )
          


           .
          ├──  setup.py
          └──  src
            ├──  demo
            │  └──  __init__.py  
            ├──  magic
            │  └──  __init__.py
            └──  demo.egg-info
                ├──  dependency_links.txt
                ├──  PKG-INFO
                ├──  SOURCES.txt
                └──  top_level.txt
          
  • editable installation will pick up import magic
  • normal installation will not pick package magic import magic

Files to exclude from package not validated


            from setuptools import setup
  
            setup(
                name="demo",
                version="1.0",
                package_dir={"": "src"},
                packages=["demo"],
                exclude=["demo.tests"],
            )
            


             .
            ├──  setup.py
            └──  src
              ├──  demo
              │  ├──   __init__.py
              │  └──   tests
              │     └──  __init__.py
              └──  demo.egg-info
                  ├──  dependency_links.txt
                  ├──  PKG-INFO
                  ├──  SOURCES.txt
                  └──  top_level.txt
            
  • editable installation will allow you to run import demo.tests
  • normal installation will fail import demo.tests

No support for generated code

  • you get JSON/OpenAPI schema to a service - generate Python objects from it
                
                    {
                      "type": "object",
                      "properties": {
                        "firstName": {
                          "type": "string",
                          "description": "The person's first name."
                        }
                      }
                    }
                  
  • for example, PyDantic objects via datamodel-code-generator
                
                    $ datamodel-codegen --input schema.json
                    # generated by datamodel-codegen:
                    #   filename:  schema.json
                    #   timestamp: 2022-04-17T15:16:32+00:00
    
                    from __future__ import annotations
    
                    from typing import Optional
    
                    from pydantic import BaseModel, Field
    
    
                    class Model(BaseModel):
                        firstName: Optional[str] = Field(None, description="The person's first name.")
                  

No support for C-extensions

  • for example, a Cython powered project
    
                     .
                    ├──  setup.py
                    └──  src
                      └──  demo
                         ├──   __init__.py
                         └──   _demo.pyx  
                    
  • an editable build generates (compiles) native code at install time only
    
                     .
                    ├──  setup.py
                    └──  src
                      ├──  demo
                      │  ├──   __init__.py
                      │  ├──   _demo.pyx
                      │  └──    _demo.cpython-310-x86_64-linux-gnu.so*
                      └──  demo.egg-info
                          ├──  dependency_links.txt
                          ├──  PKG-INFO
                          ├──  SOURCES.txt
                          └──  top_level.txt
                    

New approaches

Eager

Copyright 2022 Bernat Gabor All rights reserved

Life without a PTH file?

Symlinks - Folder

  • Available on UNIX, Windows 7+ supports it (with a potential one-time setup)
  • Symlink each file from within the source directory to site-packages at build time
    
                    $ lsd -tree /tmp/prj
                     . 
                    ├──  pyproject.toml
                    └──  src
                        └──  demo
                          └──  __init__.py
                    

    
                  $ lsd -alth venv/lib/python3.10/site-packages/demo
                  drwxr-xr-x bernat bernat 131 B  Sun Apr 17 17:30:12 2022  .
                  lrwxrwxrwx bernat bernat  46 B  Sun Apr 17 17:30:12 2022  demo ⇒ /tmp/prj/src/demo
                  
      • No startup performance impact
      • File paths mirror normal installation
      • New files added are automatically included but does not allow exclusion

Symlinks - Files

  • Symlink each file from within the source directory to site-packages at build time
    
                    $ lsd -tree /tmp/prj
                     . 
                    ├──  pyproject.toml
                    └──  src
                        │──  demo
                        │  └──  __init__.py
                        └──  test_demo.py
                    

    
                    $ lsd -alth venv/lib/python3.10/site-packages/demo
                    drwxr-xr-x bernat bernat  25 B Sun Apr 17 17:36:40 2022  .
                    lrwxrwxrwx bernat bernat  58 B Sun Apr 17 17:36:14 2022  __init__.py ⇒ /tmp/prj/src/demo/__init__.py
                    
      • No startup performance impact
      • File paths mirror normal installation
      • Allows file level inclusion/exclusion but new files require reinstall

Import hooks

  • We're importing modules, not files, let's use that!
  • Extend the import system to manifest the module at import time
  • Fully dynamic (load packaging config at first import, and validate config/file system)!
  • Brainstormed together with Paul Moore in 2020 - he released it as the editables library

Import hooks

  • Generates a PTH file that handles system initialization
    
                  .rw-r--r--  venv/lib/python3.10/site-packages/demo.pth
                  .rw-r--r--  venv/lib/python3.10/site-packages/_editable_impl_demo.py
                  
  • Just load the import logic
    
                  import _editable_impl_pyproject_fmt
                  
  • Register the editable import hook and define how modules map to source files
    
                  from editables.redirector import RedirectingFinder as F
                  F.install()  # install import editables import hook to sys.meta_path
                  F.map_module('demo', '/tmp/prj/src/demo/__init__.py')   # register module name to source file
                  

Import hooks

  • addition of new files as it would check during import if exists
  • exclusion of files as it could check during the import if excluded and skip
  • generate code from JSONSpec/OpenAPI during the import
  • recompile/build C-extension modules on import if the source file has been changed since last build

The Python build system

Lot of info

Copyright 2022 Bernat Gabor All rights reserved

The Python build system

    
                pip wheel -w --no-deps /tmp/prj
                pyproject-build -w .
                
    In order to build a wheel, two distinct components work together:
  • build backend (e.g., setuptools, hatchling)
    load project configuration and build the wheel file
  • build frontend (pip, build)
    • creates an isolated build environment
    • installs the build dependencies as specified by the requires key in pyproject.toml
      
                        [build-system]
                        build-backend = "hatchling.build"
                        requires = ["hatchling>=0.14"]
                        
    • call the build backend via the interface defined in PEP-517
      • get runtime build dependencies and install them
        
                              def get_requires_for_build_wheel(config_settings=None):
                                  return ["wheel"]
                              
      • perform the wheel build
        
                              def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
                                  ...  # e.g., for setuptools call python setup.py bdist_wheel
                              

The Python build system


          $ pyproject-build -w /tmp/virtualenv
          * Creating venv isolated environment...
          * Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
          * Getting dependencies for wheel...
          * Installing packages in isolated environment... (wheel >= 0.37.1)
          * Building wheel...
          running bdist_wheel
          running build
          running build_py
          ...
          Successfully built virtualenv-20.14.1-py2.py3-none-any.whl
          

The Python build system


          $ pyproject-build -w /tmp/virtualenv
          * Creating venv isolated environment...
          * Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
          * Getting dependencies for wheel...
          * Installing packages in isolated environment... (wheel >= 0.37.1)
          * Building wheel...
          running bdist_wheel
          running build
          running build_py
          ...
          Successfully built virtualenv-20.14.1-py2.py3-none-any.whl
          

The Python build system


          $ pyproject-build -w /tmp/virtualenv
          * Creating venv isolated environment...
          * Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
          * Getting dependencies for wheel...
          * Installing packages in isolated environment... (wheel >= 0.37.1)
          * Building wheel...
          running bdist_wheel
          running build
          running build_py
          ...
          Successfully built virtualenv-20.14.1-py2.py3-none-any.whl
          

The Python build system


          $ pyproject-build -w /tmp/virtualenv
          * Creating venv isolated environment...
          * Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
          * Getting dependencies for wheel...
          * Installing packages in isolated environment... (wheel >= 0.37.1)
          * Building wheel...
          running bdist_wheel
          running build
          running build_py
          ...
          Successfully built virtualenv-20.14.1-py2.py3-none-any.whl
          

The Python build system


          $ pyproject-build -w /tmp/virtualenv
          * Creating venv isolated environment...
          * Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
          * Getting dependencies for wheel...
          * Installing packages in isolated environment... (wheel >= 0.37.1)
          * Building wheel...
          running bdist_wheel
          running build
          running build_py
          ...
          Successfully built virtualenv-20.14.1-py2.py3-none-any.whl
          

The Python build system


          $ pyproject-build -w /tmp/virtualenv
          * Creating venv isolated environment...
          * Installing packages in isolated environment... (setuptools >= 62.1.0, setuptools-scm >= 6.4)
          * Getting dependencies for wheel...
          * Installing packages in isolated environment... (wheel >= 0.37.1)
          * Building wheel...
          running bdist_wheel
          running build
          running build_py
          ...
          Successfully built virtualenv-20.14.1-py2.py3-none-any.whl
          

The new solution

Time to write a Python Enhancement Proposal
Eager

Copyright 2022 Bernat Gabor All rights reserved

Two proposals

  • We could agree on what editable builds are
  • We struggled to agree on:
    • How much of the complex problems we should solve?
    • Should the build backend or the build frontend do the heavy lifting?
  • In the end, we created two proposals:

PEP-660 – Editable installs for pyproject.toml based builds (wheel based)

    Extends the API between the build backend and frontend to build editable wheels:
  • Introduce an editable wheel build environment
    
                    def get_requires_for_build_editable(config_settings=None):
                      ...
                  
  • Adds an endpoint that builds the editable wheel
    
                  def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
                    ...
                  

PEP-660 – Editable installs for pyproject.toml based builds (wheel based)

    An editable installation is made up of:
  1. Build frontend creates an isolated build environment
  2. Build frontend installs build-system.requires from pyproject.toml
  3. The frontend asks the backend for dependencies needed to build an editable wheel
    
                    def get_requires_for_build_editable(config_settings=None):
                      ...
                  
  4. Frontend installs additional dependencies
  5. Frontend asks the backend to build the editable wheel
    
                  def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
                    ...
                  
  6. Wheel installed into test environment via an installer per existing standards

PEP-660 – Editable installs for pyproject.toml based builds (wheel based)

  • The editable effect is handled by the build backend via the files it puts in the wheel
    • PTH
    • import hook
  • Backend may expose multiple ways (change via config_settings arg of build_editable)
  • Backends can reuse/delegate editable logic via libraries (e.g., editables)
  • Symlinks are not supported as of today
    • interested parties are invited to submit a PEP to extend the wheel standard
  • No change for uninstallers - reusing the wheel standard takes care of it

PEP-662 – Editable installs via virtual wheels

  • Interface extension between build backend and frontend is same as PEP-660
  • Similar build process, but the produced wheel artifact must contain at root editables.json
    
                  {
                      "version": 1,
                      "scheme": {
                         "purelib": {"/src/tree/a.py": "tree/a.py"},
                         "platlib": {},
                         "data": {"/src/tree/py.typed": "tree/py.typed"},
                         "headers": {},
                         "scripts": {}
                      }
                   }
                  
  • The JSON file tells the frontend what files need to be mapped and where
  • It's up to the frontend how it makes it happen, and should allow user to select:
    
                    pip install -e /prj --mode=symlink|pth|hook
                    
    • Symlinks
    • PTH
    • sitecustomize.py
    • import hooks

Choosing a winner

  • Paul Moore is the BDFL for packaging PEPs
  • one week cooldown period
  • PEP-660 announced as winner on June 2021
    • Primary reason given was that PEP-662 left the frontends tasks too open

Lessons learned

Lessons learned

Silky

Creative Commons Attribution-Share Alike 4.0 International

Questions?

  • Find me after at the conference!

thank you

Copyright 2022 Bernat Gabor All rights reserved

https://www.bernat.tech

we're hiring

TechAtBloomberg.com/python

bloomberg.com/engineering

@TechAtBloomberg

Bloomberg

© 2022 Bloomberg Finance L.P.
All rights reserved.