virtual environments

status quo of virtual environments

by Bernat Gabor / @gjbernat / Bloomberg
why

slides @ bit.ly/virtual-env

Who am I?

Maintainer of the virtualenv tool
PyPa (Python Packaging Authority) member
github/gaborbernat
Maintainer of the tox tool
Software Engineer at Bloomberg

What is a virtual environment? - PEP-405

isolated Python environment

access to its own site-package

shares access to host

built-ins
standard library

optionally isolated from the system site package

What? 😲 Why?

How do you get hold of a module in Python

©Creative Commons Attribution-NonCommercial 2.5 License

import

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

not all modules are equal

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
    <module 'sys' (built-in)>
>>> import os
>>> os
    <module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
    <module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>

not all modules are equal

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
    <module 'sys' (built-in)>
builtin module
>>> import os
>>> os
    <module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
    <module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>

not all modules are equal

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
    <module 'sys' (built-in)>
>>> import os
>>> os
    <module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
standard library module
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
    <module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>

not all modules are equal

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
    <module 'sys' (built-in)>
>>> import os
>>> os
    <module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
standard library dynamic load module
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
    <module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>

not all modules are equal

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
    <module 'sys' (built-in)>
>>> import os
>>> os
    <module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
global site package module
>>> import pep517
>>> pep517
    <module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>

not all modules are equal

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys
    <module 'sys' (built-in)>
>>> import os
>>> os
    <module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
>>> import _csv
>>> _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
>>> import pytoml
>>> pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
>>> import pep517
>>> pep517
    <module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>
user site package module

the import system - importers

                    
>>> import sys

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

the import system - built-in

                    
>>> import sys

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

the import system - frozen

freeze tool, pyinstaller, etc.

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

the import system - path finder

                    
>>> import os

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for importer in sys.meta_path:
...     print(repr(importer))
...
<class '_frozen_importlib.BuiltinImporter'>
<class '_frozen_importlib.FrozenImporter'>
<class '_frozen_importlib_external.PathFinder'>

the import system - path finder

                    
>>> import os

<class '_frozen_importlib_external.PathFinder'>

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

the import system - path finder

                    
>>> import os

<class '_frozen_importlib_external.PathFinder'>

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip standard library zipped
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

the import system - path finder

                    
>>> import os
<module 'os' from '/opt/python3.7/lib/python3.7/os.py'>
<class '_frozen_importlib_external.PathFinder'>

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7 standard library directory
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

the import system - path finder

                    
>>> import _csv
<module '_csv' from '/opt/python3.7/lib/python3.7/lib-dynload/_csv.cpython-37dm-darwin.so'>
<class '_frozen_importlib_external.PathFinder'>

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload standard library c-extensions
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

the import system - path finder

                    
>>> import pytoml
<module 'pytoml' from '/opt/python3.7/lib/python3.7/site-packages/pytoml/__init__.py'>
<class '_frozen_importlib_external.PathFinder'>

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/.local/lib/python3.7/site-packages user site package
/opt/python3.7/lib/python3.7/site-packages

the import system - path finder

                    
>>> pep517
<module 'pep517' from '/Users/bernat/.local/lib/python3.7/site-packages/pep517/__init__.py'>
<class '_frozen_importlib_external.PathFinder'>

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages global site package

What is a virtual environment? - PEP-405

isolated Python environment

access to its own site-package

shares access to host

built-ins
standard library

optionally isolated from the system site package

System python paths

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip standard library zipped
/opt/python3.7/lib/python3.7 standard library
/opt/python3.7/lib/python3.7/lib-dynload standard library c-extension

/Users/bernat/.local/lib/python3.7/site-packages user site package
/opt/python3.7/lib/python3.7/site-packages global site package

virtual environment python paths - no global access

                    
>>> import sys
>>> sys.executable
/Users/bernat/env/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip standard library zipped
/opt/python3.7/lib/python3.7 standard library
/opt/python3.7/lib/python3.7/lib-dynload standard library c-extension
/Users/bernat/env/lib/python3.7/site-package OWN site package

virtual environment python paths - global access

                    
>>> import sys
>>> sys.executable
/Users/bernat/env/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip standard library zipped
/opt/python3.7/lib/python3.7 standard library
/opt/python3.7/lib/python3.7/lib-dynload standard library c-extension
/Users/bernat/env/lib/python3.7/site-package OWN site package
/Users/bernat/.local/lib/python3.7/site-packages user site package
/opt/python3.7/lib/python3.7/site-packages global site package

System python paths

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip standard library zipped
/opt/python3.7/lib/python3.7 standard library
/opt/python3.7/lib/python3.7/lib-dynload standard library c-extension

/Users/bernat/.local/lib/python3.7/site-packages user site package
/opt/python3.7/lib/python3.7/site-packages global site package

WHY?

install without system administrator rights
package install without influencing system Python
create easy to reproduce test environments
easier dependency management

HOW?

sys.path - who comes up with this list?

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/env/lib/python3.7/site-package
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

sys.path - who comes up with this list? - standard library

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/env/lib/python3.7/site-package
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

sys.path - who comes up with this list? - site.py

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/env/lib/python3.7/site-package
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

sys.path - who comes up with this list?

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
/opt/python3.7/lib/python37.zip
/opt/python3.7/lib/python3.7
/opt/python3.7/lib/python3.7/lib-dynload
/Users/bernat/env/lib/python3.7/site-package
/Users/bernat/.local/lib/python3.7/site-packages
/opt/python3.7/lib/python3.7/site-packages

sys.path - who comes up with this list?

                    
>>> import sys
>>> sys.executable
/opt/python3.7/bin/python

                    
Python 3.7.4rc1+ (heads/3.7:887e0576b8, Jun 21 2019, 12:20:59)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

>>> import sys
>>> for path in sys.path:
...     print(repr(path))
...
sys.prefix + /lib/python37.zip
sys.prefix + /lib/python3.7
sys.exec_prefix + /lib/python3.7/lib-dynload
env_home + /lib/python3.7/site-package
user_home + /lib/python3.7/site-packages
sys.prefix + /lib/python3.7/site-packages

sys.prefix - who comes up with this? - getpath.c

                    
>>> import sys
>>> sys.prefix
/opt/python3.7

if PYTHONHOME set, use that
otherwise, find landmark starting at executable path
lib/pythonX.X/os.py
otherwise, use build time hardcoded value

sys.exec_prefix - who comes up with this? - getpath.c

                    
>>> import sys
>>> sys.exec_prefix
/opt/python3.7

if PYTHONHOME set, use that
otherwise, find landmark starting at executable path
lib/pythonX.X/lib-dynload
otherwise, use build time hardcoded value

and we're done

How to create a virtual environment?

isolated Python environment

shares access to host

built-ins
standard library

ensure landmarks exists relative to Python executable path

optionally isolated from the system site package

conditional logic for site.py

access to its own site-package

logic for site.py - always add for virtual environment

Changes needed for virtual environment support in python?

standard library discovery - if Python copied (keep track of host)
site.py
- flag for adding the system site package
- flag if in virtual environment

pyvenv.cfg - Python 3 - PEP-405

                    
home = /opt
include-system-site-packages = false
version = 3.7.4

standard library discovery - if Python copied (keep track of host)
site.py
- flag for adding the system site package
- flag if in virtual environment

Bootstrap - You no pip?

How we install packages - via pip!
catch 22 - don't have pip installed to install pip
solution:
- get a wheel/zipapp of pip (can be bundled)
- put the wheel/zipapp onto the sys.path - PYTHONPATH
- self install
ensurepip

Implementations

venv - https://docs.python.org/3/library/venv.html

Python 3.3+ only
part of the standard library for the Python implementation
target itself only - 3.6 can only create environments for 3.6
uses ensurepip to bootstrap pip/setuptools/wheel - offline only

Implementations

virtualenv - https://virtualenv.pypa.io/en/latest

third party package - out of band upgrade
full Python 2 and 3 support
CPython and PyPy support (open for IronPython, Jython, etc.)
cross Python support
cross platform support
built-in interpreter discovery - PEP-504 Windows, PEP-394 POSIX
uses bundled pip/setuptools/wheel to bootstrap, can use index server

virtualenv internals

virtualenv internals - Ian Bicking's

alter standard library discovery
alter site.py logic

virtualenv internals - Ian Bicking's

always copy installer to force discovery start from here

                    
    env2.7/
    └── bin
        └── python2.7

virtualenv internals - Ian Bicking's

ensure sys.prefix and sys.exec_prefix landmark files are available

                    
    env2.7/
    ├── bin
    │   └── python2.7
    └── lib
        └── python2.7
            ├── lib-dynload -> /opt/python2.7/lib/python2.7/lib-dynload
            └── os.py -> /opt/python2.7/lib/python2.7/os.py

virtualenv internals - Ian Bicking's

add custom site.py to fix things - inject host stdlib, global site, our site

                    
    env2.7/
    ├── bin
    │   └── python2.7
    └── lib
        └── python2.7
            ├── lib-dynload -> /opt/python2.7/lib/python2.7/lib-dynload
            ├── no-global-site-packages.txt
            ├── os.py -> /opt/python2.7/lib/python2.7/os.py
            ├── orig-prefix.txt
            ├── site-packages
            └── site.py

virtualenv internals - Ian Bicking's

site.py imports a few stuff, these need to be there - link them in

                    
env2.7/
├── bin
│   └── python2.7
└── lib
    └── python2.7
        ├── UserDict.py -> /opt/python2.7/lib/python2.7/UserDict.py
        ├── _abcoll.py -> /opt/python2.7/lib/python2.7/_abcoll.py
        ├── _weakrefset.py -> /opt/python2.7/lib/python2.7/_weakrefset.py
        ├── abc.py -> /opt/python2.7/lib/python2.7/abc.py
        ├── codecs.py -> /opt/python2.7/lib/python2.7/codecs.py
        ├── copy_reg.py -> /opt/python2.7/lib/python2.7/copy_reg.py
        ├── encodings -> /opt/python2.7/lib/python2.7/encodings
        ├── fnmatch.py -> /opt/python2.7/lib/python2.7/fnmatch.py
        ├── genericpath.py -> /opt/python2.7/lib/python2.7/genericpath.py
        ├── lib-dynload -> /opt/python2.7/lib/python2.7/lib-dynload
        ├── linecache.py -> /opt/python2.7/lib/python2.7/linecache.py
        ├── locale.py -> /opt/python2.7/lib/python2.7/locale.py
        ├── no-global-site-packages.txt
        ├── ntpath.py -> /opt/python2.7/lib/python2.7/ntpath.py
        ├── orig-prefix.txt
        ├── os.py -> /opt/python2.7/lib/python2.7/os.py
        ├── posixpath.py -> /opt/python2.7/lib/python2.7/posixpath.py
        ├── re.py -> /opt/python2.7/lib/python2.7/re.py
        ├── site-packages
        ├── site.py
        ├── sre.py -> /opt/python2.7/lib/python2.7/sre.py
        ├── sre_compile.py -> /opt/python2.7/lib/python2.7/sre_compile.py
        ├── sre_constants.py -> /opt/python2.7/lib/python2.7/sre_constants.py
        ├── sre_parse.py -> /opt/python2.7/lib/python2.7/sre_parse.py
        ├── stat.py -> /opt/python2.7/lib/python2.7/stat.py
        ├── types.py -> /opt/python2.7/lib/python2.7/types.py
        └── warnings.py -> /opt/python2.7/lib/python2.7/warnings.py

virtualenv internals - Ian Bicking's

site.py imports a few stuff: 21... Not Great...Not Terrible.

                    
env2.7/
├── bin
│   └── python2.7
└── lib
    └── python2.7
        ├── UserDict.py -> /opt/python2.7/lib/python2.7/UserDict.py
        ├── _abcoll.py -> /opt/python2.7/lib/python2.7/_abcoll.py
        ├── _weakrefset.py -> /opt/python2.7/lib/python2.7/_weakrefset.py
        ├── abc.py -> /opt/python2.7/lib/python2.7/abc.py
        ├── codecs.py -> /opt/python2.7/lib/python2.7/codecs.py
        ├── copy_reg.py -> /opt/python2.7/lib/python2.7/copy_reg.py
        ├── encodings -> /opt/python2.7/lib/python2.7/encodings
        ├── fnmatch.py -> /opt/python2.7/lib/python2.7/fnmatch.py
        ├── genericpath.py -> /opt/python2.7/lib/python2.7/genericpath.py
        ├── lib-dynload -> /opt/python2.7/lib/python2.7/lib-dynload
        ├── linecache.py -> /opt/python2.7/lib/python2.7/linecache.py
        ├── locale.py -> /opt/python2.7/lib/python2.7/locale.py
        ├── no-global-site-packages.txt
        ├── ntpath.py -> /opt/python2.7/lib/python2.7/ntpath.py
        ├── orig-prefix.txt
        ├── os.py -> /opt/python2.7/lib/python2.7/os.py
        ├── posixpath.py -> /opt/python2.7/lib/python2.7/posixpath.py
        ├── re.py -> /opt/python2.7/lib/python2.7/re.py
        ├── site-packages
        ├── site.py
        ├── sre.py -> /opt/python2.7/lib/python2.7/sre.py
        ├── sre_compile.py -> /opt/python2.7/lib/python2.7/sre_compile.py
        ├── sre_constants.py -> /opt/python2.7/lib/python2.7/sre_constants.py
        ├── sre_parse.py -> /opt/python2.7/lib/python2.7/sre_parse.py
        ├── stat.py -> /opt/python2.7/lib/python2.7/stat.py
        ├── types.py -> /opt/python2.7/lib/python2.7/types.py
        └── warnings.py -> /opt/python2.7/lib/python2.7/warnings.py

virtualenv internals - Ian Bicking's

add activation scripts, include,distutils,fix all bugs (tlk, platform specific)

                    
env2.7/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.ps1
│   ├── activate_this.py
│   ├── python -> python2.7
│   ├── python-config
│   ├── python2 -> python2.7
│   └── python2.7
├── include
│   └── python2.7 -> /opt/python2.7/include/python2.7
└── lib
    └── python2.7
        ├── LICENSE.txt -> /opt/python2.7/lib/python3.7/LICENSE.txt
        ├── UserDict.py -> /opt/python2.7/lib/python2.7/UserDict.py
        ├── _abcoll.py -> /opt/python2.7/lib/python2.7/_abcoll.py
        ├── _weakrefset.py -> /opt/python2.7/lib/python2.7/_weakrefset.py
        ├── abc.py -> /opt/python2.7/lib/python2.7/abc.py
        ├── codecs.py -> /opt/python2.7/lib/python2.7/codecs.py
        ├── copy_reg.py -> /opt/python2.7/lib/python2.7/copy_reg.py
        ├── distutils
        │   ├── __init__.py
        │   └── distutils.cfg
        ├── encodings -> /opt/python2.7/lib/python2.7/encodings
        ├── fnmatch.py -> /opt/python2.7/lib/python2.7/fnmatch.py
        ├── genericpath.py -> /opt/python2.7/lib/python2.7/genericpath.py
        ├── lib-dynload -> /opt/python2.7/lib/python2.7/lib-dynload
        ├── linecache.py -> /opt/python2.7/lib/python2.7/linecache.py
        ├── locale.py -> /opt/python2.7/lib/python2.7/locale.py
        ├── ntpath.py -> /opt/python2.7/lib/python2.7/ntpath.py
        ├── no-global-site-packages.txt
        ├── orig-prefix.txt
        ├── os.py -> /opt/python2.7/lib/python2.7/os.py
        ├── posixpath.py -> /opt/python2.7/lib/python2.7/posixpath.py
        ├── re.py -> /opt/python2.7/lib/python2.7/re.py
        ├── site-packages
        ├── site.py
        ├── sre.py -> /opt/python2.7/lib/python2.7/sre.py
        ├── sre_compile.py -> /opt/python2.7/lib/python2.7/sre_compile.py
        ├── sre_constants.py -> /opt/python2.7/lib/python2.7/sre_constants.py
        ├── sre_parse.py -> /opt/python2.7/lib/python2.7/sre_parse.py
        ├── stat.py -> /opt/python2.7/lib/python2.7/stat.py
        ├── types.py -> /opt/python2.7/lib/python2.7/types.py
        └── warnings.py -> /opt/python2.7/lib/python2.7/warnings.py

done

virtualenv internals - Ian Bicking's

                    
env2.7/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.ps1
│   ├── activate_this.py
│   ├── python -> python2.7
│   ├── python-config
│   ├── python2 -> python2.7
│   └── python2.7
├── include
│   └── python2.7 -> /opt/python2.7/include/python2.7
└── lib
    └── python2.7
        ├── LICENSE.txt -> /opt/python2.7/lib/python2.7/LICENSE.txt
        ├── UserDict.py -> /opt/python2.7/lib/python2.7/UserDict.py
        ├── _abcoll.py -> /opt/python2.7/lib/python2.7/_abcoll.py
        ├── _weakrefset.py -> /opt/python2.7/lib/python2.7/_weakrefset.py
        ├── abc.py -> /opt/python2.7/lib/python2.7/abc.py
        ├── codecs.py -> /opt/python2.7/lib/python2.7/codecs.py
        ├── copy_reg.py -> /opt/python2.7/lib/python2.7/copy_reg.py
        ├── distutils
        │   ├── __init__.py
        │   └── distutils.cfg
        ├── encodings -> /opt/python2.7/lib/python2.7/encodings
        ├── fnmatch.py -> /opt/python2.7/lib/python2.7/fnmatch.py
        ├── genericpath.py -> /opt/python2.7/lib/python2.7/genericpath.py
        ├── lib-dynload -> /opt/python2.7/lib/python2.7/lib-dynload
        ├── linecache.py -> /opt/python2.7/lib/python2.7/linecache.py
        ├── locale.py -> /opt/python2.7/lib/python2.7/locale.py
        ├── ntpath.py -> /opt/python2.7/lib/python2.7/ntpath.py
        ├── no-global-site-packages.txt
        ├── orig-prefix.txt
        ├── os.py -> /opt/python2.7/lib/python2.7/os.py
        ├── posixpath.py -> /opt/python2.7/lib/python2.7/posixpath.py
        ├── re.py -> /opt/python2.7/lib/python2.7/re.py
        ├── site-packages
        ├── site.py
        ├── sre.py -> /opt/python2.7/lib/python2.7/sre.py
        ├── sre_compile.py -> /opt/python2.7/lib/python2.7/sre_compile.py
        ├── sre_constants.py -> /opt/python2.7/lib/python2.7/sre_constants.py
        ├── sre_parse.py -> /opt/python2.7/lib/python2.7/sre_parse.py
        ├── stat.py -> /opt/python2.7/lib/python2.7/stat.py
        ├── types.py -> /opt/python2.7/lib/python2.7/types.py
        └── warnings.py -> /opt/python2.7/lib/python2.7/warnings.py

venv world

                    
env3.7
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── python -> python3.7
│   ├── python3 -> python3.7
│   └── python3.7 -> /opt/python3.7
├── include
├── lib
│   └── python3.7
│       └── site-packages
└── pyvenv.cfg

5 directories, 7 files

virtualenv - rewrite

proposal https://github.com/pypa/virtualenv/issues/1366

third-party package - offered via wheel and zipapp
all Python implementations welcomed
two year grace period
ability to select target Python (-p 3, -p 3.7.4-64)
prefer built-in venv
broad shell activation support (+xonsh) - unified behaviour
three layered config system - ini, environment variable, flags
plugin system for extension

status quo of virtual environments

Who am I?

What is a virtual environment? - PEP-405

What? 😲 Why?

How do you get hold of a module in Python

import

not all modules are equal

not all modules are equal

not all modules are equal

not all modules are equal

not all modules are equal

not all modules are equal

the import system - importers

the import system - built-in

the import system - frozen

the import system - path finder

the import system - path finder

the import system - path finder

the import system - path finder

the import system - path finder

the import system - path finder

the import system - path finder

What is a virtual environment? - PEP-405

System python paths

virtual environment python paths - no global access

virtual environment python paths - global access

System python paths

WHY?

HOW?

sys.path - who comes up with this list?

sys.path - who comes up with this list? - standard library

sys.path - who comes up with this list? - site.py

sys.path - who comes up with this list?

sys.path - who comes up with this list?

sys.prefix - who comes up with this? - getpath.c

sys.exec_prefix - who comes up with this? - getpath.c

and we're done

How to create a virtual environment?

Changes needed for virtual environment support in python?

pyvenv.cfg - Python 3 - PEP-405

Bootstrap - You no pip?

Implementations

Implementations

virtualenv internals

virtualenv internals - Ian Bicking's

virtualenv internals - Ian Bicking's

virtualenv internals - Ian Bicking's

virtualenv internals - Ian Bicking's

virtualenv internals - Ian Bicking's

virtualenv internals - Ian Bicking's

virtualenv internals - Ian Bicking's

done

virtualenv internals - Ian Bicking's

venv world

virtualenv - rewrite

thank you - questions