Type hints
inside the snake pit
by Bernat Gabor /
@gjbernat /
Bloomberg
https://gaborbernat.github.io/pycon-us-2019
Who am I?
- github/gaborbernat
- maintainer of the virtualenv tool (PyPa member)
- maintainer of the tox tool
- software engineer at Bloomberg (data ingestion and quality control)
- typing summit participant
In Python?
.jpg){kind=link}
Type hinting
- foundations laid down in PEP-484
- implemented by (+ reference checker - mypy)
- Guido van Rossum - BDFL
- Jukka Lehtosalo - mypy
- Ivan Levkivskyi
- Łukasz Langa
- added to Python standard library with version 3.5 (improved ever since)
from typing import Any
Type hinting
- still on Python 2?
- end of life at 2020 January 1
- it's available (backported) via PyPi package down to 2.7 😃
why do it?
easier to
- understand (and use) the source code
- maintain
- debug
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- understand (and use) the source code
- maintain
- debug
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- understand (and use) the source code
- maintain
- debug
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- understand (and use) the source code
- maintain
- debug
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- understand (and use) the source code
- maintain
- debug
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
Easier refactoring
- find usages of types
- discover call hierarchy
- better detection of objects and lifecycle flow
help editors suggestion engine
lint checks - find bugs with no tests
- IDE - PyCharm/VsCode
- mypy - reference implementation type checker
- pyre - Facebook
- pytype - Google
- pyright - Microsoft
improve documentation
data validation - pydantic
- a cleaner syntax to specify type requirements
from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError
class User(BaseModel):
id: int
name = 'John Doe'
signup_ts: datetime = None
friends: List[int] = []
external_data = {'id': '123', 'signup_ts': 'broken',
'friends': [1, 2, 'not a number']}
try:
user = User(**external_data)
except ValidationError as e:
print(e.json())
detect vulnerable code - security
Facebook - pyre
def magic_call(unsafe: UserProvidedStr, safe: str):
subprocess.check_call([unsafe])
subprocess.check_call([safe])
detect vulnerable code - security
Facebook - pyre
def magic_call(unsafe: UserProvidedStr, safe: str):
subprocess.check_call([unsafe]) # error out
subprocess.check_call([safe])
detect vulnerable code - security
Facebook - pyre
def magic_call(unsafe: UserProvidedStr, safe: str):
subprocess.check_call([unsafe])
subprocess.check_call([safe]) # allow
data validation - pydantic
- a cleaner syntax to specify type requirements
from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError
class User(BaseModel):
id: int
name = 'John Doe'
signup_ts: datetime = None
friends: List[int] = []
external_data = {'id': '123', 'signup_ts': 'broken',
'friends': [1, 2, 'not a number']}
try:
user = User(**external_data)
except ValidationError as e:
print(e.json())
data validation - pydantic
- a cleaner syntax to specify type requirements
from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError
class User(BaseModel):
id: int
name = 'John Doe'
signup_ts: datetime = None
friends: List[int] = []
external_data = {'id': '123', 'signup_ts': 'broken',
'friends': [1, 2, 'not a number']}
try:
user = User(**external_data)
except ValidationError as e:
print(e.json())
what it is not
essentially treated as comment during script evaluation
- no runtime type inference
- no performance improvement
what it is not
essentially treated as comment during script evaluation
- no runtime type inference
no performance improvement
mypyc - speed up for free (mostly)
Improve the developer experience, not performance
Improve the developer experience, not performance
Improve the developer experience, and performance
what kind of?
Gradual typing
- no type hints specified ⇒ code dynamically typed
- but we can type hint
- function parameters
- function return values
- variables
- only type hinted elements are type checked
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
How to add it
type annotations
syntax based on
- mypy
- PEP-3107 ~ function annotations - Python 3.0
- PEP-526 ~ variable annotations - Python 3.6
type annotations
for example
def greeting(name: str) -> str:
value : str = 'Hello'
return value + name
type annotations
function annotation
def greeting(name: str) -> str:
value : str = 'Hello'
return value + name
type annotations
variable annotation
def greeting(name: str) -> str:
value : str = 'Hello'
return value + name
type annotations
from typing import List
class A(object):
def __init__() -> None:
self.elements : List[int] = []
def add(element: int) -> None:
self.elements.append(element)
- the canonical and clean way
- packaging solved
- requires use of Python 3.6 (no Python 2, or <=3.4)
- requires importing all type dependencies (runtime penalty?)
- the interpreter needs to evaluate type hints at
syntax parsing - time consuming???
PEP-563 ~ postponed evaluation of annotations - Python 3.7
from __future__ import annotations
type comments
from typing import List
class A(object):
def __init__():
# type: () -> None
self.elements = [] # type: List[int]
def add(element):
# type: (List[int]) -> None
self.elements.append(element)
- works under any Python version
- type information is kept locally
- packaging solved
- kinda ugly, lot of noise beside logic
- unused imports
- sometime confuses linters, need to add noqa/pylint
type comments
- kinda ugly, lot of noise beside logic (6 lines)
@contextmanager
def swap_in_state(state, config, overrides):
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
- kinda ugly, lot of noise beside logic (10 lines)
@contextmanager
def swap_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
):
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
- kinda ugly, lot of noise beside logic (12 lines) + unused imports
from typing import Generator, Tuple, Optional
from magic import RunSate, HasGetSetMutable
@contextmanager
def swap_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
):
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
- sometimes conflicts with other linters
from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate
HasGetSetMutable = Union[Dict, List] # pylint: disable=invalid-name
@contextmanager
def swap_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
): # pylint: disable=bad-continuation
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
@contextmanager
def swap_in_state(state, config, overrides):
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate
HasGetSetMutable = Union[Dict, List] # pylint: disable=invalid-name
Result = Generator[ # pylint: disable=invalid-name
Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
@contextmanager
def config_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
): # pylint: disable=bad-continuation
# type: (...) -> Result
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
interface/stub files
class A(object):
def __init__() -> None:
self.elements = []
def add(element):
self.elements.append(element)
# a.pyi alongside a.py
from typing import List
class A(object):
elements = ... # type: List[int]
def __init__() -> None: ...
def add(element: int) -> None: ...
- works under any Python version
- no original source code change required
- can use latest Python features
- no conflicts with other linter tools
- well tested with typeshed
interface/stub files
class A(object):
def __init__() -> None:
self.elements = []
def add(element):
self.elements.append(element)
# a.pyi alongside a.py - ellipse for body stub
from typing import List
class A(object):
elements = ... # type: List[int]
def __init__() -> None: ...
- duplicate code - maintenance burden
- extra content to package
PEP-563 ~ Distributing and Packaging Type Information - Python 3.7 - troublesome to maintain - WIP python/mypy/pull/5139
- once specified source is ignored
- no checks for matching stub to implementation
- cannot annotate local variables
bonus: docstrings
class A(object):
def __init__():
self.elements = []
def add(element):
"""
:param List[int] element: the element to add
:rtype: None
"""
self.elements.append(element)
- works under any Python version
- does not clash with other linters (unless its a doc linter)
- no standard way to specify complex cases (union)
- tool dependent support
- requires changing the documentation
- does not play well with type hinted code
what to add?
nominal types
- all built in Python types (e.g., int, float, type, object, etc.)
- generic containers (a few examples only)
t : Tuple[int, float] = 0, 1.2 d : Dict[str, int] = {"a": 1, "b": 2} d : MutableMapping[str, int] = {"a": 1, "b": 2} l : List[int] = [1, 2, 3] i : Iterable[Text] = [ u'1', u'2', u'3'] - alias types
Vector = List[float] - distinct types
UserId = NewType('UserId', int) some_id = UserId(524313)
nominal types
- NamedTuple
class Employee(NamedTuple): name: str id: int - composer
Union[None, int, str] # one of Optional[float] # Union[None, float]
nominal types
- callable - functions
# Callable[[Arg1Type, Arg2Type], ReturnType] def feeder(get_next_item: Callable[[], str]) -> None: - generics - TypeVar
T = TypeVar('T') class Magic(Generic[T]): def __init__(self, value: T) -> None: self.value : T = value def square_values(vars: Iterable[Magic[int]]) -> None: v.value = v.value * v.value - Any - disable type check
def foo(item: Any) -> int: item.bar()
PEP-544 - protocols
nominal (main) vs structural typing (support)
KEY = TypeVar('KEY', contravariant=true)
class MagicGetter(Protocol[KEY], Sized):
def __getitem__(self, item: KEY) -> int: ...
def func_int(param: MagicGetter[int]) -> int:
return param['a'] * 2
def func_str(param: MagicGetter[str]) -> str:
return '{}'.format(param['a'])
PEP-544 - protocols
nominal (main) vs structural typing (support)
KEY = TypeVar('KEY', contravariant=true)
class MagicGetter(Protocol[KEY], Sized):
def __getitem__(self, item: KEY) -> int: ...
def func_int(param: MagicGetter[int]) -> int:
return param['a'] * 2
def func_str(param: MagicGetter[str]) -> str:
return '{}'.format(param['a'])
Gotchas
Gotcha #1 - str Python 2/3 diff
class A(object):
def __repr__(self):
# type: () -> str
return 'Rule({})'.format(self.full_name)
from __future__ import unicode_literals
class A(object):
def __repr__(self):
# type: () -> str
res = 'Rule({})'.format(self.full_name)
if sys.version_info > (3, 0):
# noinspection PyTypeChecker
return res
# noinspection PyTypeChecker
return res.encode('utf-8')
Gotcha #2 - multiple return type
def magic(i: Union[str, int]) -> Union[str, int]:
return i * 2
def other_func() -> int:
result = magic(2)
assert isinstance(result, int)
return result
Gotcha #2 - multiple return type
def magic(i: Union[str, int]) -> Any:
return i * 2
def other_func() -> int:
result = magic(2)
return result
Gotcha #2 - multiple return type
from typing import overload
@overload
def magic(i: int) -> int:
pass
@overload
def magic(i: str) -> str:
pass
def magic(i: Union[int, str]) -> Union[int, str]:
return i * 2
def other_func() -> int:
result = magic(2)
return result
Gotcha #2 - multiple return type
from typing import overload
@overload
def magic(i: int) -> int: # pylint: disable=function-redefined
pass
@overload
def magic(i: str) -> str: # pylint: disable=function-redefined
pass
def magic(i: Union[int, str]) -> Union[int, str]:
return i * 2
def other_func() -> int:
result = magic(2)
return result
Gotcha #3 - type lookup
class A(object):
def float(self):
# type: () -> float
return 1.0
test.py:3: error: Invalid type "test.A.float"
look for type in the closest namespace - 3775
Gotcha #3 - type lookup
if typing.TYPE_CHECKING:
import builtins
class A(object):
def float(self):
# type: () -> builtins.float
return 1.0
test.py:3: error: Invalid type "test.A.float"
look for type in the closest namespace - 3775
Gotcha #4 - contravariant argument
from abc import ABCMeta, abstractmethod
from typing import Union
class A(metaclass=ABCMeta):
@abstractmethod
def func(self, key): # type: (Union[int, str]) -> str
raise NotImplementedError
class B(A):
def func(self, key): # type: (int) -> str
return str(key)
class C(A):
def func(self, key): # type: (str) -> str
return key
test.py:12: error: Argument 1 of "func" incompatible with supertype "A"
test.py:17: error: Argument 1 of "func" incompatible with supertype "A"
Gotcha #4 - contravariant argument
specialization can handle more
from abc import ABCMeta, abstractmethod
from typing import Union
class A(metaclass=ABCMeta):
@abstractmethod
def func(self, key): # type: (Union[int, str]) -> str
raise NotImplementedError
class B(A):
def func(self, key): # type: (Union[int, str, bool]) -> str
return str(key)
class C(A):
def func(self, key): # type: (Union[int, str, List]) -> str
return key
Gotcha #5 - compatibility
class A:
@classmethod
def magic(cls, a: int) -> 'A':
return cls()
class B(A):
@classmethod
def magic(cls, a: int, b: bool) -> 'B':
return cls()
from typing import List, Type
elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])
print( [e.magic(1) for e in elements])
TypeError: magic() missing 1 required positional argument: 'b'
test.py:9: error: Signature of "magic" incompatible with supertype "A"
Gotcha #5 - compatibility
class A:
@classmethod
def magic(cls, a: int) -> 'A':
return cls()
class B(A):
@classmethod
def magic(cls, a: int, b: bool = False) -> 'B':
return cls()
from typing import List, Type
elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])
Gotcha #5 - compatibility
class A:
def __init__(self, a: int) -> None:
pass
class B(A):
def __init__(self, a: int, b: bool) -> None:
super().__init__(a)
from typing import List, Type
elements : List[Type[A]]= [A, B]
print( [e(1) for e in elements])
print( [e(1) for e in elements])
TypeError: __init__() missing 1 required positional argument: 'b'
- too common to prohibit incompatible __init__ and __new__
when you hit the wall
- take out the bigger hammer
- use reveal_type to see inferred type
- use cast to force a given type:
from typing import List, cast a = [4] reveal_type(a) # -> error: Revealed type is 'builtins.list[builtins.int*]' b = cast(List[int], a) # passes fine c = cast(List[str], a) # type: List[str] # passes fine (no runtime check) reveal_type(c) # -> error: Revealed type is 'builtins.list[builtins.str]' - use the # type: ignore comment to ignore an
error:
x = confusing_function() # type: ignore # see mypy/issues/1167
Merge docstring and typing?
how to document types?
- PEP-257 defines how-to document Python code
- also supports adding type information for:
- variable
- parameter
class A(object): def __init__(): self.elements = [] def add(element): """ :param List[int] element: the element to add :rtype: None """ self.elements.append(element)
sphinx-autodoc-typehints
- avoid type duplication between docstring and type hinting
- during document generation get type hinted types and insert it into the docstring
how to use it
- install the library
pip install sphinx-autodoc-types>=2.1.1 - enable it inside the conf.py
# conf.py extensions = ['sphinx_autodoc_typehints'] - generate the documentation as usual
outcome - RookieGameDevs/revived
outcome - RookieGameDevs/revived
what's next for typing
Literal types - PEP-586
@overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: pass
@overload
def open(fn: str, mode: Literal['rb', 'wb']) -> IO[bytes]: pass
def open(fn: str, mode: str):
...
with open("/etc/passwd", "r") as ft:
lines = ft.readlines() # List[Text]
with open("/bin/sh", "rb") as fb:
data = fb.read(512) # bytes
what's next for typing
Literal types - PEP-586
@overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: pass
@overload
def open(fn: str, mode: Literal['rb', 'wb']) -> IO[bytes]: pass
def open(fn: str, mode: str):
...
with open("/etc/passwd", "r") as ft:
lines = ft.readlines() # List[Text]
with open("/bin/sh", "rb") as fb:
data = fb.read(512) # bytes
what's next for typing
Literal types - PEP-586
@overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: pass
@overload
def open(fn: str, mode: Literal['rb', 'wb']) -> IO[bytes]: pass
def open(fn: str, mode: str):
...
with open("/etc/passwd", "r") as ft:
lines = ft.readlines() # List[Text]
with open("/bin/sh", "rb") as fb:
data = fb.read(512) # bytes
what's next for typing
Typed dictionaries - PEP-589
class Movie(TypedDict):
name: str
year: int
movie: Movie = {'name': 'Blade Runner', 'year': 1982}
what's next for typing
Final qualifier - PEP-591
RATE: Final = 3000
RATE = 2500 # ERROR -- cannot set Final variable
# Final class variable
class User:
DEFAULT_ID: Final = 0
class BusinessUser(User):
DEFAULT_ID = 1 # ERROR -- cannot set inherited Final class variable
what's next for typing
Final qualifier - PEP-591
RATE: Final = 3000
RATE = 2500 # ERROR -- cannot set Final variable
# Final class variable
class User:
DEFAULT_ID: Final = 0
class BusinessUser(User):
DEFAULT_ID = 1 # ERROR -- cannot set inherited Final class variable
what's next for typing
Final qualifier - PEP-591
RATE: Final = 3000
RATE = 2500 # ERROR -- cannot set Final variable
# Final class variable
class User:
DEFAULT_ID: Final = 0
class BusinessUser(User):
DEFAULT_ID = 1 # ERROR -- cannot set inherited Final class variable
what's next for typing
merge stubs and sources during check
- troublesome to maintain stubs - WIP python/mypy/pull/5139
- once specified source is ignored
- no checks for matching stub to implementation
- cannot annotate local variables
what's next for typing
numeric types
- numpy/pandas support
- exact sized arrays
- variadic type variables
- key types
what's next for typing
less camel case and imports
from typing import List
a: List[int] = [1, 2]
b : Dict[int, str] = {}
what's next for typing
less camel case and imports
a: list[int] = [1, 2]
b : dict[int, str] = {}
what's next for typing
shorter syntax
a : Union[str, int]
b : Optional[int] = {}
what's next for typing
shorter syntax
a : str | int
b : ?int = {}
what's next for typing
define unified core error codes
- now that we have 4+ type checkers
- have common error codes
- disabling errors should be cross tool compatible
- common error FAQ base
conclusion
when to use it?
- think of them as unit tests ensuring type correctness
- so use them whenever you would write unit tests
- but remember they can do much more
- checked docstring typing
- runtime type validation
- etc.
- but prepare for some discomfort and challenge sometimes
thank you
