Type hints
inside the snake pit
by Bernat Gabor /
@gjbernat /
Bloomberg
In Python?
.jpg){kind=link}
Type hinting
- foundations laid down in PEP-484
- implemented by
- Guido van Rossum - BDFL
- Jukka Lehtosalo - mypy
- Ivan Levkivskyi
- Łukasz Langa
- added to Python standard library with version 3.5
from typing import Any
Type hinting
- still on Python 2?
- end of life at 2020 January 1
- it's available via PyPi package down to 2.7 😃
why do it?
easier to
- debug
- maintain
- understand (use)
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- debug
- maintain
- understand (use)
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- debug
- maintain
- understand (use)
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- debug
- maintain
- understand (use)
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
easier to
- debug
- maintain
- understand (use)
def send_request(request_data : Any,
headers: Optional[Dict[str, str]],
user_id: Optional[UserId] = None,
as_json: bool = True):
pass
Easier refactoring
- find usages of types
- better detection of objects
help editors suggestion engine
lint checks - find bugs - mypy
improve documentation
data validation - pydantic
- a cleaner syntax to specify type requirements
from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError
class User(BaseModel):
id: int
name = 'John Doe'
signup_ts: datetime = None
friends: List[int] = []
external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22',
'friends': [1, 2, 3]}
user = User(**external_data)
try:
User(signup_ts='broken', friends=[1, 2, 'not number'])
except ValidationError as e:
print(e.json())
what it is not
- no runtime type inference
- no performance improvement
- treated as comment during script evaluation
Improve the developer experience, not performance
what kind of?
Gradual typing
- no type hints specified ⇒ code dynamically typed
- but we can type hint
- function parameters
- function return values
- variables
- only type hinted elements are type checked
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
gradual typing
- running linter will report errors (if called code is type hinted):
# tests/test_magic_field.py
f = MagicField(1, MagicType.DEFAULT)
f.names()
bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
How to add it
type annotations
syntax based on
- mypy
- PEP-3107 ~ function annotations - Python 3.0
- PEP-526 ~ variable annotations - Python 3.6
type annotations
for example
def greeting(name: str) -> str:
value : str = 'Hello'
return value + name
type annotations
function annotation
def greeting(name: str) -> str:
value : str = 'Hello'
return value + name
type annotations
variable annotation
def greeting(name: str) -> str:
value : str = 'Hello'
return value + name
type annotations
from typing import List
class A(object):
def __init__() -> None:
self.elements : List[int] = []
def add(element: int) -> None:
self.elements.append(element)
- the canonical and clean way
- packaging solved
- requires use of Python 3.6 (no Python 2, or <=3.4)
- requires importing all type dependencies (runtime penalty?)
- the interpreter needs to evaluate type hints at
syntax parsing - time consuming???
PEP-563 ~ postponed evaluation of annotations - Python 3.7
from __future__ import annotations
type comments
from typing import List
class A(object):
def __init__():
# type: () -> None
self.elements = [] # type: List[int]
def add(element):
# type: (List[int]) -> None
self.elements.append(element)
- works under any Python version
- type information is kept locally
- packaging solved
- kinda ugly, lot of noise beside logic
- unused imports
- often conflicts with other linter comments, noqa/pylint
type comments
- kinda ugly, lot of noise beside logic
@contextmanager
def swap_in_state(state, config, overrides):
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
- kinda ugly, lot of noise beside logic
@contextmanager
def swap_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
):
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
- kinda ugly, lot of noise beside logic
- unused imports
from typing import Generator, Tuple, Optional
from magic import RunSate, HasGetSetMutable
@contextmanager
def swap_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
):
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
- often conflicts with other linters
from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate
HasGetSetMutable = Union[Dict, List] # pylint: disable=invalid-name
@contextmanager
def swap_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
): # pylint: disable=bad-continuation
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
type comments
@contextmanager
def swap_in_state(state, config, overrides):
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate
HasGetSetMutable = Union[Dict, List] # pylint: disable=invalid-name
@contextmanager
def config_in_state(state, # type: State
config, # type: HasGetSetMutable
overrides # type: Optional[HasGetSetMutable]
): # pylint: disable=bad-continuation
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
old_config, old_overrides = state.config, state.overrides
state.config, state.overrides = config, overrides
yield old_config, old_overrides
state.config, state.overrides = old_config, old_overrides
interface/stub files
class A(object):
def __init__() -> None:
self.elements = []
def add(element):
self.elements.append(element)
# a.pyi alongside a.py
from typing import List
class A(object):
elements = ... # type: List[int]
def __init__() -> None: ...
def add(element: int) -> None: ...
- works under any Python version
- no original source code change required
- can use latest Python features
- no conflicts with other linter tools
- well tested with typeshed
interface/stub files
class A(object):
def __init__() -> None:
self.elements = []
def add(element):
self.elements.append(element)
# a.pyi alongside a.py
from typing import List
class A(object):
elements = ... # type: List[int]
def __init__() -> None: ...
- duplicate code - maintenance burden
- extra content to package
PEP-563 ~ Distributing and Packaging Type Information - Python 3.7 - once specified source is ignored
- no checks for matching stub to implementation
- cannot annotate local variables
WIP python/mypy/pull/5139 - merge stubs into source trees
bonus: docstrings
class A(object):
def __init__():
self.elements = []
def add(element):
"""
:param List[int] element: the element to add
:rtype: None
"""
self.elements.append(element)
- works under any Python version
- does not clash with other linters
- no standard way to specify complex cases (union)
- tool dependent support
- requires changing the documentation
- does not play well with type hinted code
what to add?
nominal types
- all built in Python types (e.g., int, float, type, object, etc.)
- generic containers (a few examples only)
t : Tuple[int, float] = 0, 1.2 d : Dict[str, int] = {"a": 1, "b": 2} d : MutableMapping[str, int] = {"a": 1, "b": 2} l : List[int] = [1, 2, 3] i : Iterable[Text] = [ u'1', u'2', u'3'] - alias types
Vector = List[float] - distinct types
UserId = NewType('UserId', int) some_id = UserId(524313)
nominal types
- NamedTuple
class Employee(NamedTuple): name: str id: int - composer
Union[None, int, str] # one of Optional[float] # Union[None, float]
nominal types
- callable - functions
# Callable[[Arg1Type, Arg2Type], ReturnType] def feeder(get_next_item: Callable[[], str]) -> None: - generics - TypeVar
T = TypeVar('T') class Magic(Generic[T]): def __init__(self, value: T) -> None: self.value : T = value def square_values(vars: Iterable[Magic[int]]) -> None: v.value = v.value * v.value - Any - disable type check
def foo(item: Any) -> int: item.bar()
PEP-544 - protocols
nominal (main) vs structural typing (support)
KEY = TypeVar('KEY', contravariant=true)
class MagicGetter(Protocol[KEY], Sized):
def __getitem__(self, item: KEY) -> int: ...
def func_int(param: MagicGetter[int]) -> int:
return param['a'] * 2
def func_str(param: MagicGetter[str]) -> str:
return '{}'.format(param['a'])
PEP-544 - protocols
nominal (main) vs structural typing (support)
KEY = TypeVar('KEY', contravariant=true)
class MagicGetter(Protocol[KEY], Sized):
def __getitem__(self, item: KEY) -> int: ...
def func_int(param: MagicGetter[int]) -> int:
return param['a'] * 2
def func_str(param: MagicGetter[str]) -> str:
return '{}'.format(param['a'])
Gotchas
Gotcha #1 - str Python 2/3 diff
class A(object):
def __repr__(self):
# type: () -> str
return 'A({})'.format(self.full_name)
from __future__ import unicode_literals
class A(object):
def __repr__(self):
# type: () -> str
res = 'A({})'.format(self.full_name)
if sys.version_info > (3, 0):
# noinspection PyTypeChecker
return res
# noinspection PyTypeChecker
return res.encode('utf-8')
Gotcha #2 - multiple return type
def magic(i: Union[str, int]) -> Union[str, int]:
return i * 2
def other_func() -> int:
result = magic(2)
assert isinstance(result, int)
return result
Gotcha #2 - multiple return type
def magic(i: Union[str, int]) -> Any:
return i * 2
def other_func() -> int:
result = magic(2)
return result
Gotcha #2 - multiple return type
from typing import overload
@overload
def magic(i: int) -> int:
pass
@overload
def magic(i: str) -> str:
pass
def magic(i: Union[int, str]) -> Union[int, str]:
return i * 2
def other_func() -> int:
result = magic(2)
return result
Gotcha #2 - multiple return type
from typing import overload
@overload
def magic(i: int) -> int: # pylint: disable=function-redefined
pass
@overload
def magic(i: str) -> str: # pylint: disable=function-redefined
pass
def magic(i: Union[int, str]) -> Union[int, str]:
return i * 2
def other_func() -> int:
result = magic(2)
return result
Gotcha #3 - type lookup
class A(object):
def float(self):
# type: () -> float
return 1.0
test.py:3: error: Invalid type "test.A.float"
look for type in the closest namespace - 3775
Gotcha #3 - type lookup
if typing.TYPE_CHECKING:
import builtins
class A(object):
def float(self):
# type: () -> builtins.float
return 1.0
test.py:3: error: Invalid type "test.A.float"
look for type in the closest namespace - 3775
Gotcha #4 - contravariant argument
from abc import ABCMeta, abstractmethod
from typing import Union
class A(metaclass=ABCMeta):
@abstractmethod
def func(self, key): # type: (Union[int, str]) -> str
raise NotImplementedError
class B(A):
def func(self, key): # type: (int) -> str
return str(key)
class C(A):
def func(self, key): # type: (str) -> str
return key
test.py:12: error: Argument 1 of "func" incompatible with supertype "A"
test.py:17: error: Argument 1 of "func" incompatible with supertype "A"
Gotcha #4 - contravariant argument
specialization can handle more
from abc import ABCMeta, abstractmethod
from typing import Union
class A(metaclass=ABCMeta):
@abstractmethod
def func(self, key): # type: (Union[int, str]) -> str
raise NotImplementedError
class B(A):
def func(self, key): # type: (Union[int, str, bool]) -> str
return str(key)
class C(A):
def func(self, key): # type: (Union[int, str, List]) -> str
return key
Gotcha #5 - compatibility
class A:
@classmethod
def magic(cls, a: int) -> 'A':
return cls()
class B(A):
@classmethod
def magic(cls, a: int, b: bool) -> 'B':
return cls()
from typing import List, Type
elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])
print( [e.magic(1) for e in elements])
TypeError: magic() missing 1 required positional argument: 'b'
test.py:9: error: Signature of "magic" incompatible with supertype "A"
Gotcha #5 - compatibility
class A:
@classmethod
def magic(cls, a: int) -> 'A':
return cls()
class B(A):
@classmethod
def magic(cls, a: int, b: bool = False) -> 'B':
return cls()
from typing import List, Type
elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])
Gotcha #5 - compatibility
class A:
def __init__(self, a: int) -> None:
pass
class B(A):
def __init__(self, a: int, b: bool) -> None:
super().__init__(a)
from typing import List, Type
elements : List[Type[A]]= [A, B]
print( [e(1) for e in elements])
print( [e(1) for e in elements])
TypeError: __init__() missing 1 required positional argument: 'b'
- too common to prohibit incompatible __init__ and __new__
when you hit the wall
- take out the bigger hammer
- use reveal_type to see inferred type
- use cast to force a given type:
from typing import List, cast a = [4] reveal_type(a) # -> error: Revealed type is 'builtins.list[builtins.int*]' b = cast(List[int], a) # passes fine c = cast(List[str], a) # type: List[str] # passes fine (no runtime check) reveal_type(c) # -> error: Revealed type is 'builtins.list[builtins.str]' - use the # type: ignore comment to ignore an
error:
x = confusing_function() # type: ignore # see mypy/issues/1167
Merge docstring and typing?
how to document types?
- PEP-257 defines how-to document Python code
- also supports adding type information for:
- variable
- parameter
class A(object): def __init__(): self.elements = [] def add(element): """ :param List[int] element: the element to add :rtype: None """ self.elements.append(element)
creating human readable pages
- Sphinx is the most common tool used
- can generate many outputs:
- html
- Apple help
- Epub
- Latex
- text (stripped markdown)
- supports plugins to add runtime transformations on the base docstring
sphinx-autodoc-typehints
- avoid type duplication between docstring and type hinting
- during document generation get type hinted types and insert it into the docstring
how to use it
- install the library
pip install sphinx-autodoc-types>=2.1.1 - enable it inside the conf.py
# conf.py extensions = ['sphinx_autodoc_typehints'] - generate the documentation as usual
outcome - RookieGameDevs/revived
what's next for mypy
- performance
what's next for mypy
-
performance- dmypy released in May - define an API and plugin system
- enrich typeshed with more popular libraries
- ability to merge stub type annotations into source trees
conclusion - when to use it?
- think of them as unit tests ensuring type correctness
- so use them whenever you would write unit tests
- but remember they can do much more
- checked docstring typing
- runtime type validation
- etc.
thank you
