Who am I?

In Python?

© Creative Commons CC0 1.0

Type hinting

• foundations laid down in PEP-484
• implemented by (+ reference checker - mypy)
  • Guido van Rossum - BDFL
  • Jukka Lehtosalo - mypy
  • Ivan Levkivskyi
  • Łukasz Langa
added to Python standard library with version 3.5 (improved ever since)

                    from typing import Any
                

Type hinting

• still on Python 2?
end of life at 2020 January 1

  © Copyright Chris Downer and licensed for reuse under this Creative Commons Licence.
it's available (backported) via PyPi package down to 2.7 😃

why do it?

© Creative Commons CC0 1.0

easier to

• understand (and use) the source code
• maintain
• debug

                def send_request(request_data : Any,
                 headers: Optional[Dict[str, str]],
                 user_id: Optional[UserId] = None,
                 as_json: bool = True):
    pass
            

easier to

• understand (and use) the source code
• maintain
• debug

                def send_request(request_data : Any,
                 headers: Optional[Dict[str, str]],
                 user_id: Optional[UserId] = None,
                 as_json: bool = True):
    pass
            

easier to

• understand (and use) the source code
• maintain
• debug

                def send_request(request_data : Any,
                 headers: Optional[Dict[str, str]],
                 user_id: Optional[UserId] = None,
                 as_json: bool = True):
    pass
            

easier to

• understand (and use) the source code
• maintain
• debug

                def send_request(request_data : Any,
                 headers: Optional[Dict[str, str]],
                 user_id: Optional[UserId] = None,
                 as_json: bool = True):
    pass
            

easier to

• understand (and use) the source code
• maintain
• debug

                def send_request(request_data : Any,
                 headers: Optional[Dict[str, str]],
                 user_id: Optional[UserId] = None,
                 as_json: bool = True):
    pass
            

Easier refactoring

• find usages of types
• discover call hierarchy
• better detection of objects and lifecycle flow

help editors suggestion engine

lint checks - find bugs with no tests

• IDE - PyCharm/VsCode
  
• mypy - reference implementation type checker
• pyre - Facebook
• pytype - Google
• pyright - Microsoft

improve documentation

data validation - pydantic

• a cleaner syntax to specify type requirements

                    from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {'id': '123', 'signup_ts': 'broken',
                 'friends': [1, 2, 'not a number']}
try:
    user = User(**external_data)
except ValidationError as e:
    print(e.json())
                

detect vulnerable code - security

Facebook - pyre

                    def magic_call(unsafe: UserProvidedStr, safe: str):
    subprocess.check_call([unsafe])
    subprocess.check_call([safe])
                

detect vulnerable code - security

Facebook - pyre

                    def magic_call(unsafe: UserProvidedStr, safe: str):
    subprocess.check_call([unsafe]) # error out
    subprocess.check_call([safe])
                

detect vulnerable code - security

Facebook - pyre

                    def magic_call(unsafe: UserProvidedStr, safe: str):
    subprocess.check_call([unsafe])
    subprocess.check_call([safe]) # allow
                

data validation - pydantic

• a cleaner syntax to specify type requirements

                    from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {'id': '123', 'signup_ts': 'broken',
                 'friends': [1, 2, 'not a number']}
try:
    user = User(**external_data)
except ValidationError as e:
    print(e.json())
                

data validation - pydantic

• a cleaner syntax to specify type requirements

                    from datetime import datetime
from typing import List
from pydantic import BaseModel, ValidationError

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {'id': '123', 'signup_ts': 'broken',
                 'friends': [1, 2, 'not a number']}
try:
    user = User(**external_data)
except ValidationError as e:
    print(e.json())
                

what it is not

essentially treated as comment during script evaluation

• no runtime type inference
• no performance improvement

what it is not

essentially treated as comment during script evaluation

• no runtime type inference
• no performance improvement

mypyc - speed up for free (mostly)

• compile a C extension type hinted code (not full syntax support yet)
• 4 to 20 X performance improvement (due to avoid hashtable lookups)
• mypy uses it for now, planned black

Improve the developer experience, not performance

© Creative Commons CC0 License

Improve the developer experience, not performance

© Creative Commons CC0 License

Improve the developer experience, and performance

© Creative Commons CC0 License

what kind of?

© Creative Commons CC0 1.0

Gradual typing

• no type hints specified ⇒ code dynamically typed
• but we can type hint
  • function parameters
  • function return values
  • variables
• only type hinted elements are type checked

gradual typing

• running linter will report errors (if called code is type hinted):

                # tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()
            

                bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
    expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
            

gradual typing

• running linter will report errors (if called code is type hinted):

                # tests/test_magic_field.py
f = MagicField(name=1, MagicType.DEFAULT)
f.names()
            

                bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
    expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
            

gradual typing

• running linter will report errors (if called code is type hinted):

                # tests/test_magic_field.py
f = MagicField(1, MagicType.DEFAULT)
f.names()
            

                bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
    expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
            

gradual typing

• running linter will report errors (if called code is type hinted):

                # tests/test_magic_field.py
f = MagicField(1, MagicType.DEFAULT)
f.names()
            

                bernat@uvm ~/python-magic (master●)$ mypy --ignore-missing-imports tests/test_magic_field.py
tests/test_magic_field.py:21: error: Argument 1 to "MagicField" has incompatible type "int";
    expected "Union[str, bytes]"
tests/test_magic_field.py:22: error: "MagicField" has no attribute "names"; maybe "name" or "_name"?
            

How to add it

© Creative Commons CC0 License

type annotations

syntax based on

• mypy
• PEP-3107 ~ function annotations - Python 3.0
• PEP-526 ~ variable annotations - Python 3.6

type annotations

for example

                    def greeting(name: str) -> str:
    value : str = 'Hello'
    return value + name
                

type annotations

function annotation

                    def greeting(name: str) -> str:
    value : str = 'Hello'
    return value + name
                

type annotations

variable annotation

                    def greeting(name: str) -> str:
    value : str = 'Hello'
    return value + name
                

type annotations

                from typing import List

class A(object):
    def __init__() -> None:
         self.elements : List[int] = []

   def add(element: int) -> None:
         self.elements.append(element)
            

• the canonical and clean way
• packaging solved

• requires use of Python 3.6 (no Python 2, or <=3.4)
• requires importing all type dependencies (runtime penalty?)
• the interpreter needs to evaluate type hints at syntax parsing - time consuming???
  PEP-563 ~ postponed evaluation of annotations - Python 3.7

                  from __future__ import annotations
              

type comments

                from typing import List

class A(object):
    def __init__():
         # type: () -> None
         self.elements = []  # type: List[int]

   def add(element):
         # type: (List[int]) -> None
         self.elements.append(element)
            

• works under any Python version
• type information is kept locally
• packaging solved

• kinda ugly, lot of noise beside logic
• unused imports
• sometime confuses linters, need to add noqa/pylint

type comments

• kinda ugly, lot of noise beside logic (6 lines)

                

@contextmanager
def swap_in_state(state, config, overrides):




    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides
        

type comments

• kinda ugly, lot of noise beside logic (10 lines)

                

@contextmanager
def swap_in_state(state,  # type: State
                  config,  # type: HasGetSetMutable
                  overrides  # type: Optional[HasGetSetMutable]
                 ):
# type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides
            

type comments

• kinda ugly, lot of noise beside logic (12 lines) + unused imports

                from typing import Generator, Tuple, Optional
from magic import RunSate, HasGetSetMutable

@contextmanager
def swap_in_state(state,  # type: State
                  config,  # type: HasGetSetMutable
                  overrides  # type: Optional[HasGetSetMutable]
                 ):
    # type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides
        

type comments

• sometimes conflicts with other linters

                from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate

HasGetSetMutable = Union[Dict, List]  # pylint: disable=invalid-name

@contextmanager
def swap_in_state(state,  # type: State
                  config,  # type: HasGetSetMutable
                  overrides  # type: Optional[HasGetSetMutable]
                 ):  # pylint: disable=bad-continuation
    # type: (...) -> Generator[Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides
            

type comments

                @contextmanager
def swap_in_state(state, config, overrides):
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides
            

                from typing import Generator, Tuple, Optional, Dict, List
from magic import RunSate

HasGetSetMutable = Union[Dict, List]  # pylint: disable=invalid-name
Result = Generator[   # pylint: disable=invalid-name
    Tuple[HasGetSetMutable, Optional[HasGetSetMutable]], None, None]

@contextmanager
def config_in_state(state,  # type: State
                   config,  # type: HasGetSetMutable
                   overrides  # type: Optional[HasGetSetMutable]
                    ):  # pylint: disable=bad-continuation
    # type: (...) -> Result
    old_config, old_overrides = state.config, state.overrides
    state.config, state.overrides = config, overrides
    yield old_config, old_overrides
    state.config, state.overrides = old_config, old_overrides
            

interface/stub files

                class A(object):
  def __init__() -> None:
      self.elements = []

  def add(element):
      self.elements.append(element)
            

                # a.pyi alongside a.py
from typing import List

class A(object):
  elements = ... # type: List[int]
  def __init__() -> None: ...
  def add(element: int) -> None: ...
            

• works under any Python version
• no original source code change required
• can use latest Python features
• no conflicts with other linter tools
• well tested with typeshed

interface/stub files

                class A(object):
  def __init__() -> None:
      self.elements = []

  def add(element):
      self.elements.append(element)
            

                # a.pyi alongside a.py - ellipse for body stub
from typing import List

class A(object):
  elements = ... # type: List[int]
  def __init__() -> None: ...
    
            

• duplicate code - maintenance burden
• extra content to package
  PEP-563 ~ Distributing and Packaging Type Information - Python 3.7
• troublesome to maintain - WIP python/mypy/pull/5139
  • once specified source is ignored
  • no checks for matching stub to implementation
  • cannot annotate local variables

bonus: docstrings

                class A(object):
    def __init__():
         self.elements = []

   def add(element):
       """
       :param List[int] element: the element to add
        :rtype: None
       """
       self.elements.append(element)
            

• works under any Python version
• does not clash with other linters (unless its a doc linter)

• no standard way to specify complex cases (union)
• tool dependent support
• requires changing the documentation
• does not play well with type hinted code

what to add?

© Creative Commons License

nominal types

• all built in Python types (e.g., int, float, type, object, etc.)
• generic containers (a few examples only)

                      t : Tuple[int, float] = 0, 1.2
  d : Dict[str, int] = {"a": 1, "b": 2}
  d : MutableMapping[str, int] = {"a": 1, "b": 2}
  l : List[int] = [1, 2, 3]
  i : Iterable[Text] = [ u'1', u'2', u'3']
                      

• alias types

                      Vector = List[float]
                      

• distinct types

                      UserId = NewType('UserId', int)
  some_id = UserId(524313)
                      

nominal types

• NamedTuple

                      class Employee(NamedTuple):
       name: str
       id: int
                      

• composer

                      Union[None, int, str] # one of
  Optional[float] # Union[None, float]
                      

nominal types

• callable - functions

                      # Callable[[Arg1Type, Arg2Type], ReturnType]
  def feeder(get_next_item: Callable[[], str]) -> None:
                      

• generics - TypeVar

                      T = TypeVar('T')
  class Magic(Generic[T]):
        def __init__(self, value: T) -> None:
           self.value : T = value
  
   def square_values(vars: Iterable[Magic[int]]) -> None:
       v.value = v.value * v.value
                      

• Any - disable type check

                      def foo(item: Any) -> int:
       item.bar()
                  

PEP-544 - protocols

nominal (main) vs structural typing (support)

                KEY = TypeVar('KEY', contravariant=true)

class MagicGetter(Protocol[KEY], Sized):
        def __getitem__(self, item: KEY) -> int: ...

def func_int(param: MagicGetter[int]) -> int:
    return param['a'] * 2

def func_str(param: MagicGetter[str]) -> str:
    return '{}'.format(param['a'])
            

PEP-544 - protocols

nominal (main) vs structural typing (support)

                KEY = TypeVar('KEY', contravariant=true)

class MagicGetter(Protocol[KEY], Sized):
        def __getitem__(self, item: KEY) -> int: ...

def func_int(param: MagicGetter[int]) -> int:
    return param['a'] * 2

def func_str(param: MagicGetter[str]) -> str:
    return '{}'.format(param['a'])
            

Gotchas

© Creative Commons CC 2.0 License

Gotcha #1 - str Python 2/3 diff

                class A(object):
    def __repr__(self):
        # type: () -> str
        return 'Rule({})'.format(self.full_name)
            

                from __future__ import unicode_literals

class A(object):
    def __repr__(self):
        # type: () -> str
        res = 'Rule({})'.format(self.full_name)
        if sys.version_info > (3, 0):
            # noinspection PyTypeChecker
            return res
        # noinspection PyTypeChecker
        return res.encode('utf-8')
            

Gotcha #2 - multiple return type

                def magic(i: Union[str, int]) -> Union[str, int]:
    return i * 2
            

                def other_func() -> int:
    result = magic(2)
    assert isinstance(result, int)
    return result
            

Gotcha #2 - multiple return type

                def magic(i: Union[str, int]) -> Any:
    return i * 2
            

                def other_func() -> int:
    result = magic(2)

    return result
            

Gotcha #2 - multiple return type

                from typing import overload

@overload
def magic(i: int) -> int:
    pass

@overload
def magic(i: str) -> str:
    pass

def magic(i: Union[int, str]) -> Union[int, str]:
    return i * 2
            

                def other_func() -> int:
    result = magic(2)

    return result
            

Gotcha #2 - multiple return type

                from typing import overload

@overload
def magic(i: int) -> int:  # pylint: disable=function-redefined
    pass

@overload
def magic(i: str) -> str:  # pylint: disable=function-redefined
    pass

def magic(i: Union[int, str]) -> Union[int, str]:
    return i * 2
            

                def other_func() -> int:
    result = magic(2)

    return result
            

Gotcha #3 - type lookup

                class A(object):

    def float(self):
            # type: () -> float
           return 1.0
            

                test.py:3: error: Invalid type "test.A.float"
            

look for type in the closest namespace - 3775

Gotcha #3 - type lookup

                if typing.TYPE_CHECKING:
    import builtins

class A(object):

    def float(self):
            # type: () -> builtins.float
           return 1.0
            

                test.py:3: error: Invalid type "test.A.float"
            

look for type in the closest namespace - 3775

Gotcha #4 - contravariant argument

                from abc import ABCMeta, abstractmethod
from typing import Union

class A(metaclass=ABCMeta):
    @abstractmethod
    def func(self, key):  # type: (Union[int, str]) -> str
        raise NotImplementedError

class B(A):
    def func(self, key):  # type: (int) -> str
        return str(key)

class C(A):
    def func(self, key):  # type: (str) -> str
        return key
            

                test.py:12: error: Argument 1 of "func" incompatible with supertype "A"
test.py:17: error: Argument 1 of "func" incompatible with supertype "A"
            

Gotcha #4 - contravariant argument

specialization can handle more

                from abc import ABCMeta, abstractmethod
from typing import Union

class A(metaclass=ABCMeta):
    @abstractmethod
    def func(self, key):  # type: (Union[int, str]) -> str
        raise NotImplementedError

class B(A):
    def func(self, key):  # type: (Union[int, str, bool]) -> str
        return str(key)

class C(A):
    def func(self, key):  # type: (Union[int, str, List]) -> str
        return key
            

Gotcha #5 - compatibility

                class A:
    @classmethod
    def magic(cls, a: int) -> 'A':
        return cls()

class B(A):
    @classmethod
    def magic(cls, a: int, b: bool) -> 'B':
        return cls()
            

                from typing import List, Type

elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])
            

                    print( [e.magic(1) for e in elements])
TypeError: magic() missing 1 required positional argument: 'b'
            

                test.py:9: error: Signature of "magic" incompatible with supertype "A"
            

Gotcha #5 - compatibility

                class A:
    @classmethod
    def magic(cls, a: int) -> 'A':
        return cls()

class B(A):
    @classmethod
    def magic(cls, a: int, b: bool = False) -> 'B':
        return cls()
            

                from typing import List, Type

elements : List[Type[A]] = [A, B]
print( [e.magic(1) for e in elements])
            

Gotcha #5 - compatibility

                class A:
    def __init__(self, a: int) -> None:
        pass

class B(A):
    def __init__(self, a: int, b: bool) -> None:
        super().__init__(a)
            

                from typing import List, Type

elements : List[Type[A]]= [A, B]
print( [e(1) for e in elements])
            

                    print( [e(1) for e in elements])
TypeError: __init__() missing 1 required positional argument: 'b'
            

• too common to prohibit incompatible __init__ and __new__

when you hit the wall

• take out the bigger hammer
• use reveal_type to see inferred type
• use cast to force a given type:

                  from typing import List, cast
  
  a = [4]
  reveal_type(a)         # -> error: Revealed type is 'builtins.list[builtins.int*]'
  
  b = cast(List[int], a) # passes fine
  c = cast(List[str], a) # type: List[str] # passes fine (no runtime check)
  reveal_type(c)         # -> error: Revealed type is 'builtins.list[builtins.str]'
              

• use the # type: ignore comment to ignore an error:

                  x = confusing_function() # type: ignore # see mypy/issues/1167
              

Merge docstring and typing?

how to document types?

• PEP-257 defines how-to document Python code
• also supports adding type information for:
  • variable
  • parameter

                  class A(object):
      def __init__():
           self.elements = []
  
     def add(element):
         """
         :param List[int] element: the element to add
          :rtype: None
         """
         self.elements.append(element)
              

sphinx-autodoc-typehints

• avoid type duplication between docstring and type hinting
• during document generation get type hinted types and insert it into the docstring

how to use it

• install the library

                  pip install sphinx-autodoc-types>=2.1.1
              

• enable it inside the conf.py

                  # conf.py
  extensions = ['sphinx_autodoc_typehints']
              

• generate the documentation as usual

what's next for typing

                @overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: pass
@overload
def open(fn: str, mode: Literal['rb', 'wb']) -> IO[bytes]: pass
def open(fn: str, mode: str):
    ...

with open("/etc/passwd", "r") as ft:
    lines = ft.readlines()  # List[Text]

with open("/bin/sh", "rb") as fb:
    data = fb.read(512) # bytes
            

what's next for typing

                @overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: pass
@overload
def open(fn: str, mode: Literal['rb', 'wb']) -> IO[bytes]: pass
def open(fn: str, mode: str):
    ...

with open("/etc/passwd", "r") as ft:
    lines = ft.readlines()  # List[Text]

with open("/bin/sh", "rb") as fb:
    data = fb.read(512) # bytes
            

what's next for typing

                @overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: pass
@overload
def open(fn: str, mode: Literal['rb', 'wb']) -> IO[bytes]: pass
def open(fn: str, mode: str):
    ...

with open("/etc/passwd", "r") as ft:
    lines = ft.readlines()  # List[Text]

with open("/bin/sh", "rb") as fb:
    data = fb.read(512) # bytes
            

what's next for typing

                class Movie(TypedDict):
    name: str
    year: int

movie: Movie = {'name': 'Blade Runner', 'year': 1982}
            

what's next for typing

                RATE: Final = 3000
RATE = 2500  # ERROR -- cannot set Final variable

# Final class variable
class User:
    DEFAULT_ID: Final = 0
class BusinessUser(User):
    DEFAULT_ID = 1  # ERROR -- cannot set inherited Final class variable
            

what's next for typing

                RATE: Final = 3000
RATE = 2500  # ERROR -- cannot set Final variable

# Final class variable
class User:
    DEFAULT_ID: Final = 0
class BusinessUser(User):
    DEFAULT_ID = 1  # ERROR -- cannot set inherited Final class variable
            

what's next for typing

                RATE: Final = 3000
RATE = 2500  # ERROR -- cannot set Final variable

# Final class variable
class User:
    DEFAULT_ID: Final = 0
class BusinessUser(User):
    DEFAULT_ID = 1  # ERROR -- cannot set inherited Final class variable
            

what's next for typing

merge stubs and sources during check

• troublesome to maintain stubs - WIP python/mypy/pull/5139
  • once specified source is ignored
  • no checks for matching stub to implementation
  • cannot annotate local variables

what's next for typing

numeric types

• numpy/pandas support
  • exact sized arrays
  • variadic type variables
  • key types

what's next for typing

less camel case and imports

                from typing import List

a: List[int] = [1, 2]
b : Dict[int, str] = {}
            

what's next for typing

less camel case and imports

                

a: list[int] = [1, 2]
b : dict[int, str] = {}
            

what's next for typing

shorter syntax

                a : Union[str, int]
b : Optional[int] = {}
            

what's next for typing

shorter syntax

                a : str | int
b : ?int = {}
            

what's next for typing

define unified core error codes

• now that we have 4+ type checkers
  • have common error codes
  • disabling errors should be cross tool compatible
  • common error FAQ base

conclusion

when to use it?

• think of them as unit tests ensuring type correctness
• so use them whenever you would write unit tests
• but remember they can do much more
  • checked docstring typing
  • runtime type validation
  • etc.
• but prepare for some discomfort and challenge sometimes

thank you

https://www.bernat.tech/the-state-of-type-hints-in-python/

© Creative Commons Attribution-Share Alike 2.5 Generic

we're hiring

https://bloomberg.com/engineering

see TechAtBloomberg.com / @TechAtBloomberg