Use mypy 1.0 (#15052)

* Update mypy and mypy-zope
* Remove unused ignores

These used to suppress

```
synapse/storage/engines/__init__.py:28: error: "__new__" must return a
class instance (got "NoReturn")  [misc]
```

and

```
synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons"  [attr-defined]
```

(note that we check `hasattr(e, "reasons")` above)

* Avoid empty body warnings, sometimes by marking methods as abstract

E.g.

```
tests/handlers/test_register.py:58: error: Missing return statement  [empty-body]
tests/handlers/test_register.py:108: error: Missing return statement  [empty-body]
```

* Suppress false positive about `JaegerConfig`

Complaint was

```
synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context  [truthy-function]
```

* Fix not calling `is_state()`

Oops!

```
tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context  [truthy-function]
```

* Suppress false positives from ParamSpecs

````
synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
````

* Drive-by improvement to `wrapping_logic` annotation

* Workaround false "unreachable" positives

See https://github.com/Shoobx/mypy-zope/issues/91

```
tests/http/test_proxyagent.py:626: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:762: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:826: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:838: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:845: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:60: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:93: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:127: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:152: error: Statement is unreachable  [unreachable]
```

* Changelog

* Tweak DBAPI2 Protocol to be accepted by mypy 1.0

Some extra context in:
- https://github.com/matrix-org/python-canonicaljson/pull/57
- https://github.com/python/mypy/issues/6002
- https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected

* Pull in updated canonicaljson lib

so the protocol check just works

* Improve comments in opentracing

I tried to workaround the ignores but found it too much trouble.

I think the corresponding issue is
https://github.com/python/mypy/issues/12909. The mypy repo has a PR
claiming to fix this (https://github.com/python/mypy/pull/14677) which
might mean this gets resolved soon?

* Better annotation for INTERACTIVE_AUTH_CHECKERS

* Drive-by AUTH_TYPE annotation, to remove an ignore
This commit is contained in:
David Robertson 2023-02-16 16:09:11 +00:00 committed by GitHub
parent 979f237b28
commit ffc2ee521d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
17 changed files with 209 additions and 104 deletions

1
changelog.d/15052.misc Normal file
View file

@ -0,0 +1 @@
Improve type hints.

69
poetry.lock generated
View file

@ -146,14 +146,14 @@ css = ["tinycss2 (>=1.1.0,<1.2)"]
[[package]] [[package]]
name = "canonicaljson" name = "canonicaljson"
version = "1.6.4" version = "1.6.5"
description = "Canonical JSON" description = "Canonical JSON"
category = "main" category = "main"
optional = false optional = false
python-versions = ">=3.7" python-versions = ">=3.7"
files = [ files = [
{file = "canonicaljson-1.6.4-py3-none-any.whl", hash = "sha256:55d282853b4245dbcd953fe54c39b91571813d7c44e1dbf66e3c4f97ff134a48"}, {file = "canonicaljson-1.6.5-py3-none-any.whl", hash = "sha256:806ea6f2cbb7405d20259e1c36dd1214ba5c242fa9165f5bd0bf2081f82c23fb"},
{file = "canonicaljson-1.6.4.tar.gz", hash = "sha256:6c09b2119511f30eb1126cfcd973a10824e20f1cfd25039cde3d1218dd9c8d8f"}, {file = "canonicaljson-1.6.5.tar.gz", hash = "sha256:68dfc157b011e07d94bf74b5d4ccc01958584ed942d9dfd5fdd706609e81cd4b"},
] ]
[package.dependencies] [package.dependencies]
@ -1146,36 +1146,38 @@ files = [
[[package]] [[package]]
name = "mypy" name = "mypy"
version = "0.981" version = "1.0.0"
description = "Optional static typing for Python" description = "Optional static typing for Python"
category = "dev" category = "dev"
optional = false optional = false
python-versions = ">=3.7" python-versions = ">=3.7"
files = [ files = [
{file = "mypy-0.981-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:4bc460e43b7785f78862dab78674e62ec3cd523485baecfdf81a555ed29ecfa0"}, {file = "mypy-1.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e0626db16705ab9f7fa6c249c017c887baf20738ce7f9129da162bb3075fc1af"},
{file = "mypy-0.981-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:756fad8b263b3ba39e4e204ee53042671b660c36c9017412b43af210ddee7b08"}, {file = "mypy-1.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:1ace23f6bb4aec4604b86c4843276e8fa548d667dbbd0cb83a3ae14b18b2db6c"},
{file = "mypy-0.981-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a16a0145d6d7d00fbede2da3a3096dcc9ecea091adfa8da48fa6a7b75d35562d"}, {file = "mypy-1.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:87edfaf344c9401942883fad030909116aa77b0fa7e6e8e1c5407e14549afe9a"},
{file = "mypy-0.981-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce65f70b14a21fdac84c294cde75e6dbdabbcff22975335e20827b3b94bdbf49"}, {file = "mypy-1.0.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:0ab090d9240d6b4e99e1fa998c2d0aa5b29fc0fb06bd30e7ad6183c95fa07593"},
{file = "mypy-0.981-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:6e35d764784b42c3e256848fb8ed1d4292c9fc0098413adb28d84974c095b279"}, {file = "mypy-1.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:7cc2c01dfc5a3cbddfa6c13f530ef3b95292f926329929001d45e124342cd6b7"},
{file = "mypy-0.981-cp310-cp310-win_amd64.whl", hash = "sha256:e53773073c864d5f5cec7f3fc72fbbcef65410cde8cc18d4f7242dea60dac52e"}, {file = "mypy-1.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:14d776869a3e6c89c17eb943100f7868f677703c8a4e00b3803918f86aafbc52"},
{file = "mypy-0.981-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:6ee196b1d10b8b215e835f438e06965d7a480f6fe016eddbc285f13955cca659"}, {file = "mypy-1.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bb2782a036d9eb6b5a6efcdda0986774bf798beef86a62da86cb73e2a10b423d"},
{file = "mypy-0.981-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8ad21d4c9d3673726cf986ea1d0c9fb66905258709550ddf7944c8f885f208be"}, {file = "mypy-1.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5cfca124f0ac6707747544c127880893ad72a656e136adc935c8600740b21ff5"},
{file = "mypy-0.981-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:d1debb09043e1f5ee845fa1e96d180e89115b30e47c5d3ce53bc967bab53f62d"}, {file = "mypy-1.0.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:8845125d0b7c57838a10fd8925b0f5f709d0e08568ce587cc862aacce453e3dd"},
{file = "mypy-0.981-cp37-cp37m-win_amd64.whl", hash = "sha256:9f362470a3480165c4c6151786b5379351b790d56952005be18bdbdd4c7ce0ae"}, {file = "mypy-1.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b1b9e1ed40544ef486fa8ac022232ccc57109f379611633ede8e71630d07d2"},
{file = "mypy-0.981-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:c9e0efb95ed6ca1654951bd5ec2f3fa91b295d78bf6527e026529d4aaa1e0c30"}, {file = "mypy-1.0.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:c7cf862aef988b5fbaa17764ad1d21b4831436701c7d2b653156a9497d92c83c"},
{file = "mypy-0.981-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:e178eaffc3c5cd211a87965c8c0df6da91ed7d258b5fc72b8e047c3771317ddb"}, {file = "mypy-1.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5cd187d92b6939617f1168a4fe68f68add749902c010e66fe574c165c742ed88"},
{file = "mypy-0.981-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:06e1eac8d99bd404ed8dd34ca29673c4346e76dd8e612ea507763dccd7e13c7a"}, {file = "mypy-1.0.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:4e5175026618c178dfba6188228b845b64131034ab3ba52acaffa8f6c361f805"},
{file = "mypy-0.981-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fa38f82f53e1e7beb45557ff167c177802ba7b387ad017eab1663d567017c8ee"}, {file = "mypy-1.0.0-cp37-cp37m-win_amd64.whl", hash = "sha256:2f6ac8c87e046dc18c7d1d7f6653a66787a4555085b056fe2d599f1f1a2a2d21"},
{file = "mypy-0.981-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:64e1f6af81c003f85f0dfed52db632817dabb51b65c0318ffbf5ff51995bbb08"}, {file = "mypy-1.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:7306edca1c6f1b5fa0bc9aa645e6ac8393014fa82d0fa180d0ebc990ebe15964"},
{file = "mypy-0.981-cp38-cp38-win_amd64.whl", hash = "sha256:e1acf62a8c4f7c092462c738aa2c2489e275ed386320c10b2e9bff31f6f7e8d6"}, {file = "mypy-1.0.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:3cfad08f16a9c6611e6143485a93de0e1e13f48cfb90bcad7d5fde1c0cec3d36"},
{file = "mypy-0.981-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:b6ede64e52257931315826fdbfc6ea878d89a965580d1a65638ef77cb551f56d"}, {file = "mypy-1.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:67cced7f15654710386e5c10b96608f1ee3d5c94ca1da5a2aad5889793a824c1"},
{file = "mypy-0.981-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:eb3978b191b9fa0488524bb4ffedf2c573340e8c2b4206fc191d44c7093abfb7"}, {file = "mypy-1.0.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:a86b794e8a56ada65c573183756eac8ac5b8d3d59daf9d5ebd72ecdbb7867a43"},
{file = "mypy-0.981-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:77f8fcf7b4b3cc0c74fb33ae54a4cd00bb854d65645c48beccf65fa10b17882c"}, {file = "mypy-1.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:50979d5efff8d4135d9db293c6cb2c42260e70fb010cbc697b1311a4d7a39ddb"},
{file = "mypy-0.981-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f64d2ce043a209a297df322eb4054dfbaa9de9e8738291706eaafda81ab2b362"}, {file = "mypy-1.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:3ae4c7a99e5153496243146a3baf33b9beff714464ca386b5f62daad601d87af"},
{file = "mypy-0.981-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:2ee3dbc53d4df7e6e3b1c68ac6a971d3a4fb2852bf10a05fda228721dd44fae1"}, {file = "mypy-1.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:5e398652d005a198a7f3c132426b33c6b85d98aa7dc852137a2a3be8890c4072"},
{file = "mypy-0.981-cp39-cp39-win_amd64.whl", hash = "sha256:8e8e49aa9cc23aa4c926dc200ce32959d3501c4905147a66ce032f05cb5ecb92"}, {file = "mypy-1.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:be78077064d016bc1b639c2cbcc5be945b47b4261a4f4b7d8923f6c69c5c9457"},
{file = "mypy-0.981-py3-none-any.whl", hash = "sha256:794f385653e2b749387a42afb1e14c2135e18daeb027e0d97162e4b7031210f8"}, {file = "mypy-1.0.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:92024447a339400ea00ac228369cd242e988dd775640755fa4ac0c126e49bb74"},
{file = "mypy-0.981.tar.gz", hash = "sha256:ad77c13037d3402fbeffda07d51e3f228ba078d1c7096a73759c9419ea031bf4"}, {file = "mypy-1.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:fe523fcbd52c05040c7bee370d66fee8373c5972171e4fbc323153433198592d"},
{file = "mypy-1.0.0-py3-none-any.whl", hash = "sha256:2efa963bdddb27cb4a0d42545cd137a8d2b883bd181bbc4525b568ef6eca258f"},
{file = "mypy-1.0.0.tar.gz", hash = "sha256:f34495079c8d9da05b183f9f7daec2878280c2ad7cc81da686ef0b484cea2ecf"},
] ]
[package.dependencies] [package.dependencies]
@ -1186,6 +1188,7 @@ typing-extensions = ">=3.10"
[package.extras] [package.extras]
dmypy = ["psutil (>=4.0)"] dmypy = ["psutil (>=4.0)"]
install-types = ["pip"]
python2 = ["typed-ast (>=1.4.0,<2)"] python2 = ["typed-ast (>=1.4.0,<2)"]
reports = ["lxml"] reports = ["lxml"]
@ -1203,18 +1206,18 @@ files = [
[[package]] [[package]]
name = "mypy-zope" name = "mypy-zope"
version = "0.3.11" version = "0.9.0"
description = "Plugin for mypy to support zope interfaces" description = "Plugin for mypy to support zope interfaces"
category = "dev" category = "dev"
optional = false optional = false
python-versions = "*" python-versions = "*"
files = [ files = [
{file = "mypy-zope-0.3.11.tar.gz", hash = "sha256:d4255f9f04d48c79083bbd4e2fea06513a6ac7b8de06f8c4ce563fd85142ca05"}, {file = "mypy-zope-0.9.0.tar.gz", hash = "sha256:88bf6cd056e38b338e6956055958a7805b4ff84404ccd99e29883a3647a1aeb3"},
{file = "mypy_zope-0.3.11-py3-none-any.whl", hash = "sha256:ec080a6508d1f7805c8d2054f9fdd13c849742ce96803519e1fdfa3d3cab7140"}, {file = "mypy_zope-0.9.0-py3-none-any.whl", hash = "sha256:e1bb4b57084f76ff8a154a3e07880a1af2ac6536c491dad4b143d529f72c5d15"},
] ]
[package.dependencies] [package.dependencies]
mypy = "0.981" mypy = "1.0.0"
"zope.interface" = "*" "zope.interface" = "*"
"zope.schema" = "*" "zope.schema" = "*"
@ -1705,7 +1708,7 @@ files = [
cffi = ">=1.4.1" cffi = ">=1.4.1"
[package.extras] [package.extras]
docs = ["sphinx (>=1.6.5)", "sphinx-rtd-theme"] docs = ["sphinx (>=1.6.5)", "sphinx_rtd_theme"]
tests = ["hypothesis (>=3.27.0)", "pytest (>=3.2.1,!=3.3.0)"] tests = ["hypothesis (>=3.27.0)", "pytest (>=3.2.1,!=3.3.0)"]
[[package]] [[package]]

View file

@ -201,7 +201,7 @@ class AuthHandler:
for auth_checker_class in INTERACTIVE_AUTH_CHECKERS: for auth_checker_class in INTERACTIVE_AUTH_CHECKERS:
inst = auth_checker_class(hs) inst = auth_checker_class(hs)
if inst.is_enabled(): if inst.is_enabled():
self.checkers[inst.AUTH_TYPE] = inst # type: ignore self.checkers[inst.AUTH_TYPE] = inst
self.bcrypt_rounds = hs.config.registration.bcrypt_rounds self.bcrypt_rounds = hs.config.registration.bcrypt_rounds

View file

@ -13,7 +13,8 @@
# limitations under the License. # limitations under the License.
import logging import logging
from typing import TYPE_CHECKING, Any from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, ClassVar, Sequence, Type
from twisted.web.client import PartialDownloadError from twisted.web.client import PartialDownloadError
@ -27,19 +28,28 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class UserInteractiveAuthChecker: class UserInteractiveAuthChecker(ABC):
"""Abstract base class for an interactive auth checker""" """Abstract base class for an interactive auth checker"""
def __init__(self, hs: "HomeServer"): # This should really be an "abstract class property", i.e. it should
# be an error to instantiate a subclass that doesn't specify an AUTH_TYPE.
# But calling this a `ClassVar` is simpler than a decorator stack of
# @property @abstractmethod and @classmethod (if that's even the right order).
AUTH_TYPE: ClassVar[str]
def __init__(self, hs: "HomeServer"): # noqa: B027
pass pass
@abstractmethod
def is_enabled(self) -> bool: def is_enabled(self) -> bool:
"""Check if the configuration of the homeserver allows this checker to work """Check if the configuration of the homeserver allows this checker to work
Returns: Returns:
True if this login type is enabled. True if this login type is enabled.
""" """
raise NotImplementedError()
@abstractmethod
async def check_auth(self, authdict: dict, clientip: str) -> Any: async def check_auth(self, authdict: dict, clientip: str) -> Any:
"""Given the authentication dict from the client, attempt to check this step """Given the authentication dict from the client, attempt to check this step
@ -304,7 +314,7 @@ class RegistrationTokenAuthChecker(UserInteractiveAuthChecker):
) )
INTERACTIVE_AUTH_CHECKERS = [ INTERACTIVE_AUTH_CHECKERS: Sequence[Type[UserInteractiveAuthChecker]] = [
DummyAuthChecker, DummyAuthChecker,
TermsAuthChecker, TermsAuthChecker,
RecaptchaAuthChecker, RecaptchaAuthChecker,

View file

@ -1267,7 +1267,7 @@ class MatrixFederationHttpClient:
def _flatten_response_never_received(e: BaseException) -> str: def _flatten_response_never_received(e: BaseException) -> str:
if hasattr(e, "reasons"): if hasattr(e, "reasons"):
reasons = ", ".join( reasons = ", ".join(
_flatten_response_never_received(f.value) for f in e.reasons # type: ignore[attr-defined] _flatten_response_never_received(f.value) for f in e.reasons
) )
return "%s:[%s]" % (type(e).__name__, reasons) return "%s:[%s]" % (type(e).__name__, reasons)

View file

@ -188,7 +188,7 @@ from typing import (
) )
import attr import attr
from typing_extensions import ParamSpec from typing_extensions import Concatenate, ParamSpec
from twisted.internet import defer from twisted.internet import defer
from twisted.web.http import Request from twisted.web.http import Request
@ -445,7 +445,7 @@ def init_tracer(hs: "HomeServer") -> None:
opentracing = None # type: ignore[assignment] opentracing = None # type: ignore[assignment]
return return
if not opentracing or not JaegerConfig: if opentracing is None or JaegerConfig is None:
raise ConfigError( raise ConfigError(
"The server has been configured to use opentracing but opentracing is not " "The server has been configured to use opentracing but opentracing is not "
"installed." "installed."
@ -872,7 +872,7 @@ def extract_text_map(carrier: Dict[str, str]) -> Optional["opentracing.SpanConte
def _custom_sync_async_decorator( def _custom_sync_async_decorator(
func: Callable[P, R], func: Callable[P, R],
wrapping_logic: Callable[[Callable[P, R], Any, Any], ContextManager[None]], wrapping_logic: Callable[Concatenate[Callable[P, R], P], ContextManager[None]],
) -> Callable[P, R]: ) -> Callable[P, R]:
""" """
Decorates a function that is sync or async (coroutines), or that returns a Twisted Decorates a function that is sync or async (coroutines), or that returns a Twisted
@ -902,10 +902,14 @@ def _custom_sync_async_decorator(
""" """
if inspect.iscoroutinefunction(func): if inspect.iscoroutinefunction(func):
# In this branch, R = Awaitable[RInner], for some other type RInner
@wraps(func) @wraps(func)
async def _wrapper(*args: P.args, **kwargs: P.kwargs) -> R: async def _wrapper(
*args: P.args, **kwargs: P.kwargs
) -> Any: # Return type is RInner
with wrapping_logic(func, *args, **kwargs): with wrapping_logic(func, *args, **kwargs):
# type-ignore: func() returns R, but mypy doesn't know that R is
# Awaitable here.
return await func(*args, **kwargs) # type: ignore[misc] return await func(*args, **kwargs) # type: ignore[misc]
else: else:
@ -972,7 +976,11 @@ def trace_with_opname(
if not opentracing: if not opentracing:
return func return func
return _custom_sync_async_decorator(func, _wrapping_logic) # type-ignore: mypy seems to be confused by the ParamSpecs here.
# I think the problem is https://github.com/python/mypy/issues/12909
return _custom_sync_async_decorator(
func, _wrapping_logic # type: ignore[arg-type]
)
return _decorator return _decorator
@ -1018,7 +1026,9 @@ def tag_args(func: Callable[P, R]) -> Callable[P, R]:
set_tag(SynapseTags.FUNC_KWARGS, str(kwargs)) set_tag(SynapseTags.FUNC_KWARGS, str(kwargs))
yield yield
return _custom_sync_async_decorator(func, _wrapping_logic) # type-ignore: mypy seems to be confused by the ParamSpecs here.
# I think the problem is https://github.com/python/mypy/issues/12909
return _custom_sync_async_decorator(func, _wrapping_logic) # type: ignore[arg-type]
@contextlib.contextmanager @contextlib.contextmanager

View file

@ -16,6 +16,7 @@
import logging import logging
import os import os
import urllib import urllib
from abc import ABC, abstractmethod
from types import TracebackType from types import TracebackType
from typing import Awaitable, Dict, Generator, List, Optional, Tuple, Type from typing import Awaitable, Dict, Generator, List, Optional, Tuple, Type
@ -284,13 +285,14 @@ async def respond_with_responder(
finish_request(request) finish_request(request)
class Responder: class Responder(ABC):
"""Represents a response that can be streamed to the requester. """Represents a response that can be streamed to the requester.
Responder is a context manager which *must* be used, so that any resources Responder is a context manager which *must* be used, so that any resources
held can be cleaned up. held can be cleaned up.
""" """
@abstractmethod
def write_to_consumer(self, consumer: IConsumer) -> Awaitable: def write_to_consumer(self, consumer: IConsumer) -> Awaitable:
"""Stream response into consumer """Stream response into consumer
@ -300,11 +302,12 @@ class Responder:
Returns: Returns:
Resolves once the response has finished being written Resolves once the response has finished being written
""" """
raise NotImplementedError()
def __enter__(self) -> None: def __enter__(self) -> None: # noqa: B027
pass pass
def __exit__( def __exit__( # noqa: B027
self, self,
exc_type: Optional[Type[BaseException]], exc_type: Optional[Type[BaseException]],
exc_val: Optional[BaseException], exc_val: Optional[BaseException],

View file

@ -25,7 +25,7 @@ try:
except ImportError: except ImportError:
class PostgresEngine(BaseDatabaseEngine): # type: ignore[no-redef] class PostgresEngine(BaseDatabaseEngine): # type: ignore[no-redef]
def __new__(cls, *args: object, **kwargs: object) -> NoReturn: # type: ignore[misc] def __new__(cls, *args: object, **kwargs: object) -> NoReturn:
raise RuntimeError( raise RuntimeError(
f"Cannot create {cls.__name__} -- psycopg2 module is not installed" f"Cannot create {cls.__name__} -- psycopg2 module is not installed"
) )
@ -36,7 +36,7 @@ try:
except ImportError: except ImportError:
class Sqlite3Engine(BaseDatabaseEngine): # type: ignore[no-redef] class Sqlite3Engine(BaseDatabaseEngine): # type: ignore[no-redef]
def __new__(cls, *args: object, **kwargs: object) -> NoReturn: # type: ignore[misc] def __new__(cls, *args: object, **kwargs: object) -> NoReturn:
raise RuntimeError( raise RuntimeError(
f"Cannot create {cls.__name__} -- sqlite3 module is not installed" f"Cannot create {cls.__name__} -- sqlite3 module is not installed"
) )

View file

@ -12,7 +12,18 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from types import TracebackType from types import TracebackType
from typing import Any, Iterator, List, Mapping, Optional, Sequence, Tuple, Type, Union from typing import (
Any,
Callable,
Iterator,
List,
Mapping,
Optional,
Sequence,
Tuple,
Type,
Union,
)
from typing_extensions import Protocol from typing_extensions import Protocol
@ -112,15 +123,35 @@ class DBAPI2Module(Protocol):
# extends from this hierarchy. See # extends from this hierarchy. See
# https://docs.python.org/3/library/sqlite3.html?highlight=sqlite3#exceptions # https://docs.python.org/3/library/sqlite3.html?highlight=sqlite3#exceptions
# https://www.postgresql.org/docs/current/errcodes-appendix.html#ERRCODES-TABLE # https://www.postgresql.org/docs/current/errcodes-appendix.html#ERRCODES-TABLE
Warning: Type[Exception] #
Error: Type[Exception] # Note: rather than
# x: T
# we write
# @property
# def x(self) -> T: ...
# which expresses that the protocol attribute `x` is read-only. The mypy docs
# https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected
# explain why this is necessary for safety. TL;DR: we shouldn't be able to write
# to `x`, only read from it. See also https://github.com/python/mypy/issues/6002 .
@property
def Warning(self) -> Type[Exception]:
...
@property
def Error(self) -> Type[Exception]:
...
# Errors are divided into `InterfaceError`s (something went wrong in the database # Errors are divided into `InterfaceError`s (something went wrong in the database
# driver) and `DatabaseError`s (something went wrong in the database). These are # driver) and `DatabaseError`s (something went wrong in the database). These are
# both subclasses of `Error`, but we can't currently express this in type # both subclasses of `Error`, but we can't currently express this in type
# annotations due to https://github.com/python/mypy/issues/8397 # annotations due to https://github.com/python/mypy/issues/8397
InterfaceError: Type[Exception] @property
DatabaseError: Type[Exception] def InterfaceError(self) -> Type[Exception]:
...
@property
def DatabaseError(self) -> Type[Exception]:
...
# Everything below is a subclass of `DatabaseError`. # Everything below is a subclass of `DatabaseError`.
@ -128,7 +159,9 @@ class DBAPI2Module(Protocol):
# - An integer was too big for its data type. # - An integer was too big for its data type.
# - An invalid date time was provided. # - An invalid date time was provided.
# - A string contained a null code point. # - A string contained a null code point.
DataError: Type[Exception] @property
def DataError(self) -> Type[Exception]:
...
# Roughly: something went wrong in the database, but it's not within the application # Roughly: something went wrong in the database, but it's not within the application
# programmer's control. Examples: # programmer's control. Examples:
@ -138,28 +171,45 @@ class DBAPI2Module(Protocol):
# - A serialisation failure occurred. # - A serialisation failure occurred.
# - The database ran out of resources, such as storage, memory, connections, etc. # - The database ran out of resources, such as storage, memory, connections, etc.
# - The database encountered an error from the operating system. # - The database encountered an error from the operating system.
OperationalError: Type[Exception] @property
def OperationalError(self) -> Type[Exception]:
...
# Roughly: we've given the database data which breaks a rule we asked it to enforce. # Roughly: we've given the database data which breaks a rule we asked it to enforce.
# Examples: # Examples:
# - Stop, criminal scum! You violated the foreign key constraint # - Stop, criminal scum! You violated the foreign key constraint
# - Also check constraints, non-null constraints, etc. # - Also check constraints, non-null constraints, etc.
IntegrityError: Type[Exception] @property
def IntegrityError(self) -> Type[Exception]:
...
# Roughly: something went wrong within the database server itself. # Roughly: something went wrong within the database server itself.
InternalError: Type[Exception] @property
def InternalError(self) -> Type[Exception]:
...
# Roughly: the application did something silly that needs to be fixed. Examples: # Roughly: the application did something silly that needs to be fixed. Examples:
# - We don't have permissions to do something. # - We don't have permissions to do something.
# - We tried to create a table with duplicate column names. # - We tried to create a table with duplicate column names.
# - We tried to use a reserved name. # - We tried to use a reserved name.
# - We referred to a column that doesn't exist. # - We referred to a column that doesn't exist.
ProgrammingError: Type[Exception] @property
def ProgrammingError(self) -> Type[Exception]:
...
# Roughly: we've tried to do something that this database doesn't support. # Roughly: we've tried to do something that this database doesn't support.
NotSupportedError: Type[Exception] @property
def NotSupportedError(self) -> Type[Exception]:
...
def connect(self, **parameters: object) -> Connection: # We originally wrote
# def connect(self, *args, **kwargs) -> Connection: ...
# But mypy doesn't seem to like that because sqlite3.connect takes a mandatory
# positional argument. We can't make that part of the signature though, because
# psycopg2.connect doesn't have a mandatory positional argument. Instead, we use
# the following slightly unusual workaround.
@property
def connect(self) -> Callable[..., Connection]:
... ...

View file

@ -11,7 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from abc import ABC, abstractmethod
from typing import Generic, List, Optional, Tuple, TypeVar from typing import Generic, List, Optional, Tuple, TypeVar
from synapse.types import StrCollection, UserID from synapse.types import StrCollection, UserID
@ -22,7 +22,8 @@ K = TypeVar("K")
R = TypeVar("R") R = TypeVar("R")
class EventSource(Generic[K, R]): class EventSource(ABC, Generic[K, R]):
@abstractmethod
async def get_new_events( async def get_new_events(
self, self,
user: UserID, user: UserID,
@ -32,4 +33,4 @@ class EventSource(Generic[K, R]):
is_guest: bool, is_guest: bool,
explicit_room_id: Optional[str] = None, explicit_room_id: Optional[str] = None,
) -> Tuple[List[R], K]: ) -> Tuple[List[R], K]:
... raise NotImplementedError()

View file

@ -62,7 +62,7 @@ class TestSpamChecker:
request_info: Collection[Tuple[str, str]], request_info: Collection[Tuple[str, str]],
auth_provider_id: Optional[str], auth_provider_id: Optional[str],
) -> RegistrationBehaviour: ) -> RegistrationBehaviour:
pass return RegistrationBehaviour.ALLOW
class DenyAll(TestSpamChecker): class DenyAll(TestSpamChecker):
@ -111,7 +111,7 @@ class TestLegacyRegistrationSpamChecker:
username: Optional[str], username: Optional[str],
request_info: Collection[Tuple[str, str]], request_info: Collection[Tuple[str, str]],
) -> RegistrationBehaviour: ) -> RegistrationBehaviour:
pass return RegistrationBehaviour.ALLOW
class LegacyAllowAll(TestLegacyRegistrationSpamChecker): class LegacyAllowAll(TestLegacyRegistrationSpamChecker):

View file

@ -63,7 +63,7 @@ from tests.http import (
get_test_ca_cert_file, get_test_ca_cert_file,
) )
from tests.server import FakeTransport, ThreadedMemoryReactorClock from tests.server import FakeTransport, ThreadedMemoryReactorClock
from tests.utils import default_config from tests.utils import checked_cast, default_config
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -146,8 +146,10 @@ class MatrixFederationAgentTests(unittest.TestCase):
# #
# Normally this would be done by the TCP socket code in Twisted, but we are # Normally this would be done by the TCP socket code in Twisted, but we are
# stubbing that out here. # stubbing that out here.
client_protocol = client_factory.buildProtocol(dummy_address) # NB: we use a checked_cast here to workaround https://github.com/Shoobx/mypy-zope/issues/91)
assert isinstance(client_protocol, _WrappingProtocol) client_protocol = checked_cast(
_WrappingProtocol, client_factory.buildProtocol(dummy_address)
)
client_protocol.makeConnection( client_protocol.makeConnection(
FakeTransport(server_protocol, self.reactor, client_protocol) FakeTransport(server_protocol, self.reactor, client_protocol)
) )
@ -446,7 +448,6 @@ class MatrixFederationAgentTests(unittest.TestCase):
server_ssl_protocol = _wrap_server_factory_for_tls( server_ssl_protocol = _wrap_server_factory_for_tls(
_get_test_protocol_factory() _get_test_protocol_factory()
).buildProtocol(dummy_address) ).buildProtocol(dummy_address)
assert isinstance(server_ssl_protocol, TLSMemoryBIOProtocol)
# Tell the HTTP server to send outgoing traffic back via the proxy's transport. # Tell the HTTP server to send outgoing traffic back via the proxy's transport.
proxy_server_transport = proxy_server.transport proxy_server_transport = proxy_server.transport
@ -1529,7 +1530,7 @@ def _check_logcontext(context: LoggingContextOrSentinel) -> None:
def _wrap_server_factory_for_tls( def _wrap_server_factory_for_tls(
factory: IProtocolFactory, sanlist: Optional[List[bytes]] = None factory: IProtocolFactory, sanlist: Optional[List[bytes]] = None
) -> IProtocolFactory: ) -> TLSMemoryBIOFactory:
"""Wrap an existing Protocol Factory with a test TLSMemoryBIOFactory """Wrap an existing Protocol Factory with a test TLSMemoryBIOFactory
The resultant factory will create a TLS server which presents a certificate The resultant factory will create a TLS server which presents a certificate
signed by our test CA, valid for the domains in `sanlist` signed by our test CA, valid for the domains in `sanlist`

View file

@ -43,6 +43,7 @@ from tests.http import (
) )
from tests.server import FakeTransport, ThreadedMemoryReactorClock from tests.server import FakeTransport, ThreadedMemoryReactorClock
from tests.unittest import TestCase from tests.unittest import TestCase
from tests.utils import checked_cast
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -620,7 +621,6 @@ class MatrixFederationAgentTests(TestCase):
server_ssl_protocol = _wrap_server_factory_for_tls( server_ssl_protocol = _wrap_server_factory_for_tls(
_get_test_protocol_factory() _get_test_protocol_factory()
).buildProtocol(dummy_address) ).buildProtocol(dummy_address)
assert isinstance(server_ssl_protocol, TLSMemoryBIOProtocol)
# Tell the HTTP server to send outgoing traffic back via the proxy's transport. # Tell the HTTP server to send outgoing traffic back via the proxy's transport.
proxy_server_transport = proxy_server.transport proxy_server_transport = proxy_server.transport
@ -757,12 +757,14 @@ class MatrixFederationAgentTests(TestCase):
assert isinstance(proxy_server, HTTPChannel) assert isinstance(proxy_server, HTTPChannel)
# fish the transports back out so that we can do the old switcheroo # fish the transports back out so that we can do the old switcheroo
s2c_transport = proxy_server.transport # To help mypy out with the various Protocols and wrappers and mocks, we do
assert isinstance(s2c_transport, FakeTransport) # some explicit casting. Without the casts, we hit the bug I reported at
client_protocol = s2c_transport.other # https://github.com/Shoobx/mypy-zope/issues/91 .
assert isinstance(client_protocol, _WrappingProtocol) # We also double-checked these casts at runtime (test-time) because I found it
c2s_transport = client_protocol.transport # quite confusing to deduce these types in the first place!
assert isinstance(c2s_transport, FakeTransport) s2c_transport = checked_cast(FakeTransport, proxy_server.transport)
client_protocol = checked_cast(_WrappingProtocol, s2c_transport.other)
c2s_transport = checked_cast(FakeTransport, client_protocol.transport)
# the FakeTransport is async, so we need to pump the reactor # the FakeTransport is async, so we need to pump the reactor
self.reactor.advance(0) self.reactor.advance(0)
@ -822,9 +824,9 @@ class MatrixFederationAgentTests(TestCase):
@patch.dict(os.environ, {"http_proxy": "proxy.com:8888"}) @patch.dict(os.environ, {"http_proxy": "proxy.com:8888"})
def test_proxy_with_no_scheme(self) -> None: def test_proxy_with_no_scheme(self) -> None:
http_proxy_agent = ProxyAgent(self.reactor, use_proxy=True) http_proxy_agent = ProxyAgent(self.reactor, use_proxy=True)
assert isinstance(http_proxy_agent.http_proxy_endpoint, HostnameEndpoint) proxy_ep = checked_cast(HostnameEndpoint, http_proxy_agent.http_proxy_endpoint)
self.assertEqual(http_proxy_agent.http_proxy_endpoint._hostStr, "proxy.com") self.assertEqual(proxy_ep._hostStr, "proxy.com")
self.assertEqual(http_proxy_agent.http_proxy_endpoint._port, 8888) self.assertEqual(proxy_ep._port, 8888)
@patch.dict(os.environ, {"http_proxy": "socks://proxy.com:8888"}) @patch.dict(os.environ, {"http_proxy": "socks://proxy.com:8888"})
def test_proxy_with_unsupported_scheme(self) -> None: def test_proxy_with_unsupported_scheme(self) -> None:
@ -834,25 +836,21 @@ class MatrixFederationAgentTests(TestCase):
@patch.dict(os.environ, {"http_proxy": "http://proxy.com:8888"}) @patch.dict(os.environ, {"http_proxy": "http://proxy.com:8888"})
def test_proxy_with_http_scheme(self) -> None: def test_proxy_with_http_scheme(self) -> None:
http_proxy_agent = ProxyAgent(self.reactor, use_proxy=True) http_proxy_agent = ProxyAgent(self.reactor, use_proxy=True)
assert isinstance(http_proxy_agent.http_proxy_endpoint, HostnameEndpoint) proxy_ep = checked_cast(HostnameEndpoint, http_proxy_agent.http_proxy_endpoint)
self.assertEqual(http_proxy_agent.http_proxy_endpoint._hostStr, "proxy.com") self.assertEqual(proxy_ep._hostStr, "proxy.com")
self.assertEqual(http_proxy_agent.http_proxy_endpoint._port, 8888) self.assertEqual(proxy_ep._port, 8888)
@patch.dict(os.environ, {"http_proxy": "https://proxy.com:8888"}) @patch.dict(os.environ, {"http_proxy": "https://proxy.com:8888"})
def test_proxy_with_https_scheme(self) -> None: def test_proxy_with_https_scheme(self) -> None:
https_proxy_agent = ProxyAgent(self.reactor, use_proxy=True) https_proxy_agent = ProxyAgent(self.reactor, use_proxy=True)
assert isinstance(https_proxy_agent.http_proxy_endpoint, _WrapperEndpoint) proxy_ep = checked_cast(_WrapperEndpoint, https_proxy_agent.http_proxy_endpoint)
self.assertEqual( self.assertEqual(proxy_ep._wrappedEndpoint._hostStr, "proxy.com")
https_proxy_agent.http_proxy_endpoint._wrappedEndpoint._hostStr, "proxy.com" self.assertEqual(proxy_ep._wrappedEndpoint._port, 8888)
)
self.assertEqual(
https_proxy_agent.http_proxy_endpoint._wrappedEndpoint._port, 8888
)
def _wrap_server_factory_for_tls( def _wrap_server_factory_for_tls(
factory: IProtocolFactory, sanlist: Optional[List[bytes]] = None factory: IProtocolFactory, sanlist: Optional[List[bytes]] = None
) -> IProtocolFactory: ) -> TLSMemoryBIOFactory:
"""Wrap an existing Protocol Factory with a test TLSMemoryBIOFactory """Wrap an existing Protocol Factory with a test TLSMemoryBIOFactory
The resultant factory will create a TLS server which presents a certificate The resultant factory will create a TLS server which presents a certificate

View file

@ -21,6 +21,7 @@ from synapse.logging import RemoteHandler
from tests.logging import LoggerCleanupMixin from tests.logging import LoggerCleanupMixin
from tests.server import FakeTransport, get_clock from tests.server import FakeTransport, get_clock
from tests.unittest import TestCase from tests.unittest import TestCase
from tests.utils import checked_cast
def connect_logging_client( def connect_logging_client(
@ -56,8 +57,8 @@ class RemoteHandlerTestCase(LoggerCleanupMixin, TestCase):
client, server = connect_logging_client(self.reactor, 0) client, server = connect_logging_client(self.reactor, 0)
# Trigger data being sent # Trigger data being sent
assert isinstance(client.transport, FakeTransport) client_transport = checked_cast(FakeTransport, client.transport)
client.transport.flush() client_transport.flush()
# One log message, with a single trailing newline # One log message, with a single trailing newline
logs = server.data.decode("utf8").splitlines() logs = server.data.decode("utf8").splitlines()
@ -89,8 +90,8 @@ class RemoteHandlerTestCase(LoggerCleanupMixin, TestCase):
# Allow the reconnection # Allow the reconnection
client, server = connect_logging_client(self.reactor, 0) client, server = connect_logging_client(self.reactor, 0)
assert isinstance(client.transport, FakeTransport) client_transport = checked_cast(FakeTransport, client.transport)
client.transport.flush() client_transport.flush()
# Only the 7 infos made it through, the debugs were elided # Only the 7 infos made it through, the debugs were elided
logs = server.data.splitlines() logs = server.data.splitlines()
@ -123,8 +124,8 @@ class RemoteHandlerTestCase(LoggerCleanupMixin, TestCase):
# Allow the reconnection # Allow the reconnection
client, server = connect_logging_client(self.reactor, 0) client, server = connect_logging_client(self.reactor, 0)
assert isinstance(client.transport, FakeTransport) client_transport = checked_cast(FakeTransport, client.transport)
client.transport.flush() client_transport.flush()
# The 10 warnings made it through, the debugs and infos were elided # The 10 warnings made it through, the debugs and infos were elided
logs = server.data.splitlines() logs = server.data.splitlines()
@ -148,8 +149,8 @@ class RemoteHandlerTestCase(LoggerCleanupMixin, TestCase):
# Allow the reconnection # Allow the reconnection
client, server = connect_logging_client(self.reactor, 0) client, server = connect_logging_client(self.reactor, 0)
assert isinstance(client.transport, FakeTransport) client_transport = checked_cast(FakeTransport, client.transport)
client.transport.flush() client_transport.flush()
# The first five and last five warnings made it through, the debugs and # The first five and last five warnings made it through, the debugs and
# infos were elided # infos were elided

View file

@ -43,6 +43,9 @@ class DummyRecaptchaChecker(UserInteractiveAuthChecker):
super().__init__(hs) super().__init__(hs)
self.recaptcha_attempts: List[Tuple[dict, str]] = [] self.recaptcha_attempts: List[Tuple[dict, str]] = []
def is_enabled(self) -> bool:
return True
def check_auth(self, authdict: dict, clientip: str) -> Any: def check_auth(self, authdict: dict, clientip: str) -> Any:
self.recaptcha_attempts.append((authdict, clientip)) self.recaptcha_attempts.append((authdict, clientip))
return succeed(True) return succeed(True)

View file

@ -425,7 +425,7 @@ class ThirdPartyRulesTestCase(unittest.FederatingHomeserverTestCase):
async def test_fn( async def test_fn(
event: EventBase, state_events: StateMap[EventBase] event: EventBase, state_events: StateMap[EventBase]
) -> Tuple[bool, Optional[JsonDict]]: ) -> Tuple[bool, Optional[JsonDict]]:
if event.is_state and event.type == EventTypes.PowerLevels: if event.is_state() and event.type == EventTypes.PowerLevels:
await api.create_and_send_event_into_room( await api.create_and_send_event_into_room(
{ {
"room_id": event.room_id, "room_id": event.room_id,

View file

@ -15,7 +15,7 @@
import atexit import atexit
import os import os
from typing import Any, Callable, Dict, List, Tuple, Union, overload from typing import Any, Callable, Dict, List, Tuple, Type, TypeVar, Union, overload
import attr import attr
from typing_extensions import Literal, ParamSpec from typing_extensions import Literal, ParamSpec
@ -341,3 +341,27 @@ async def create_room(hs: HomeServer, room_id: str, creator_id: str) -> None:
context = await unpersisted_context.persist(event) context = await unpersisted_context.persist(event)
await persistence_store.persist_event(event, context) await persistence_store.persist_event(event, context)
T = TypeVar("T")
def checked_cast(type: Type[T], x: object) -> T:
"""A version of typing.cast that is checked at runtime.
We have our own function for this for two reasons:
1. typing.cast itself is deliberately a no-op at runtime, see
https://docs.python.org/3/library/typing.html#typing.cast
2. To help workaround a mypy-zope bug https://github.com/Shoobx/mypy-zope/issues/91
where mypy would erroneously consider `isinstance(x, type)` to be false in all
circumstances.
For this to make sense, `T` needs to be something that `isinstance` can check; see
https://docs.python.org/3/library/functions.html?highlight=isinstance#isinstance
https://docs.python.org/3/glossary.html#term-abstract-base-class
https://docs.python.org/3/library/typing.html#typing.runtime_checkable
for more details.
"""
assert isinstance(x, type)
return x