8560 commits

Author	SHA1	Message	Date
Mathieu Velten	5cf74c2da0	Fix bug when rejecting withdrew invite with a third_party_rules module (#17930) ... When rejecting a withdrew invite through federation, an out of band event needs to be created. When doing so with a third_party_rules module installed, `get_prev_state_ids` [is called](e0fdb862cb/synapse/module_api/callbacks/third_party_event_rules_callbacks.py (L285)) on the context to calculate the state to pass at `check_event_allowed` callbacks. The context for outliers is defined [here](e0fdb862cb/synapse/events/snapshot.py (L168)), and `state_group_before_event` is None. This change makes the behavior of `get_prev_state_ids` and `get_current_state_ids` match the one presented in the docstring regarding null state_group.	2024-12-10 14:26:38 +00:00
Rafał Hirsch	adce8a0111	Reorganize account data, receipts and presence request regexps in generic_worker docs (#17954) ... POST requests for account data, receipts and presence require the worker to be configured as a stream writer. The regular expressions in the default list don't assume any HTTP method, so if the worker is not a stream writer, the request fails. The stream writer section of the documentation lists the same regexps as the one I'm removing, so people configuring stream writers can still configure their routing properly. More context: https://github.com/element-hq/synapse/issues/17243#issuecomment-2493621645	2024-12-09 10:30:03 -06:00
Devon Hudson	02d09e3f0c	Add RoomID & EventID rust types (#17996) ... Adds the RoomID & EventID rust types to the rust lib. Also adds a Deserialize impl to the existing UserID type. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))	2024-12-05 15:41:57 +00:00
Travis Ralston	b90ad26ebc	Promote account suspension to stable (#17964) ... MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/3823	2024-12-04 17:56:42 -06:00
Andrew Morgan	a00d0b3d0e	1.121.0rc1	2024-12-04 14:49:28 +00:00
Andrew Morgan	45ca6392f4	Pin Rust to 1.82.0 when building Python wheels (#17993) ... Addresses step 1 of #17988.	2024-12-04 12:58:26 +00:00
Andrew Morgan	05d58b86ac	Pin softprops/action-gh-release to v0.1.15 (#17995) ... We are still seeing duplicate releases on v2.0.5, so roll back further. [Other](f8a5a60b7c (diff-88ab30345d9874c4336fe50b54b083ba5bdd925be961c34060e6a192b56b0433R72)) [repositories](55fca4fec7 (diff-e426ed45842837026e10e66af23d9c7077e89eacbe6958ce7cb991130ad05adaR105)) seem to have settled on this version. Addresses https://github.com/element-hq/synapse/issues/17991 We're just going to test this during 1.121.0rc1.	2024-12-04 12:53:51 +00:00
Quentin Gliech	23b626f2e6	Support for MSC4190: device management for application services (#17705) ... This is an implementation of MSC4190, which allows appservices to manage their user's devices without /login & /logout. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-12-04 12:04:49 +01:00
manuroe	abf44ad324	MSC4076: Add disable_badge_count to pusher configuration (#17975) ... This PR implements [MSC4076: Let E2EE clients calculate app badge counts themselves (disable_badge_count)](https://github.com/matrix-org/matrix-spec-proposals/pull/4076).	2024-12-03 22:58:43 +00:00
Eric Eastwood	b257c7ab19	Be able to test /login/sso/redirect in Complement (#17986) ... Be able to test `/login/sso/redirect` in Complement Spawning from https://github.com/element-hq/sbg/pull/421#discussion_r1854926218 where we have a proxy that intercepts responses to `/_matrix/client/v3/login/sso/redirect(/{idpId})` in order to upgrade them to use OAuth 2.0 Pushed Authorization Requests (PAR). We have some Complement tests in that codebase that go over this flow and these changes are required [in order for the URL's to line up](d648c8ce3f/synapse/rest/client/login.py (L652-L673)).	2024-12-03 12:54:25 +00:00
Eric Eastwood	e5d3bfba30	Sliding Sync: Include invite, ban, kick, targets when $LAZY-loading room members (#17947) ... Part of https://github.com/element-hq/synapse/issues/17929	2024-12-02 10:17:55 -06:00
Travis Ralston	9b2ae62d20	Use stable error code for account locking (#17965)	2024-12-02 15:28:47 +00:00
Erik Johnston	a82f5f206f	Fix release process to not create duplicate releases (#17970) ... This is to work around https://github.com/softprops/action-gh-release/issues/445 --------- Co-authored-by: Quentin Gliech <quenting@element.io>	2024-12-02 10:54:14 +00:00
Eric Eastwood	6a909aade2	Consolidate SSO redirects through /_matrix/client/v3/login/sso/redirect(/{idpId}) (#17972) ... Consolidate SSO redirects through `/_matrix/client/v3/login/sso/redirect(/{idpId})` Spawning from https://github.com/element-hq/sbg/pull/421#discussion_r1859497330 where we have a proxy that intercepts responses to `/_matrix/client/v3/login/sso/redirect(/{idpId})` in order to upgrade them to use OAuth 2.0 Pushed Authorization Requests (PAR). Instead of needing to intercept multiple endpoints that redirect to the authorization endpoint, it seems better to just have Synapse consolidate to a single flow. ### Testing strategy 1. Create a new OAuth application. I'll be using GitHub for example but there are [many options](be65a8ec01/docs/openid.md). Visit https://github.com/settings/developers -> **New OAuth App** - Application name: `Synapse local testing` - Homepage URL: `http://localhost:8008` - Authorization callback URL: `http://localhost:8008/_synapse/client/oidc/callback` 1. Update your Synapse `homeserver.yaml` ```yaml server_name: "my.synapse.server" public_baseurl: http://localhost:8008/ listeners: - port: 8008 bind_addresses: [ #'::1', '127.0.0.1' ] tls: false type: http x_forwarded: true resources: - names: [client, federation, metrics] compress: false # SSO login testing oidc_providers: - idp_id: github idp_name: Github idp_brand: "github" # optional: styling hint for clients discover: false issuer: "https://github.com/" client_id: "xxx" # TO BE FILLED client_secret: "xxx" # TO BE FILLED authorization_endpoint: "https://github.com/login/oauth/authorize" token_endpoint: "https://github.com/login/oauth/access_token" userinfo_endpoint: "https://api.github.com/user" scopes: ["read:user"] user_mapping_provider: config: subject_claim: "id" localpart_template: "{{ user.login }}" display_name_template: "{{ user.name }}" ``` 1. Start Synapse: `poetry run synapse_homeserver --config-path homeserver.yaml` 1. Visit `http://localhost:8008/_synapse/client/pick_idp?redirectUrl=http%3A%2F%2Fexample.com` 1. Choose GitHub 1. Notice that you're redirected to GitHub to sign in (`https://github.com/login/oauth/authorize?...`) Tested locally and works: 1. `http://localhost:8008/_synapse/client/pick_idp?idp=oidc-github&redirectUrl=http%3A//example.com` -> 1. `http://localhost:8008/_matrix/client/v3/login/sso/redirect/oidc-github?redirectUrl=http://example.com` -> 1. `https://github.com/login/oauth/authorize?response_type=code&client_id=xxx&redirect_uri=http%3A%2F%2Flocalhost%3A8008%2F_synapse%2Fclient%2Foidc%2Fcallback&scope=read%3Auser&state=xxx&nonce=xxx`	2024-11-29 11:26:37 -06:00
Richard van der Hoff	d80cd57c54	Fix new scheduled tasks jumping the queue (#17962) ... Currently, when a new scheduled task is added and its scheduled time has already passed, we set it to ACTIVE. This is problematic, because it means it will jump the queue ahead of all other SCHEDULED tasks; furthermore, if the Synapse process gets restarted, it will jump ahead of any ACTIVE tasks which have been started but are taking a while to run. Instead, we leave it set to SCHEDULED, but kick off a call to `_launch_scheduled_tasks`, which will decide if we actually have capacity to start a new task, and start the newly-added task if so.	2024-11-28 18:06:19 +00:00
Erik Johnston	59ad4b18fc	Update setuptools-rust and fix building abi3 wheels (#17969) ... Newer versions of `setuptools-rust` ignore the `py_limited_api` flag to `RustExtension`, and instead read it from `bdist_wheel` config. c.f. https://github.com/PyO3/setuptools-rust/blob/main/CHANGELOG.md#190-2024-02-24	2024-11-27 13:31:43 +00:00
V02460	a58f09acc7	Bump pyo3 to v0.23.2 (#17966) ... Keep up-to-date with pyo3 releases. This bump enables Python 3.13 support and resolves deprecations. Links for quick reference: https://github.com/PyO3/pyo3/releases https://github.com/davidhewitt/pythonize/releases https://github.com/vorner/pyo3-log	2024-11-27 10:46:00 +00:00
Quentin Gliech	cee9da0da5	MSC4108: Add a Content-Type header on the PUT response (#17253) ... This is a workaround for some proxy setup, where the ETag header gets stripped from the response headers unless there is a Content-Type header set. In particular, we saw this bug when putting Cloudflare in front of Synapse. I'm pretty sure this is a Cloudflare bug, as this behaviour isn't documented anywhere, and doesn't make sense whatsoever. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-11-26 19:43:26 +01:00
Erik Johnston	3943d2fde7	Fix up logic for delaying sending read receipts over federation. (#17933) ... For context of why we delay read receipts, see https://github.com/matrix-org/synapse/issues/4730. Element Web often sends read receipts in quick succession, if it reloads the timeline it'll send one for the last message in the old timeline and again for the last message in the new timeline. This caused remote users to see a read receipt for older messages come through quickly, but then the second read receipt taking a while to arrive for the most recent message. There are two things going on in this PR: 1. There was a mismatch between seconds and milliseconds, and so we ended up delaying for far longer than intended. 2. Changing the logic to reuse the `DestinationWakeupQueue` (used for presence) The changes in logic are: - Treat the first receipt and subsequent receipts in a room in the same way - Whitelist certain classes of receipts to never delay being sent, i.e. receipts in small rooms, receipts for events that were sent within the last 60s, and sending receipts to the event sender's server. - The maximum delay a receipt can have before being sent to a server is 30s, and we'll send out receipts to remotes at least at 50Hz (by default) The upshot is that this should make receipts feel more snappy over federation. This new logic should send roughly between 10%–20% of transactions immediately on matrix.org.	2024-11-25 18:12:33 +00:00
Shay	4587decd67	Return suspended status when querying user account (#17952)	2024-11-22 12:37:19 +00:00
Matthew Hodgson	4c67d20af7	link to element-docker-demo from contrib/docker* (#17953)	2024-11-22 12:35:03 +00:00
Valentin Iovene	80e39fd834	Add Forgejo oidc provider config example (#17872)	2024-11-20 16:06:08 -06:00
Olivier 'reivilibre	573bdbc824	Merge branch 'release-v1.120' into develop	2024-11-20 17:26:16 +00:00
Erik Johnston	79c02cada0	Fix incorrect comment in new schema delta (#17936) ... Added in #17912, was a bad copy and paste.	2024-11-20 17:12:17 +00:00
V02460	84ec15c47e	Raise setuptools_rust version cap to 1.10.2 (#17944)	2024-11-20 16:49:21 +00:00
Will Hunt	f73edbe4d2	Add encrypted appservice extensions to Complement test image. (#17945)	2024-11-20 16:35:43 +00:00
Olivier 'reivilibre	ec4d136965	1.120.0rc1	2024-11-20 15:13:32 +00:00
Travis Ralston	d0a474d312	Enable authenticated media by default (#17889) ... Co-authored-by: Olivier 'reivilibre <oliverw@matrix.org>	2024-11-20 14:48:22 +00:00
Renaud Allard	8291aa8fd7	Support both import names of PyPI package python-multipart. (#17932)	2024-11-20 11:48:04 +00:00
Erik Johnston	1092a35a2a	Speed up slow initial sliding syncs on large servers (#17946) ... This was due to a missing index, which meant that deleting previous connections associated with the device and `conn_id` took a long time.	2024-11-19 15:03:32 +00:00
Richard van der Hoff	c5e89f5fae	Create one-off scheduled task to delete old OTKs (#17934) ... To work around the fact that, pre-https://github.com/element-hq/synapse/pull/17903, our database may have old one-time-keys that the clients have long thrown away the private keys for, we want to delete OTKs that look like they came from libolm. To spread the load a bit, without holding up other background database updates, we use a scheduled task to do the work.	2024-11-19 11:20:48 +00:00
Richard van der Hoff	d72843056b	Add some documentation about backing up Synapse (#17931) ... Fixes: https://github.com/element-hq/element-meta/issues/2155 Fixes: https://github.com/element-hq/synapse/issues/2046	2024-11-18 14:05:49 +00:00
Devon Hudson	e80dad5fa9	Move server event filtering logic to rust (#17928) ... ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))	2024-11-14 16:18:24 +00:00
Poruri Sai Rahul	c812a79422	Removal: Remove support for experimental msc3886 (#17638)	2024-11-13 14:10:20 +00:00
Erik Johnston	e0fdb862cb	Bump macos version used to build wheels (#17924) ... MacOS 12 is end-of-life and GitHub is deprecating support for it (including doing brown outs). Let's bump to MacOS 13.	2024-11-13 11:30:04 +00:00
Erik Johnston	73dc05c993	Unpin the upload release GHA action (#17923) ... We were pinned to an old version that had deprecation warnings. In new versions of the action leaving off properties (i.e. `draft` and `prerelease`) tells the action to not modify those properties of the release.	2024-11-12 16:52:00 +00:00
Benjamin Bouvier	bfb197c596	Fix typo in error message when a media ID isn't known (#17865)	2024-11-12 16:41:14 +00:00
Erik Johnston	f387f47a6a	Merge branch 'release-v1.119' into develop	2024-11-11 15:47:27 +00:00
Erik Johnston	a4c503674f	1.119.0rc2	2024-11-11 14:33:37 +00:00
Erik Johnston	2637b26cfe	Fix building and attaching release artifacts (#17921) ... Broke in #17905 due to upgrading the `upload-artifact` action, as we didn't rename debs. I think we also need to change how we download the artefacts and attach them to a release, as they'll download to a different place. Docs: - https://github.com/actions/upload-artifact/tree/v4/ - https://github.com/actions/download-artifact/tree/v4/	2024-11-11 14:32:45 +00:00
Erik Johnston	92fcca8ed7	Update changelog	2024-11-11 10:46:34 +00:00
Erik Johnston	c486ec8bc2	Add index to current_state_delta_stream (#17912) ... As we're now using it in the sync APIs to get state changes within a room	2024-11-11 10:45:46 +00:00
reivilibre	20fc9fcc33	Clarify the semantics of the enable_authenticated_media configuration option. (#17913) ... Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>	2024-11-11 10:44:47 +00:00
Devon Hudson	2f41f6d947	Update changelog for release	2024-11-08 10:23:07 -07:00
Devon Hudson	f377cee7ec	Merge branch 'develop' into release-v1.119	2024-11-08 10:06:46 -07:00
Erik Johnston	cacd4fd7bd	Fix MSC4222 returning full state (#17915) ... There was a bug that meant we would return the full state of the room on incremental syncs when using lazy loaded members and there were no entries in the timeline. This was due to trying to use `state_filter or state_filter.all()` as a short hand for handling `None` case, however `state_filter` implements `__bool__` so if the state filter was empty it would be set to full. c.f. MSC4222 and #17888	2024-11-08 16:41:24 +00:00
Erik Johnston	c7a1d0aa1a	Fix Twisted tests with latest release (#17911) ... c.f. #17906 and #17907	2024-11-07 16:22:09 +00:00
Andrew Morgan	c92639df21	Switch portdb CI to python 3.13, pg 17 (#17909)	2024-11-07 16:09:45 +00:00
Erik Johnston	d0fc1e904a	Fix cancellation tests with new Twisted. (#17906) ... The latest Twisted release changed how they implemented `__await__` on deferreds, which broke the machinery we used to test cancellation. This PR changes things a bit to instead patch the `__await__` method, which is a stable API. This mostly doesn't change the core logic, except for fixing two bugs: - We previously did not intercept all await points - After cancellation we now need to not only unblock currently blocked await points, but also make sure we don't block any future await points. c.f. https://github.com/twisted/twisted/pull/12226 --------- Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>	2024-11-07 15:26:14 +00:00
Erik Johnston	77eafd47df	Fix other unit tests with latest twisted (#17907) ... There's also https://github.com/element-hq/synapse/pull/17906	2024-11-07 10:11:13 +00:00