mirrors/synapse
1
0
Fork 0
mirror of https://github.com/element-hq/synapse.git synced 2024-12-14 11:57:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
24065 commits 943 branches 734 tags 511 MiB
Commit graph

24065 commits

This branch
This branch
All branches
Author SHA1 Message Date
Erik Johnston
98b3f56411 SS: Hook up notification counts 2024-08-09 10:31:56 +01:00
Erik Johnston
44ac2aa3b6
SSS: Implement PREVIOUSLY room tracking (#17535) 
Implement tracking of rooms that have had updates that have not been
sent down to clients.

Simplified Sliding Sync (SSS)
 2024-08-08 10:44:17 +01:00
Eric Eastwood
11db575218
Sliding Sync: Use stream_ordering based timeline pagination for incremental sync (#17510) 
Use `stream_ordering` based `timeline` pagination for incremental
`/sync` in Sliding Sync. Previously, we were always using a
`topological_ordering` but we should only be using that for historical
scenarios (initial `/sync`, newly joined, or haven't sent the room down
the connection before).

This is slightly different than what the [spec
suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing)

> Events are ordered in this API according to the arrival time of the
event on the homeserver. This can conflict with other APIs which order
events based on their partial ordering in the event graph. This can
result in duplicate events being received (once per distinct API
called). Clients SHOULD de-duplicate events based on the event ID when
this happens.

But we've had a [discussion below in this
PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569)
and this matches what Sync v2 already does and seems like it makes
sense. Created a spec issue
https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this.

Related issues:

 - https://github.com/matrix-org/matrix-spec/issues/1917
 - https://github.com/matrix-org/matrix-spec/issues/852
 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033
 2024-08-07 11:27:50 -05:00
dependabot[bot]
30e9f6e469
Bump bytes from 1.6.1 to 1.7.1 (#17526) 2024-08-07 10:37:54 +01:00
dependabot[bot]
eb62d12063
Bump regex from 1.10.5 to 1.10.6 (#17527) 2024-08-07 10:37:13 +01:00
Erik Johnston
ceb3686dcd
Fixup sliding sync comment (#17531) 
c.f.
https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925
 2024-08-07 10:32:36 +01:00
Eric Eastwood
1dfa59b238
Sliding Sync: Add more tracing (#17514) 
Spawning from looking at a couple traces and wanting a little more info.

Follow-up to github.com/element-hq/synapse/pull/17501

The changes in this PR allow you to find slow Sliding Sync traces ignoring the
`wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user`
operation with `RESULT.result=true` indicating that it actually returned non-empty results.

If you want to find traces for your own user, you can use
`RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
 2024-08-06 11:43:43 -05:00
Andrew Morgan
bef6568537 Merge branch 'release-v1.113' into develop 2024-08-06 14:19:12 +01:00
Andrew Morgan
244a255065
Clarify auto_accept_invites.worker_to_run_on config docs (#17515) 2024-08-06 13:26:51 +01:00
Andrew Morgan
932cb0a928 1.113.0rc1 2024-08-06 12:24:47 +01:00
dependabot[bot]
2dad718265
Bump phonenumbers from 8.13.39 to 8.13.42 (#17521) 2024-08-06 11:47:19 +01:00
dependabot[bot]
5d8446298c
Bump towncrier from 23.11.0 to 24.7.1 (#17523) 2024-08-06 11:47:06 +01:00
dependabot[bot]
d845e939a9
Bump black from 24.4.2 to 24.8.0 (#17522) 2024-08-06 11:46:48 +01:00
dependabot[bot]
23727869c7
Bump serde_json from 1.0.121 to 1.0.122 (#17525) 2024-08-06 11:45:44 +01:00
Erik Johnston
c270355349
SS: Reset connection if token is unrecognized (#17529) 
This triggers the client to start a new sliding sync connection. If we
don't do this and the client asks for the full range of rooms, we end up
sending down all rooms and their state from scratch (which can be very
slow)

This causes things like
https://github.com/element-hq/element-x-ios/issues/3115 after we restart
the server

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
 2024-08-06 10:39:11 +01:00
Eric Eastwood
e3db7b2d81
Sliding Sync: Easier to understand timeline assertions in tests (#17511) 
Added `_assertTimelineEqual(...)` because I got fed up trying to
understand the crazy diffs from the standard
`self.assertEqual(...)`/`self.assertListEqual(...)`

Before:
```
[FAIL]
Traceback (most recent call last):
  File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync
    self.assertListEqual(
  File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual
    self.fail(msg)
twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU']

First differing element 0:
'$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA'
'$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E'

- ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA',
-  '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E',
? ^

+ ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E',
? ^

-  '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM']
+  '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA',
+  '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU']
```

After:

```
[FAIL]
Traceback (most recent call last):
  File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync
    self._assertTimelineEqual(
  File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual
    self._assertListEqual(
  File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual
    self.fail(f"{diff_message}\n{message}")
twisted.trial.unittest.FailTest: Items must
Expected items to be in actual ('?' = missing expected items):
 [
   (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4
   (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5
?  (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join
 ]
Actual ('+' = found expected items):
 [
+  (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5
+  (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4
   (9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3
 ]
```
 2024-08-05 13:20:15 -05:00
Eric Eastwood
2b620e0a15
Sliding Sync: Add typing notification extension (MSC3961) (#17505) 
[MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications

Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
 2024-07-31 13:20:23 -05:00
Eric Eastwood
39731bb205
Sliding Sync: Split and move tests (#17504) 
Split and move Sliding Sync tests so we have some more sane test file
sizes
 2024-07-31 12:20:46 -05:00
Eric Eastwood
1d6186265a
Sliding Sync: Fix limited response description (make accurate) (#17507) 2024-07-31 11:47:26 -05:00
Eric Eastwood
46de0ee16b
Sliding Sync: Update filters to be robust against remote invite rooms (#17450) 
Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to
be robust when dealing with remote invite rooms in Sliding Sync.

Part of
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync

Follow-up to https://github.com/element-hq/synapse/pull/17434

We now take into account current state, fallback to stripped state
for invite/knock rooms, then historical state. If we can't determine
the info needed to filter a room (either from state or stripped state),
it is filtered out.
 2024-07-30 13:20:29 -05:00
Eric Eastwood
b221f0b84b
Sliding Sync: Add receipts extension (MSC3960) (#17489) 
[MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension

Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
 2024-07-30 12:49:55 -05:00
Olivier 'reivilibre
b2c55bd049 Merge branch 'master' into develop 2024-07-30 18:09:05 +01:00
Olivier 'reivilibre
ed583d9c81 Merge branch 'release-v1.112' 2024-07-30 18:07:35 +01:00
dependabot[bot]
f76dc9923c
Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726 (#17497) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-30 17:33:43 +01:00
dependabot[bot]
7e997fb8b1
Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722 (#17496) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-30 17:33:18 +01:00
dependabot[bot]
dbc2290cbe
Bump bcrypt from 4.1.3 to 4.2.0 (#17495) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-30 17:32:49 +01:00
dependabot[bot]
2f6b86e79a
Bump serde_json from 1.0.120 to 1.0.121 (#17493) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-30 17:32:16 +01:00
Olivier 'reivilibre
37f9876ccf 1.112.0 2024-07-30 17:24:09 +01:00
reivilibre
8b449a8ce6 Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502) 
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.

Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7


Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
 2024-07-30 17:14:14 +01:00
Olivier 'reivilibre
53db8a914e Merge branch 'master' into develop 2024-07-30 17:10:46 +01:00
Olivier 'reivilibre
e4868f8a1e Add bold emphasis to some parts of the changelog 2024-07-30 16:23:58 +01:00
Olivier 'reivilibre
dcad81082c 1.111.1 2024-07-30 16:16:35 +01:00
reivilibre
c56b070e6f
Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502) 
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.

Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7


Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
 2024-07-30 15:23:23 +01:00
dependabot[bot]
be726724a8
Bump ruff from 0.5.4 to 0.5.5 (#17494) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-30 11:44:54 +01:00
Erik Johnston
62ae56a4ac
Add some more opentracing to sliding sync (#17501) 
This will make it easier to see what it is doing in jaeger.
 2024-07-30 10:54:11 +01:00
Richard van der Hoff
808dab0699
Fix failures property in /keys/query (#17499) 
Fixes: https://github.com/element-hq/synapse/issues/17498
Fixes: https://github.com/element-hq/element-web/issues/27867
 2024-07-30 09:51:24 +01:00
Erik Johnston
34306be5aa
Only send rooms with updates down sliding sync (#17479) 
Rather than always including all rooms in range.

Also adds a pre-filter to rooms that checks the stream change cache to
see if anything might have happened.

Based on #17447

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
 2024-07-30 09:30:44 +01:00
Erik Johnston
be4a16ff44
Sliding Sync: Track whether we have sent rooms down to clients (#17447) 
The basic idea is that we introduce a new token for a sliding sync
connection, which stores the mapping of room to room "status" (i.e. have
we sent the room down?). This token allows us to handle duplicate
requests properly. In future it can be used to store more
"per-connection" information safely.

In future this should be migrated into the DB, so its important that we
try to reduce the number of syncs where we need to update the
per-connection information. In this PoC this only happens when we: a)
send down a set of room for the first time, or b) we have previously
sent down a room and there are updates but we are not sending the room
down the sync (due to not falling in a list range)

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
 2024-07-29 22:45:48 +01:00
Eric Eastwood
568051c0f0
Refactor Sliding Sync tests to better utilize the SlidingSyncBase.do_sync(...) (pt. 2) (#17482) 
`SlidingSyncBase.do_sync()` for tests was first introduced in
https://github.com/element-hq/synapse/pull/17452

Part 1: https://github.com/element-hq/synapse/pull/17481
 2024-07-25 11:01:47 -05:00
Eric Eastwood
ebbabfe782
Refactor Sliding Sync tests to better utilize the SlidingSyncBase (pt. 1) (#17481) 
`SlidingSyncBase` for tests was first introduced in
https://github.com/element-hq/synapse/pull/17452

Part 2: https://github.com/element-hq/synapse/pull/17482
 2024-07-25 10:43:35 -05:00
YLong Shi
69ac4b6a6e
Update config_documentation - Change example of msisdn in allowed_local_3pids (#17476) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2024-07-25 11:07:44 +00:00
Eric Eastwood
729026e604
Sliding Sync: Add Account Data extension (MSC3959) (#17477) 
Extensions based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
 2024-07-24 17:10:38 -05:00
Erik Johnston
bdf37ad4c4
Sliding Sync: ensure bump stamp ignores backfilled events (#17478) 
Backfill events have a negative stream ordering, and so its not useful
to use to compare with other (positive) stream orderings.

Plus, the Rust SDK currently assumes `bump_stamp` is positive.
 2024-07-24 15:21:56 +01:00
Erik Johnston
8bbc98e66d
Use a new token format for sliding sync (#17452) 
This is in preparation for adding per-connection state.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
 2024-07-24 11:47:25 +01:00
Maciej Laskowski
4b9f4c2abf
Update debian template - new link to the delegation docs (#17475) 
Update debian template - new link to the delegation docs
 2024-07-24 10:32:56 +01:00
Devon Hudson
e8ee784c75
Address changelog review comments 2024-07-23 09:14:45 -06:00
Devon Hudson
48c1307911
1.112.0rc1 2024-07-23 09:01:43 -06:00
Erik Johnston
d225b6b3eb
Speed up SS room sorting (#17468) 
We do this by bulk fetching the latest stream ordering.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2024-07-23 14:03:14 +01:00
reivilibre
1daae43f3a
Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. (#17428) 
Introduced in: #17215

This caused us a minor bit of grief as the volume of logs produced was
much higher than normal

---------

Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
 2024-07-23 11:51:34 +01:00
Michael Hollister
a9ee832e48
Fixed presence results not returning offline users on initial sync (#17231) 
This is to address an issue in which `m.presence` results on initial
sync are not returning entries of users who are currently offline.

The original behaviour was from
https://github.com/element-hq/synapse/issues/1535

This change is useful for applications that use the
presence system for tracking user profile information/updates (e.g.
https://github.com/element-hq/synapse/pull/16992 or for profile status
messages).

This is gated behind a new configuration option to avoid performance
impact for applications that don't need this, as a pragmatic solution
for now.
 2024-07-23 09:59:24 +00:00