Henrique
beb7a951f4
docs: add note about PYTHONMALLOC for accurate jemalloc memory tracking ( #17709 )
...
Added a note in the documentation suggesting that users may set
`PYTHONMALLOC=malloc` when using `jemalloc`. This allows jemalloc to
track memory usage more accurately by bypassing Python's internal
small-object allocator (`pymalloc`), helping to ensure that
`cache_autotuning` functions as expected.
This doc change aims to provide more clarity for users configuring
jemalloc with Synapse.
Based on:
4ac783549c/synapse/metrics/jemalloc.py (L198-L201)
2024-10-07 08:37:39 +00:00
Andrew Ferrazzutti
9920417723
Don't say MSC4140 is supported when it's disabled ( #17780 )
Build release artifacts / Build sdist (push) Has been cancelled
Tests / changes (push) Has been cancelled
Tests / lint-newsfile (push) Has been cancelled
Tests / lint (push) Has been cancelled
Deploy the documentation / GitHub Pages (push) Has been cancelled
Build release artifacts / Build .deb packages (push) Has been cancelled
Build release artifacts / Attach assets to release (push) Has been cancelled
Tests / Typechecking (push) Has been cancelled
Tests / check-sampleconfig (push) Has been cancelled
Tests / check-schema-delta (push) Has been cancelled
Tests / lint-pydantic (push) Has been cancelled
Tests / lint-clippy (push) Has been cancelled
Tests / lint-clippy-nightly (push) Has been cancelled
Tests / lint-rustfmt (push) Has been cancelled
Tests / lint-readme (push) Has been cancelled
Tests / linting-done (push) Has been cancelled
Tests / calculate-test-jobs (push) Has been cancelled
Tests / trial (push) Has been cancelled
Tests / trial-olddeps (push) Has been cancelled
Tests / trial-pypy (all, pypy-3.8) (push) Has been cancelled
Tests / sytest (push) Has been cancelled
Tests / export-data (push) Has been cancelled
Tests / portdb (11, 3.8) (push) Has been cancelled
Tests / portdb (15, 3.11) (push) Has been cancelled
Tests / complement (monolith, Postgres) (push) Has been cancelled
Tests / complement (monolith, SQLite) (push) Has been cancelled
Tests / complement (workers, Postgres) (push) Has been cancelled
Tests / cargo-test (push) Has been cancelled
Tests / cargo-bench (push) Has been cancelled
Tests / tests-done (push) Has been cancelled
2024-10-04 13:42:34 +01:00
Andrew Morgan
316d635906
Fix NAME attribute of ReplicationRemovePusherRestServlet ( #17779 )
Build release artifacts / Build sdist (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / lint-newsfile (push) Waiting to run
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / lint (push) Blocked by required conditions
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / Typechecking (push) Blocked by required conditions
Tests / trial (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
2024-10-04 09:53:35 +01:00
Dirk Klimpel
8bbe66a9b9
explain load balancing for federation_sender_instances ( #17776 )
...
Build release artifacts / Build wheels on macos-12 for x86_64 (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / lint-newsfile (push) Waiting to run
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / lint (push) Blocked by required conditions
Tests / Typechecking (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / trial (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
Adding information on how the load is distributed for
`federation_sender_instances`.
Thx to @devonh for the information.
causal source:
c2e5e9e67c/synapse/config/_base.py (L946-L989)https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
---------
Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>
2024-10-03 22:01:33 +00:00
Erik Johnston
81e0f57800
Fix perf when streams don't change often ( #17767 )
...
Tests / check-sampleconfig (push) Has been cancelled
Tests / complement (monolith, Postgres) (push) Has been cancelled
Tests / complement (monolith, SQLite) (push) Has been cancelled
Tests / check-lockfile (push) Has been cancelled
Tests / lint-crlf (push) Has been cancelled
Tests / lint-newsfile (push) Has been cancelled
Deploy the documentation / GitHub Pages (push) Has been cancelled
Build release artifacts / Build .deb packages (push) Has been cancelled
Build release artifacts / Attach assets to release (push) Has been cancelled
Tests / check-schema-delta (push) Has been cancelled
Tests / lint (push) Has been cancelled
Tests / Typechecking (push) Has been cancelled
Tests / lint-pydantic (push) Has been cancelled
Tests / lint-clippy (push) Has been cancelled
Tests / lint-clippy-nightly (push) Has been cancelled
Tests / lint-rustfmt (push) Has been cancelled
Tests / lint-readme (push) Has been cancelled
Tests / linting-done (push) Has been cancelled
Tests / calculate-test-jobs (push) Has been cancelled
Tests / trial (push) Has been cancelled
Tests / trial-olddeps (push) Has been cancelled
Tests / trial-pypy (all, pypy-3.8) (push) Has been cancelled
Tests / sytest (push) Has been cancelled
Tests / export-data (push) Has been cancelled
Tests / portdb (11, 3.8) (push) Has been cancelled
Tests / portdb (15, 3.11) (push) Has been cancelled
Tests / complement (workers, Postgres) (push) Has been cancelled
Tests / cargo-test (push) Has been cancelled
Tests / cargo-bench (push) Has been cancelled
Tests / tests-done (push) Has been cancelled
There is a bug with the `StreamChangeCache` where it would incorrectly
return that all entities had changed if asked for entities changed
*since* the earliest stream position.
Note that for streams we use the inequalities: `$min_stream_id <
stream_id <= $max_stream_id`, i.e. when we ask the stream change cache
for all things that have changed since `$stream_id` we don't care for
events that happened *at* `$stream_id`.
Specifically: `_earliest_known_stream_pos` is the position at which we
know that we'll have entries for all changes since that point, we can
use the cache for any stream IDs that equal
`_earliest_known_stream_pos`.
`_earliest_known_stream_pos` is set in three places:
- On startup we set it either to:
- the current maximum stream ID, with not prefilled values; or
- the minimum of the latest N values we pulled from the DB
- When we evict items from the bottom, we set it to the stream ID of the
evicted items.
This was changed in https://github.com/matrix-org/synapse/pull/14435 ,
but I think we were overly conservative there.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-09-30 13:52:33 +01:00
Erik Johnston
ae4862c38f
Optimise notifier mk2 ( #17766 )
...
Based on #17765 .
Basically the idea is to reduce the overhead of calling
`ObservableDeferred` in a loop. The two gains are: a) just using a list
of deferreds rather than the machinery of `ObservableDeferred`, and b)
only calling `PreseverLoggingContext` once.
`PreseverLoggingContext` in particular is expensive to call a lot as
each time it needs to call `get_thread_resource_usage` twice, so that it
an update the CPU metrics of the log context.
2024-09-30 13:32:31 +01:00
Erik Johnston
5210565c12
Reduce overhead of sliding sync E2EE loops ( #17771 )
...
Mainly toning down logging and only calling
`get_membership_from_event_ids` if something has changed.
2024-09-30 13:00:14 +01:00
Erik Johnston
de955293cf
Add fast path for sliding sync streams that only ask for extensions ( #17768 )
...
Principally useful for EX e2ee sliding sync connections.
2024-09-30 12:59:50 +01:00
Erik Johnston
93889eb2e7
Optimise notifier ( #17765 )
...
The notifier is quite inefficient when it has to wake up many user
streams all at once
From a silly benchmark this takes the time to notify 1M user streams
from ~30s to ~5s
2024-09-30 12:58:13 +01:00
Erik Johnston
ece66ba61c
Minor perf speed up for large accounts on SSS ( #17751 )
...
This works as instead of passing *all* rooms to `record_sent_rooms` we
only need to pass rooms that were previously not in the LIVE state.
This came from a py-spy where we were spending ~10% CPU calling these
functions. Note that `record_sent_rooms` is a no-op for rooms that are
already in the `LIVE` state, so we only need to call them for
`PREVIOUSLY` or `INITIAL` rooms.
2024-09-30 12:58:02 +01:00
Erik Johnston
f144b4c7e9
Remove spurious TODO in debian install step ( #17749 )
...
This was a note added in the PR to move to AGPL, which we failed to
remove before landing.
(The context for this was that we needed to decide if we were going to
change which debian repository we published too, but decided not to in
the end)
2024-09-26 13:18:28 +01:00
Quentin Gliech
386cabda83
1.116.0rc1
2024-09-25 11:34:36 +02:00
dependabot[bot]
f53a3a56e2
Bump treq from 23.11.0 to 24.9.1 ( #17744 )
...
Bumps [treq](https://github.com/twisted/treq ) from 23.11.0 to 24.9.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/twisted/treq/releases ">treq's
releases</a>.</em></p>
<blockquote>
<h2>Treq 24.9.0</h2>
<h2>Features</h2>
<ul>
<li>treq now ships type annotations. (<a
href="https://redirect.github.com/twisted/treq/issues/366 ">#366</a>)</li>
<li>The new <code>treq.cookies</code> module provides helper functions
for working with <code>http.cookiejar.Cookie</code> and
<code>CookieJar</code> objects. (<a
href="https://redirect.github.com/twisted/treq/issues/384 ">#384</a>)</li>
<li>Python 3.13 is now supported. (<a
href="https://redirect.github.com/twisted/treq/issues/391 ">#391</a>)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li><code>treq.content.text_content()</code> no longer generates
deprecation warnings due to use of the <code>cgi</code> module. (<a
href="https://redirect.github.com/twisted/treq/issues/355 ">#355</a>)</li>
</ul>
<h2>Deprecations and Removals</h2>
<ul>
<li>Mixing the <em>json</em> argument with <em>files</em> or
<em>data</em> now raises <code>TypeError</code>. (<a
href="https://redirect.github.com/twisted/treq/issues/297 ">#297</a>)</li>
<li>Passing non-string (<code>str</code> or <code>bytes</code>) values
as part of a dict to the <em>headers</em> argument now results in a
<code>TypeError</code>, as does passing any collection other than a
<code>dict</code> or <code>Headers</code> instance. (<a
href="https://redirect.github.com/twisted/treq/issues/302 ">#302</a>)</li>
<li>Support for Python 3.7 and PyPy 3.8, which have reached end of
support, has been dropped. (<a
href="https://redirect.github.com/twisted/treq/issues/378 ">#378</a>)</li>
</ul>
<h2>Misc</h2>
<ul>
<li><a
href="https://redirect.github.com/twisted/treq/issues/336 ">#336</a>, <a
href="https://redirect.github.com/twisted/treq/issues/382 ">#382</a>, <a
href="https://redirect.github.com/twisted/treq/issues/395 ">#395</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/twisted/treq/blob/trunk/CHANGELOG.rst ">treq's
changelog</a>.</em></p>
<blockquote>
<h1>24.9.1 (2024-09-19)</h1>
<h2>Bugfixes</h2>
<ul>
<li>treq has vendored its dependency on the <code>multipart</code>
library to avoid import
conflicts with <code>python-multipart</code>; it should now be
installable alongside
that library. (<code>[#399 ](https://github.com/twisted/treq/issues/399 )
<https://github.com/twisted/treq/issues/399> ;</code>__)</li>
</ul>
<h1>24.9.0 (2024-09-17)</h1>
<h2>Features</h2>
<ul>
<li>treq now ships type annotations.
(<code>[#366 ](https://github.com/twisted/treq/issues/366 )
<https://github.com/twisted/treq/issues/366> ;</code>__)</li>
<li>The new :mod:<code>treq.cookies</code> module provides helper
functions for working with <code>http.cookiejar.Cookie</code> and
<code>CookieJar</code> objects.
(<code>[#384 ](https://github.com/twisted/treq/issues/384 )
<https://github.com/twisted/treq/issues/384> ;</code>__)</li>
<li>Python 3.13 is now supported.
(<code>[#391 ](https://github.com/twisted/treq/issues/391 )
<https://github.com/twisted/treq/issues/391> ;</code>__)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>:mod:<code>treq.content.text_content()</code> no longer generates
deprecation warnings due to use of the <code>cgi</code> module.
(<code>[#355 ](https://github.com/twisted/treq/issues/355 )
<https://github.com/twisted/treq/issues/355> ;</code>__)</li>
</ul>
<h2>Deprecations and Removals</h2>
<ul>
<li>Mixing the <em>json</em> argument with <em>files</em> or
<em>data</em> now raises <code>TypeError</code>.
(<code>[#297 ](https://github.com/twisted/treq/issues/297 )
<https://github.com/twisted/treq/issues/297> ;</code>__)</li>
<li>Passing non-string (<code>str</code> or <code>bytes</code>) values
as part of a dict to the <em>headers</em> argument now results in a
<code>TypeError</code>, as does passing any collection other than a
<code>dict</code> or <code>Headers</code> instance.
(<code>[#302 ](https://github.com/twisted/treq/issues/302 )
<https://github.com/twisted/treq/issues/302> ;</code>__)</li>
<li>Support for Python 3.7 and PyPy 3.8, which have reached end of
support, has been dropped.
(<code>[#378 ](https://github.com/twisted/treq/issues/378 )
<https://github.com/twisted/treq/issues/378> ;</code>__)</li>
</ul>
<h2>Misc</h2>
<ul>
<li><code>[#336 ](https://github.com/twisted/treq/issues/336 )
<https://github.com/twisted/treq/issues/336> ;</code><strong>,
<code>[#382 ](https://github.com/twisted/treq/issues/382 )
<https://github.com/twisted/treq/issues/382> ;</code></strong>,
<code>[#395 ](https://github.com/twisted/treq/issues/395 )
<https://github.com/twisted/treq/issues/395> ;</code>__</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="caaf9fcb629cedb088b4https://redirect.github.com/twisted/treq/issues/400 ">#400</a> from
twisted/vendor-multipart-for-now</li>
<li><a
href="4aa1ee8a3cd7c16de8f54fd3c842c2f0a5148cba7f16b87f0a1526431a37https://github.com/defnull/multipa ">https://github.com/defnull/multipa </a>...</li>
<li><a
href="7c52d4917fca3966f57ahttps://redirect.github.com/twisted/treq/issues/398 ">#398</a> from
twisted/397-release-24.9.0</li>
<li>Additional commits viewable in <a
href="https://github.com/twisted/treq/compare/release-23.11.0...treq-24.9.1 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Quentin Gliech <quenting@element.io>
2024-09-25 11:19:03 +02:00
V02460
2fc43e4219
Remove the deprecated cgi module ( #17741 )
...
Removes all uses of the `cgi` module from Synapse. It was deprecated in
Python version 3.11 and removed in version 3.13 ([“dead
battery”](https://docs.python.org/3.13/whatsnew/3.13.html#pep-594-remove-dead-batteries-from-the-standard-library )).
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
---------
Co-authored-by: Quentin Gliech <quenting@element.io>
2024-09-25 11:15:34 +02:00
Erik Johnston
aad26cb93f
Never return negative bump stamp ( #17748 )
...
Fixes #17737
2024-09-24 10:07:23 +00:00
Andrew Ferrazzutti
5173741c71
Support MSC4140: Delayed events (Futures) ( #17326 )
2024-09-23 13:33:48 +01:00
Erik Johnston
75e2c17d2a
Speed up sorting of sliding sync rooms in initial request ( #17734 )
...
We do this by using the event stream cache.
---------
Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>
2024-09-20 08:12:56 +01:00
Erik Johnston
a851f6b237
Sliding sync: Add connection tracking to the account_data extension ( #17695 )
...
This is basically exactly the same logic as for receipts. Essentially we
just need to track which room account data we have and haven't sent down
to clients, and use that when we pull stuff out.
I think this just needs a couple of extra tests written
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-19 19:51:51 +01:00
Eric Eastwood
c2e5e9e67c
Sliding Sync: Avoid fetching left rooms and add back newly_left rooms ( #17725 )
...
Performance optimization: We can avoid fetching rooms that the user has
left themselves (which could be a significant amount), then only add
back rooms that the user has `newly_left` (left in the token range of an
incremental sync). It's a lot faster to fetch less rooms than fetch them
all and throw them away in most cases. Since the user only leaves a room
(or is state reset out) once in a blue moon, we can avoid a lot of work.
Based on @erikjohnston's branch, erikj/ss_perf
---------
Co-authored-by: Erik Johnston <erik@matrix.org>
2024-09-19 10:07:18 -05:00
Erik Johnston
07a51d2a56
Fix sliding sync for rooms with unknown room version ( #17733 )
...
Follow on from #17727
2024-09-19 14:01:11 +01:00
Eric Eastwood
83fc225030
Sliding Sync: Add cache to get_tags_for_room(...) ( #17730 )
...
Add cache to `get_tags_for_room(...)`
This helps Sliding Sync because `get_tags_for_room(...)` is going to be
used in https://github.com/element-hq/synapse/pull/17695
Essentially, we're just trying to match `get_account_data_for_room(...)`
which already has a tree cache.
2024-09-19 12:43:26 +01:00
Eric Eastwood
a9c0e27eb7
Sliding Sync: No need to sort if the range is large enough to cover all of the rooms ( #17731 )
...
No need to sort if the range is large enough to cover all of the rooms
in the list. Previously, we would only do this optimization if the range
was exactly large enough.
Follow-up to https://github.com/element-hq/synapse/pull/17672
2024-09-19 09:33:34 +01:00
Eric Eastwood
faf5b40520
Sliding Sync: Fix _bulk_get_max_event_pos(...) being inefficient ( #17728 )
...
Fix `_bulk_get_max_event_pos(...)` being inefficient. It kept adding all
of the `batch_results` to the `results` over and over every time we
checked a single room in the batch.
I think we still ended up with the right answer before because we
accumulate `recheck_rooms` and actually recheck them to overwrite the
bad data we wrote to the `results` before.
Introduced in
https://github.com/element-hq/synapse/pull/17606/files#diff-cbd54e4b5a2a1646299d659a2d5884d6cb14e608efd2e1658e72b465bb66e31bR1481
2024-09-19 09:32:16 +01:00
Eric Eastwood
af998e6c66
Sliding sync: Ignore invites from ignored users ( #17729 )
...
`m.ignored_user_list` in account data
2024-09-18 18:09:23 -05:00
Eric Eastwood
61b7c31772
Sliding Sync: Shortcut for checking if certain background updates have completed ( #17724 )
...
Shortcut for checking if certain background updates have completed
Pulling this change out from one of @erikjohnston's branches
(https://github.com/element-hq/synapse/compare/develop...erikj/ss_perf )
---------
Co-authored-by: Erik Johnston <erikj@element.io>
2024-09-18 13:12:14 -05:00
Kegan Dougal
3c8a116e1a
Sliding Sync: bugfix: ensure we can sync with SSS even with missing rooms ( #17727 )
...
Fixes https://github.com/element-hq/element-x-ios/issues/3300
Some rooms are missing from `sliding_sync_joined_rooms`. When this
happens, the first call will succeed, but any subsequent calls for this
room ID will cause the cache to return `None` for the room ID, rather
than not having the key at all. This then causes the `<=` check to
throw.
Root cause: https://github.com/element-hq/synapse/issues/17726
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [ ] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
2024-09-18 16:25:50 +00:00
Shay
51dd4df0a3
Add an Admin API endpoint to redact all a user's events ( #17506 )
2024-09-18 10:08:01 +00:00
Eric Eastwood
8881ad6d4b
Sliding Sync: Short-circuit have_finished_sliding_sync_background_jobs ( #17723 )
...
We only need to check it if returned bump stamp is `None`, which is rare.
Pulling this change out from one of @erikjohnston's branches
(https://github.com/element-hq/synapse/compare/develop...erikj/ss_perf )
2024-09-17 17:36:59 -05:00
Eric Eastwood
03937a1cae
Sliding Sync: Return room tags in account data extension ( #17707 )
...
The account data extension was also updated to avoid copies when we pull
the data out of the cache.
Fix https://github.com/element-hq/synapse/issues/17694
2024-09-16 13:47:35 -05:00
David Baker
4ac783549c
Sliding Sync: Support filtering by 'tags' / 'not_tags' in SSS ( #17662 )
...
This appears to be enough to make Element Web work (or at least move it
on to the next hurdle)
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-12 20:18:19 -05:00
Erik Johnston
1cb84aaab5
Sliding Sync: Increase concurrency of sliding sync a bit ( #17696 )
...
For initial requests a typical page size is 20 rooms, so we may as well
do the batching as 20.
This should speed up bigger syncs a little bit.
2024-09-12 16:36:16 -05:00
Eric Eastwood
9b83fb7c16
Sliding Sync: Move filters tests to rest layer ( #17703 )
...
Move filters tests to rest layer in order to test the new (with sliding
sync tables) and fallback paths that Sliding Sync can use.
Also found a bug in the new path because it's not being tested which is
also fixed in this PR. We now take into account `has_known_state` when
filtering.
Spawning from
https://github.com/element-hq/synapse/pull/17662#discussion_r1755574791 .
This should have been done when we started using the new sliding sync
tables in https://github.com/element-hq/synapse/pull/17630
2024-09-12 15:27:03 -05:00
Andrew Morgan
c5b4be6d07
Merge branch 'release-v1.115' into develop
2024-09-12 13:05:43 +01:00
Andrew Morgan
4c66a7cbed
1.115.0rc2
2024-09-12 11:10:31 +01:00
Éloi Rivard
ebad618bf0
import pydantic objects from the _pydantic_compat module ( #17667 )
...
This PR changes `from pydantic import BaseModel` to `from
synapse._pydantic_compat import BaseModel` (as well as `constr`,
`conbytes`, `conint`, `confloat`).
It allows `check_pydantic_models.py` to mock those pydantic objects only
in the synapse module, and not interfere with pydantic objects in
external dependencies.
This should solve the CI problems for #17144 , which breaks because
`check_pydantic_models.py` patches pydantic models from
[scim2-models](https://scim2-models.readthedocs.io/ ).
/cc @DMRobertson @gotmax23
fixes #17659
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
2024-09-11 21:01:43 +00:00
Eric Eastwood
16af80b8fb
Sliding Sync: Use Sliding Sync tables for sorting ( #17693 )
...
Use Sliding Sync tables for sorting
(`bulk_get_last_event_pos_in_room_before_stream_ordering(...)` ->
`_bulk_get_max_event_pos(...)`)
2024-09-11 12:16:24 -05:00
Eric Eastwood
e4a1f271b9
Sliding Sync: Make sure we get up-to-date information from get_sliding_sync_rooms_for_user(...) ( #17692 )
...
We need to bust the `get_sliding_sync_rooms_for_user`
cache when the room encryption is updated and any
other field that is used in the query.
Follow-up to https://github.com/element-hq/synapse/pull/17630
- Bust cache for membership change (cross-reference
`get_rooms_for_user`)
- Bust cache for room `encryption` (cross-reference
`get_room_encryption`)
- Bust cache for `forgotten` (cross-reference
`did_forget`/`get_forgotten_rooms_for_user`)
2024-09-11 12:13:54 -05:00
Erik Johnston
76f7c91e44
Sliding sync: don't fetch room summary for named rooms. ( #17683 )
...
For rooms with a name we can skip fetching a full room summary, as we
don't need to calculate heroes, and instead just fetch the room counts
directly.
This also changes things to not return counts and heroes for non-joined
rooms. For left/banned rooms we were returning zero values anyway, and
for invite/knock rooms we don't really want to leak such information
(even if some of is included in the stripped state).
2024-09-11 16:42:50 +01:00
Erik Johnston
b732d13d4c
Sliding sync: various fixups to the background update ( #17652 )
2024-09-11 16:42:15 +01:00
Erik Johnston
596b96411b
Sliding sync: various fixups to the background update ( #17652 )
2024-09-11 15:38:46 +01:00
Erik Johnston
f6c2b0ec2e
Sliding sync: don't fetch room summary for named rooms. ( #17683 )
...
For rooms with a name we can skip fetching a full room summary, as we
don't need to calculate heroes, and instead just fetch the room counts
directly.
This also changes things to not return counts and heroes for non-joined
rooms. For left/banned rooms we were returning zero values anyway, and
for invite/knock rooms we don't really want to leak such information
(even if some of is included in the stripped state).
2024-09-11 13:16:57 +01:00
Travis Ralston
a7fcac5648
Enable guest access on new media endpoints, per MSC4189 ( #17675 )
2024-09-10 18:29:24 +01:00
V02460
e06e3c4004
Add config option turn_shared_secret_path ( #17690 )
...
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-09-10 17:27:46 +00:00
Devon Hudson
59bcbcec0a
1.115.0rc1
2024-09-10 08:42:01 -06:00
Erik Johnston
b3047f3f17
Sliding sync: various fixups to the sliding sync joined room background job ( #17673 )
...
Follow-up to #17652 , https://github.com/element-hq/synapse/pull/17641 ,
https://github.com/element-hq/synapse/pull/17634 ,
https://github.com/element-hq/synapse/pull/17631 and
https://github.com/element-hq/synapse/pull/17632 to fix-up
https://github.com/element-hq/synapse/pull/17512
2024-09-10 10:22:46 +01:00
Erik Johnston
9689ac3294
Sliding Sync: Look for bump _stamp in the room timeline ( #17684 )
...
This allows us to skip checking the database a lot of the time.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-10 10:20:30 +01:00
Erik Johnston
588e5b521d
Sliding Sync: Retrieve fewer events from DB in sync ( #17688 )
...
When using timeline limit of 1 we end up fetching 2 events from the DB
purely to tell if the response was "limited" or not. Lets not do that.
2024-09-10 09:52:42 +01:00
Eric Eastwood
515c1cc0a1
Sliding Sync: Add comment to explain extra case where you can be invited -> banned -> unbanned ( #17654 )
...
Add comment to explain extra case where you can be
invited -> banned -> unbanned and we want to be able
to find the invite event.
Follow-up to https://github.com/element-hq/synapse/pull/17636#discussion_r1738993330
2024-09-09 17:55:59 -05:00
Eric Eastwood
e1ed959a68
Sliding Sync: Get bump_stamp from new sliding sync tables because it's faster ( #17658 )
...
Get `bump_stamp` from [new sliding sync
tables](https://github.com/element-hq/synapse/pull/17512 ) which should
be faster (performance) than flipping through the latest events in the
room.
2024-09-09 16:41:25 +01:00
Eric Eastwood
5389374ef8
Sliding Sync: Speed up incremental sync by avoiding extra work ( #17665 )
...
Speed up incremental sync by avoiding extra work. We first look at the
state delta changes and only fetch and calculate further derived things
if they have changed.
2024-09-09 10:36:22 +01:00
