286 lines
12 KiB
Markdown
286 lines
12 KiB
Markdown
|
## Key Takeaways
|
||
|
|
|
||
|
|
* Matrix Dendrite can be run behind Tailscale Funnel to shape the
|
||
|
|
network logically. This way we can avoid opening our local
|
||
|
|
infrastructure directly to the Internet.
|
||
|
|
|
||
|
|
* Using Traefik we can route traffic for security monitoring
|
||
|
|
and to apply policies, also adding centralised logging such as
|
||
|
|
with LimaCharlie Adapters.
|
||
|
|
|
||
|
|
* Bandwidth limitations in Tailscale Funnel and general stability
|
||
|
|
have been a breeze so far for a small Dendrite-based Matrix server.
|
||
|
|
|
||
|
|
## Background
|
||
|
|
|
||
|
|
Matrix has been quite a ride for me since I first deployed a Synapse server
|
||
|
|
in 2016. I've trained others in the cyber security community for it
|
||
|
|
and also decomissioned Matrix for XMPP, not only once, but twice!
|
||
|
|
|
||
|
|
In the end, I've decided that Matrix is the currently best common
|
||
|
|
denominator between safety, security and usability. Slack has
|
||
|
|
been my goto over the past couple of years, but having Matrix as an
|
||
|
|
option and for select conversation has been good for the inherent
|
||
|
|
vulnerability and complexity of the Slack ecosystem when used in a
|
||
|
|
cyber security context.
|
||
|
|
|
||
|
|
While the Matrix ecosystem are driven forward by the Element client
|
||
|
|
and Dendrite Homeserver in particular, other parts of infrastructure
|
||
|
|
components evolves as well. Due to this I recently sat down to migrate my
|
||
|
|
Synapse server on OpenBSD to Docker (on Debian for now).
|
||
|
|
|
||
|
|
## Architecture Using Tailscale Funnel+Proxy, Docker, Traefik and Dendrite
|
||
|
|
|
||
|
|
When redesigning my setup, I wanted to put emphasis on a sane flow
|
||
|
|
that is re-usable across Docker setups. I also wanted something with easy
|
||
|
|
visibility and traceability. Since Tailscale recently announced public
|
||
|
|
availability of Tailscale Funnel I decided to move the entrypoint of my
|
||
|
|
infrastructure to a Funnel.
|
||
|
|
|
||
|
|
Summarized I wanted to use the following components:
|
||
|
|
|
||
|
|
1. Docker: Recent pain with upgrading Postgres on OpenBSD,
|
||
|
|
convinced me to take the leap to Docker on Debian for stability
|
||
|
|
and compatibility
|
||
|
|
|
||
|
|
2. Dendrite: The next Golang-based Homeserver for Matrix, supported
|
||
|
|
by a PostgreSQL database.
|
||
|
|
|
||
|
|
3. Traefik: Reverse proxy for controlling and auditing access to the
|
||
|
|
Docker network in addition to the Tailscale ACL which if focused on
|
||
|
|
transport over application security.
|
||
|
|
|
||
|
|
4. Tailscale: Software-defined infrastructure building on Wireguard.
|
||
|
|
|
||
|
|
[^tailscale]: Tailscale is a coordination layer built on top of the
|
||
|
|
Wireguard VPN protocol. It makes it easy to control on the application
|
||
|
|
layer which accounts, devices etc should have access to what resources
|
||
|
|
and has a great advantage for inventory that it is opt-in.
|
||
|
|
|
||
|
|
Organizing these components into the network flow chart shown below, we can see
|
||
|
|
that the only external point of contact for a Matrix device or federated homerserver will be
|
||
|
|
`matrix.{{tailnet-id}}.ts.net`. This is an ingress point that Tailscale
|
||
|
|
has made available through their so-called Funnel. For all practical purposes it is a
|
||
|
|
tunnel that connects the Taiscale Docker container with the Internet
|
||
|
|
for non-Tailscale devices and for Matrix federation.
|
||
|
|
|
||
|
|
We will open for port 443 to allow for client traffic, and port 8443 that allows
|
||
|
|
for federation with homeservers such as `matrix.org`. These both reads from static
|
||
|
|
hosting at Firebase to figure out where to go. The .well-known files is step 1 for
|
||
|
|
both clients and servers when you'd like to connect with me at `@tommy:rodl.no`.
|
||
|
|
|
||
|
|
```
|
||
|
|
┌────────────────┐ ┌───────────┐
|
||
|
|
│ Matrix Client │ ┌─────┐ │ Tailscale │- Proxy-mode to localhost
|
||
|
|
│ Device │──443────┬────────▶│ │────────────────▶│ container │- socat tcp/tls to Traefik
|
||
|
|
└────────────────┘ │ └─────┘ └───────────┘
|
||
|
|
│ │ matrix.{{tailnet-id}}.ts.net │
|
||
|
|
│ Tailscale ingress node │
|
||
|
|
│ │ ▼
|
||
|
|
▼ │ ┌───────────┐
|
||
|
|
┌──────────────────────────┐ │ │ Traefik │- Tailscale 3.0 mode certs
|
||
|
|
│rodl.no/.well-known/client│ │ │ container │- Terminate TLS
|
||
|
|
├──────────────────────────┤ │ │ │- Route matrix.*.ts.net to Dendrite
|
||
|
|
│rodl.no/.well-known/server│ │ └───────────┘
|
||
|
|
└──────────────────────────┘ │ │
|
||
|
|
▲ │ │
|
||
|
|
│ │ ▼
|
||
|
|
│ ┌───────────┐
|
||
|
|
│ │ │ Dendrite │
|
||
|
|
┌────────────────┐ │ │ container │
|
||
|
|
│ Other Matrix │ │ ├───────────┤
|
||
|
|
│ Homeserver │──8443───┘ │ Postgres │
|
||
|
|
└────────────────┘ │ container │
|
||
|
|
└───────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
The Tailscale container is a lightly modified Tailscale image as shown in `Dockerfile.tailscale`
|
||
|
|
below. The Dockerfile extends the proxying capabilities of Tailscale (Tailscale only supports
|
||
|
|
localhost) to forward TCP to another container.
|
||
|
|
|
||
|
|
```docker
|
||
|
|
FROM tailscale/tailscale:stable
|
||
|
|
|
||
|
|
RUN apk add socat
|
||
|
|
|
||
|
|
COPY config/tailscale/startup.sh /tmp/startup.sh c
|
||
|
|
```
|
||
|
|
|
||
|
|
`startup.sh` contains the following. The serve commands enables local proxying,
|
||
|
|
while the funnel opens up port 443 and 8443 to the world, and finally socat forwards
|
||
|
|
TCP to the static IP of the Traefik container.
|
||
|
|
|
||
|
|
```sh
|
||
|
|
#!/usr/bin/env sh
|
||
|
|
|
||
|
|
tailscaled &
|
||
|
|
|
||
|
|
sleep 5
|
||
|
|
|
||
|
|
tailscale serve tcp:443 tcp://127.0.0.1:8000
|
||
|
|
|
||
|
|
tailscale serve tcp:8443 tcp://127.0.0.1:8001
|
||
|
|
|
||
|
|
tailscale funnel 443 on
|
||
|
|
|
||
|
|
tailscale funnel 8443 on
|
||
|
|
|
||
|
|
socat -v tcp-listen:8000,fork,reuseaddr tcp:172.25.0.10:4443 &
|
||
|
|
|
||
|
|
socat -v tcp-listen:8001,fork,reuseaddr tcp:172.25.0.10:8443
|
||
|
|
```
|
||
|
|
|
||
|
|
Traefik is the receiver of the TLS traffic to `matrix.{{tailnet-id}}.ts.net`. When Traefik
|
||
|
|
sees traffic to this domain on port 8443 and 443 it routes it to the Dendrite container.
|
||
|
|
This is also a central vantage point where one can enforce policies and do security monitoring.
|
||
|
|
It is also re-usable across networks with various services and can easily embed e.g. a
|
||
|
|
LimaCharlie adapter for central logging.
|
||
|
|
|
||
|
|
[^limacharlie]: [LimaCharlie](https://limacharlie.io) is a transparent security infrastructure
|
||
|
|
vendor focusing on inputs and outputs, in combination with an excellent detection and transform
|
||
|
|
engine. LimaCharlie adapters are described here:
|
||
|
|
https://doc.limacharlie.io/docs/documentation/73a613e8e43ed-lima-charlie-adapter
|
||
|
|
|
||
|
|
Finally we communicate with Dendrite, but the external devices and homeservers will only
|
||
|
|
see `matrix.{{tailnet-id}}.ts.net`.
|
||
|
|
|
||
|
|
## Now in Docker Compose
|
||
|
|
|
||
|
|
An example of how to do this with docker-compose is shown below with the following services
|
||
|
|
structure:
|
||
|
|
|
||
|
|
* tailscale
|
||
|
|
* reverse-proxy
|
||
|
|
* postgres
|
||
|
|
* dendrite
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
version: '3'
|
||
|
|
|
||
|
|
networks:
|
||
|
|
web:
|
||
|
|
name: web
|
||
|
|
external: true
|
||
|
|
matrix-internal:
|
||
|
|
name: matrix-internal
|
||
|
|
external: true
|
||
|
|
|
||
|
|
services:
|
||
|
|
tailscale:
|
||
|
|
hostname: matrix
|
||
|
|
#image: tailscale/tailscale:stable
|
||
|
|
build:
|
||
|
|
context: .
|
||
|
|
dockerfile: Dockerfile.tailscale
|
||
|
|
command: /tmp/startup.sh
|
||
|
|
restart: unless-stopped
|
||
|
|
env_file:
|
||
|
|
- ./config/tailscale/.tailscale_env
|
||
|
|
network_mode: host
|
||
|
|
privileged: true
|
||
|
|
cap_add: # Required for tailscale to work
|
||
|
|
- net_admin
|
||
|
|
- sys_module
|
||
|
|
volumes:
|
||
|
|
- /dev/net/tun:/dev/net/tun
|
||
|
|
- ./data/tailscale/lib:/var/lib/tailscale
|
||
|
|
- ./data/tailscale/run:/var/run/tailscale
|
||
|
|
|
||
|
|
reverse-proxy:
|
||
|
|
depends_on:
|
||
|
|
- tailscale
|
||
|
|
# The official v2 Traefik docker image
|
||
|
|
image: traefik:v3.0.0-beta2
|
||
|
|
# Enables the web UI and tells Traefik to listen to docker
|
||
|
|
command:
|
||
|
|
#- --log.level=DEBUG
|
||
|
|
- --api.insecure=true
|
||
|
|
- --providers.docker=true
|
||
|
|
- --providers.docker.exposedbydefault=false
|
||
|
|
- --providers.docker.network=matrix-internal
|
||
|
|
- --entrypoints.client.address=:4443
|
||
|
|
- --entrypoints.federation.address=:8443
|
||
|
|
- --entrypoints.web.address=:80
|
||
|
|
- --certificatesresolvers.tailscaleResolver.tailscale=true
|
||
|
|
ports:
|
||
|
|
- "80:80"
|
||
|
|
- "4443:4443"
|
||
|
|
- "443:443/udp"
|
||
|
|
- "8443:8443"
|
||
|
|
volumes:
|
||
|
|
# make the Tailscale socket available to Traefik
|
||
|
|
- ./data/tailscale/run/tailscaled.sock:/var/run/tailscale/tailscaled.sock
|
||
|
|
# Add Docker as a mounted volume, so that Traefik can read the labels of other services
|
||
|
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||
|
|
networks:
|
||
|
|
web:
|
||
|
|
matrix-internal:
|
||
|
|
ipv4_address: 172.25.0.10
|
||
|
|
|
||
|
|
postgres:
|
||
|
|
hostname: postgres
|
||
|
|
image: postgres:15
|
||
|
|
restart: always
|
||
|
|
volumes:
|
||
|
|
- ./config/postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
|
||
|
|
- ./data/postgres:/var/lib/postgresql/data
|
||
|
|
environment:
|
||
|
|
POSTGRES_PASSWORD: {{ YOUR POSTGRES PASS. SAME AS IN dendrite.yaml }}
|
||
|
|
POSTGRES_USER: dendrite
|
||
|
|
healthcheck:
|
||
|
|
test: ["CMD-SHELL", "pg_isready -U dendrite"]
|
||
|
|
interval: 5s
|
||
|
|
timeout: 5s
|
||
|
|
retries: 5
|
||
|
|
networks:
|
||
|
|
- matrix-internal
|
||
|
|
|
||
|
|
dendrite:
|
||
|
|
depends_on:
|
||
|
|
- postgres
|
||
|
|
hostname: dendrite
|
||
|
|
image: matrixdotorg/dendrite-monolith:latest
|
||
|
|
command: [
|
||
|
|
"--tls-cert=server.crt",
|
||
|
|
"--tls-key=server.key"
|
||
|
|
]
|
||
|
|
ports:
|
||
|
|
- 8008:8008
|
||
|
|
- 8448:8448
|
||
|
|
volumes:
|
||
|
|
- ./config/dendrite:/etc/dendrite
|
||
|
|
- ./data/dendrite/media:/var/dendrite/media
|
||
|
|
depends_on:
|
||
|
|
- postgres
|
||
|
|
networks:
|
||
|
|
- matrix-internal
|
||
|
|
restart: unless-stopped
|
||
|
|
labels:
|
||
|
|
- traefik.enable=true
|
||
|
|
- traefik.http.routers.web.rule=Host(`matrix.{{tailnet-id}}.ts.net`)
|
||
|
|
- traefik.http.routers.web.tls.certresolver=tailscaleResolver
|
||
|
|
- traefik.http.routers.web.tls.domains[0].main=matrix.{{tailnet-id}}.ts.net
|
||
|
|
- traefik.http.routers.web.entrypoints=client
|
||
|
|
- traefik.http.routers.federation.rule=Host(`matrix.{{tailnet-id}}.ts.net`)
|
||
|
|
- traefik.http.routers.federation.tls.certresolver=tailscaleResolver
|
||
|
|
- traefik.http.routers.federation.tls.domains[0].main=matrix.{{tailnet-id}}.ts.net
|
||
|
|
- traefik.http.routers.federation.entrypoints=federation
|
||
|
|
```
|
||
|
|
|
||
|
|
The dendrite config should be configured as required based on the
|
||
|
|
[Dendrite docs](https://github.com/matrix-org/dendrite/tree/main/build/docker). However,
|
||
|
|
the Tailscale environment config file was a little harder to figure out. In this one I
|
||
|
|
ended with something along the following lines:
|
||
|
|
|
||
|
|
```
|
||
|
|
TS_HOSTNAME='matrix'
|
||
|
|
TS_AUTH_KEY={{ Authkey from the Tailscale admin console }}
|
||
|
|
TS_STATE_DIR='/var/lib/tailscale'
|
||
|
|
TS_USERSPACE=false
|
||
|
|
TS_SOCKET='/var/run/tailscale/tailscaled.sock'
|
||
|
|
TS_DEST_IP='172.25.0.10'
|
||
|
|
```
|
||
|
|
|
||
|
|
## Word of Caution
|
||
|
|
|
||
|
|
The Tailscale container is highly privileged, so make sure to put on some real monitoring.
|