thoughts/data/traefik-centralized-logging.md

## Key Takeaways

* Organisations should focus on defendable infrastructure
  to enable efficient security monitoring. This requires 
  using reverse proxies, software-defined networking, a 
  well-defined and flexible architecture.

* We can add the LimaCharlie Adapter in file-mode
  to stream JSON logs to limacharlie.io for
  centralized and actionable logging (free to test).

* Infrastructure as Config/Code enables templating so
  that the setup can be re-used across applications and
  saves a lot of engineering time.


## Background

Yesterday I published a post about how I'm moving from
OpenBSD to Docker for my Matrix Dendrite setup. At the same
time I reshaped the infrastructure delivering Dendrite
to take advantage of the reverse proxying capabilities
in Traefik.

One of the major advantages of moving to Docker is that
it facilitates security monitoring in a better way than
OpenBSD. This mostly has to do with vendor support and
the fact that OpenBSD does things differently.

Within security monitoring, centralized logging with 
[LimaCharlie Adapters](https://doc.limacharlie.io/docs/documentation/73a613e8e43ed-lima-charlie-adapter) 
is a good option, which is what I wanted to share a
simple example of here.

Please note that this is simplistic in design, since the
rig I'm building for doesn't have much activity. If
I were building the monitoring setup for e.g. Kubernetes
I would need to take into consideration that there will
be multiple containers moving around, distributed file
storage etc.

I'm separating containers by duty. It will probably bring 
a little extra overhead over using centralized storage or 
an embedded adapter with stdout-streaming per container.

Let's keep the rest of this article short and sweet.

## The Configuration

First we need to add the following [`accessLog` labels to traefik](https://doc.traefik.io/traefik/observability/access-logs/) 
in our Docker Compose:

```
[...]
	reverse-proxy:
		[...]
		command:
			[...]
			- --accessLog.filePath=/var/log/traefik/access.log
			- --accessLog.format=json
    volumes:
      [...]
      - ./data/traefik:/var/log/traefik
    [...]
  [...]
```

This will write logs to `./data/traefik` outside Docker. We'll
share the same directory with the LimaCharlie Adapter. For this
purpose we'll configure the adapter to run in `file-mode`. In a
more demanding configuration I would consider using `stdin-mode`.

There's an example in the LimaCharlie docs of how to use the
[adapter in Docker with syslog](https://doc.limacharlie.io/docs/documentation/73a613e8e43ed-lima-charlie-adapter#syslog). We can also run the
following to get the file configuration options:

```sh
$ docker run --rm -it -p 4444:4444 refractionpoint/lc-adapter:latest

For file
----------------------------------
client_options.identity.oid
client_options.identity.installation_key
client_options.hostname
client_options.platform
client_options.architecture
client_options.mapping.parsing_re
client_options.mapping.sensor_key_path
client_options.mapping.sensor_hostname_path
client_options.mapping.event_type_path
client_options.mapping.event_time_path
client_options.mapping.rename_only
client_options.mapping.mappings
client_options.mappings
client_options.indexing
client_options.is_compressed
client_options.sensor_seed_key
client_options.dest_url
write_timeout_sec
file_path
no_follow
```

Having a look at `data/traefik/access.log`, we find that Traefik
has the following pattern:

```json
{
  "ClientAddr": "[IP]:[PORT]",
  "ClientHost": "[IP]",
  "ClientPort": "[PORT]",
  "ClientUsername": "-",
  "DownstreamContentSize": 105,
  "DownstreamStatus": 200,
  "Duration": 30012385827,
  "OriginContentSize": 105,
  "OriginDuration": 30012309833,
  "OriginStatus": 200,
  "Overhead": 75994,
  "RequestAddr": "matrix.{{tailnet-id}}.ts.net",
  "RequestContentSize": 0,
  "RequestCount": 5,
  "RequestHost": "matrix.{{tailnet-id}}.ts.net",
  "RequestMethod": "GET",
  "RequestPath": "/_matrix/client/r0/sync?[...]",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/2.0",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "RouterName": "web@docker",
  "ServiceAddr": "[IP]:[PORT]",
  "ServiceName": "dendrite-docker-matrix@docker",
  "ServiceURL": "http://[IP]:[PORT]",
  "StartLocal": "2023-04-25T08:06:12.121010341Z",
  "StartUTC": "2023-04-25T08:06:12.121010341Z",
  "TLSCipher": "TLS_AES_128_GCM_SHA256",
  "TLSVersion": "1.3",
  "entryPointName": "client",
  "level": "info",
  "msg": "",
  "time": "2023-04-25T08:06:42Z"
}```

This won't matter for getting basic logging up and running as
we can use the adapter in JSON mode as follows:

```sh
docker run --rm \
-v ./data/traefik/access.log:/logs/traefik/access.log \
refractionpoint/lc-adapter:latest file \
file_path=/logs/traefik/access.log \
client_options.identity.installation_key={{ADAPTER INSTALL. KEY}} \
client_options.identity.oid={{limacharlie.io ORG ID}} \
client_options.hostname=traefik-ts-dendrite \
client_options.sensor_seed_key=traefik-ts-dendrite \
client_options.platform=json
```

And voilà, our adapter is streaming to our organisation 
(synonym to "tenant") at limacharlie.io.

To make it permanent, we convert the `docker` command to
`docker-compose` and end up with:

```docker-compose
version: '3.3'
services:
  traefik-lc-adapter:
    image: refractionpoint/lc-adapter:latest
    command: 
    - file
    - file_path=/logs/traefik/access.log
    - client_options.identity.installation_key=$LC_INSTALLATION_KEY
    - client_options.identity.oid=$LC_ORG_ID 
    - client_options.hostname=$LC_HOSTNAME 
    - client_options.sensor_seed_key=$LC_HOSTNAME 
    - client_options.platform=json
    restart: unless-stopped
    environment:
      TZ: UTC
    env_file:
      - ./config/limacharlie/.limacharlie_env
    volumes:
      - './data/traefik/access.log:/logs/traefik/access.log'
```

Add keys to environment file in `.env`:

```sh
LC_INSTALLATION_KEY={{ADAPTER INSTALLATION KEY}}
LC_ORG_ID={{limacharlie.io ORG ID}}
LC_HOSTNAME=traefik-ts-dendrite
```

Easy as that.