thoughts/data/traefik-centralized-logging.md

Key Takeaways

• Organisations should focus on defendable infrastructure to enable efficient security monitoring. This requires using reverse proxies, software-defined networking, a well-defined and flexible architecture.

• We can add the LimaCharlie Adapter in file-mode to stream JSON logs to limacharlie.io for centralized and actionable logging (free to test).

• Infrastructure as Config/Code enables templating so that the setup can be re-used across applications and saves a lot of engineering time.

Background

Yesterday I published a post about how I'm moving from OpenBSD to Docker for my Matrix Dendrite setup. At the same time I reshaped the infrastructure delivering Dendrite to take advantage of the reverse proxying capabilities in Traefik.

One of the major advantages of moving to Docker is that it facilitates security monitoring in a better way than OpenBSD. This mostly has to do with vendor support and the fact that OpenBSD does things differently.

Within security monitoring, centralized logging with LimaCharlie Adapters is a good option, which is what I wanted to share a simple example of here.

Please note that this is simplistic in design, since the rig I'm building for doesn't have much activity. If I were building the monitoring setup for e.g. Kubernetes I would need to take into consideration that there will be multiple containers moving around, distributed file storage etc.

I'm separating containers by duty. It will probably bring a little extra overhead over using centralized storage or an embedded adapter with stdout-streaming per container.

Let's keep the rest of this article short and sweet.

The Configuration

First we need to add the following accessLog labels to traefik in our Docker Compose:

[...]
	reverse-proxy:
		[...]
		command:
			[...]
			- --accessLog.filePath=/var/log/traefik/access.log
			- --accessLog.format=json
    volumes:
      [...]
      - ./data/traefik:/var/log/traefik
    [...]
  [...]

This will write logs to ./data/traefik outside Docker. We'll share the same directory with the LimaCharlie Adapter. For this purpose we'll configure the adapter to run in file-mode. In a more demanding configuration I would consider using stdin-mode.

There's an example in the LimaCharlie docs of how to use the adapter in Docker with syslog. We can also run the following to get the file configuration options:

$ docker run --rm -it -p 4444:4444 refractionpoint/lc-adapter:latest

For file
----------------------------------
client_options.identity.oid
client_options.identity.installation_key
client_options.hostname
client_options.platform
client_options.architecture
client_options.mapping.parsing_re
client_options.mapping.sensor_key_path
client_options.mapping.sensor_hostname_path
client_options.mapping.event_type_path
client_options.mapping.event_time_path
client_options.mapping.rename_only
client_options.mapping.mappings
client_options.mappings
client_options.indexing
client_options.is_compressed
client_options.sensor_seed_key
client_options.dest_url
write_timeout_sec
file_path
no_follow

Having a look at data/traefik/access.log, we find that Traefik has the following pattern:

{
  "ClientAddr": "[IP]:[PORT]",
  "ClientHost": "[IP]",
  "ClientPort": "[PORT]",
  "ClientUsername": "-",
  "DownstreamContentSize": 105,
  "DownstreamStatus": 200,
  "Duration": 30012385827,
  "OriginContentSize": 105,
  "OriginDuration": 30012309833,
  "OriginStatus": 200,
  "Overhead": 75994,
  "RequestAddr": "matrix.{{tailnet-id}}.ts.net",
  "RequestContentSize": 0,
  "RequestCount": 5,
  "RequestHost": "matrix.{{tailnet-id}}.ts.net",
  "RequestMethod": "GET",
  "RequestPath": "/_matrix/client/r0/sync?[...]",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/2.0",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "RouterName": "web@docker",
  "ServiceAddr": "[IP]:[PORT]",
  "ServiceName": "dendrite-docker-matrix@docker",
  "ServiceURL": "http://[IP]:[PORT]",
  "StartLocal": "2023-04-25T08:06:12.121010341Z",
  "StartUTC": "2023-04-25T08:06:12.121010341Z",
  "TLSCipher": "TLS_AES_128_GCM_SHA256",
  "TLSVersion": "1.3",
  "entryPointName": "client",
  "level": "info",
  "msg": "",
  "time": "2023-04-25T08:06:42Z"
}```

This won't matter for getting basic logging up and running as
we can use the adapter in JSON mode as follows:

```sh
docker run --rm \
-v ./data/traefik/access.log:/logs/traefik/access.log \
refractionpoint/lc-adapter:latest file \
file_path=/logs/traefik/access.log \
client_options.identity.installation_key={{ADAPTER INSTALL. KEY}} \
client_options.identity.oid={{limacharlie.io ORG ID}} \
client_options.hostname=traefik-ts-dendrite \
client_options.sensor_seed_key=traefik-ts-dendrite \
client_options.platform=json

And voilà, our adapter is streaming to our organisation (synonym to "tenant") at limacharlie.io.

To make it permanent, we convert the docker command to docker-compose and end up with:

version: '3.3'
services:
  traefik-lc-adapter:
    image: refractionpoint/lc-adapter:latest
    command: file file_path=/logs/traefik/access.log client_options.identity.installation_key=$LC_INSTALLATION_KEY client_options.identity.oid=$LC_ORG_ID client_options.hostname=$LC_HOSTNAME client_options.sensor_seed_key=$LC_HOSTNAME client_options.platform=json
    restart: unless-stopped
    environment:
      TZ: UTC
    env_file:
      - ./config/limacharlie/.limacharlie_env
    volumes:
      - './data/traefik/access.log:/logs/traefik/access.log'

Add keys to environment file in .env:

LC_INSTALLATION_KEY={{ADAPTER INSTALLATION KEY}}
LC_ORG_ID={{limacharlie.io ORG ID}}
LC_HOSTNAME=traefik-ts-dendrite

Easy as that.