thoughts/data/osquery.md

In another post I wrote about how telemetry is a challenge [1] of a changing and more diverse and modern landscape. Recently I have reviewed some device inventory and endpoint detection tools that will add to the solution. In the future I will get back to my view on Mozilla InvestiGator (MIG) [2], but this post will focus on a telemetry collection tool that I have grown fond of: osquery [3].

osquery was originally developed by Facebook for the purpose of [4]:

Maintaining real-time insight into the current state of your infrastructure[...]

With osquery data is abstracted, in the operating system in which the agent runs, to a SQL-based interface. It contains a near-infinite amount of available data, which is perfect to a network defender. osquery can even parse native sqlite-databases, which there are lots of in macOS. It also works in a distributed mode like GRR and MiG. In practical terms this means that queries are distributed. On the other hand, events can be streamed as well when considering operational security.

Since 2014 osquery has been open sourced and now has a large community developing about every aspect of the tool. According to the briefs that's online several major institutions, including Facebook, now uses osquery in service networks.

osquery is cross-platform, and now supports: Linux, FreeBSD, Windows and macOS. That is also some of what separates it from its alternatives, like sysmon.

Posts about osquery that you should review before moving on:

• Doug Wilson's excellent presentation on FIRST 2018 (security-usage focused) [5]
• Managing osquery with Kolide (an osquery tls server) [6]
• Another post on applying osquery for security [7]
• Palantir on osquery [8]

So that was a couple of links to get you started. The next section shows you how to quickly get a lab environment up and running.

Setup and Configuration

Prerequisites

There's only two things that you need setup for the rest of this article if you are on macOS, which can both be easily installed using Homebrew [9]:

brew install go yarn

Also you need to configure your Go-path, which can basically be:

echo "export GOPATH=$HOME/go" >> ~/.bash_profile

Server Setup

Setup Docker image of Kolide Fleet [10]:

 mkdir -p $GOPATH/src/github.com/kolide
 cd $GOPATH/src/github.com/kolide
 git clone git@github.com:kolide/fleet.git
 cd fleet
 make deps && make generate && make
 docker-compose up

Populate the database:

./build/fleet prepare db

You are now ready to boot up the web UI and API server:

./build/fleet serve --auth_jwt_key=3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP

Get enrollment secret and certificate from the Kolide UI at https://localhost:8080 after doing the registration process.

Client Setup

Make the API-token (enrollment secret) persistent at the end-point:

export {enrollment-secret} > /etc/osquery/enrollment.secret

Define flags file in /private/var/osquery/osquery.flags. This one the client uses to apply the centralised tls logging method, which is the API Kolide has implemented. It is also certificate pinned, so all is good.

 --enroll_secret_path=/etc/osquery/enrollment.secret
 --tls_server_certs=/etc/osquery/kolide.crt
 --tls_hostname=localhost:8080
 --host_identifier=uuid
 --enroll_tls_endpoint=/api/v1/osquery/enroll
 --config_plugin=tls
 --config_tls_endpoint=/api/v1/osquery/config
 --config_tls_refresh=10
 --disable_distributed=false
 --distributed_plugin=tls
 --distributed_interval=10
 --distributed_tls_max_attempts=3
 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
 --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
 --logger_plugin=tls
 --logger_tls_endpoint=/api/v1/osquery/log
 --logger_tls_period=10

You can start the osquery daemon on the client by using the following command. At this point you should start thinking about packaging, which is detailed in the osquery docs [11].

/usr/local/bin/osqueryd --disable_events=false \
                        --flagfile=/private/var/osquery/osquery.flags

osquery also has an interactive mode if you would like to test the local instance, based on a local configuration file:

sudo osqueryi --disable_events=false \
              --config_path=/etc/osquery/osquery.conf \
              --config_path=/etc/osquery/osquery.conf

To make the client persistent on macOS, use the following documentation from osquery [12].

Managing the Kolide Configuration

For this part I found what worked best was using the Kolide CLI client [13]:

./build/fleetctl config set --address https://localhost:8080
./build/fleetctl login
./build/fleetctl apply -f ./options.yaml

The options.yaml I used for testing was the following. This setup also involves setting up the osquery File Integrity Monitoring (FIM) [14], which I wasn't able to get working by the patching curl command [15] in the docs. The config monitors changes in files under /etc and a test directory at /var/tmp/filetest.

apiVersion: v1
kind: options
spec:
  config:
    decorators:
      load:
      - SELECT uuid AS host_uuid FROM system_info;
      - SELECT hostname AS hostname FROM system_info;
    file_paths:
      etc:
        - /etc/%%
      test:
        - /var/tmp/filetest/%%
    options:
      disable_distributed: false
      distributed_interval: 10
      distributed_plugin: tls
      distributed_tls_max_attempts: 3
      distributed_tls_read_endpoint: /api/v1/osquery/distributed/read
      distributed_tls_write_endpoint: /api/v1/osquery/distributed/write
      logger_plugin: tls
      logger_tls_endpoint: /api/v1/osquery/log
      logger_tls_period: 10
      pack_delimiter: /
  overrides: {}

Next Steps

Through this article we've reviewed some of the basic capabilities of osquery and also had a compact view on a lab-setup demonstrating centralised logging, to Kolide, using the tls API of osquery.

A couple of things that I would have liked to see was support for OpenBSD [16], Android and Ios [17].

The local setup obviously does not scale beyond your own computer. I briefly toyed with the idea that this would be a perfect fit for ingesting into a Hadoop environment, and not surprising there's a nice starting point over at the Hortonworks forums [18].

There's a lot of open source information on osquery. I also found the Uptycs blog useful [19].

[1] https://secdiary.com/2018-02-25-telemetry.html
[2] https://mig.mozilla.org
[3] https://osquery.io
[4] https://code.fb.com/security/introducing-osquery/
[5] https://www.first.org/resources/papers/conf2018/Wilson-Doug_FIRST_20180629.pdf
[6] https://blog.kolide.com/managing-osquery-with-kolide-launcher-and-fleet-b33b4536acb4
[7] https://medium.com/@clong/osquery-for-security-part-2-2e03de4d3721
[8] https://github.com/palantir/osquery-configuration
[9] https://brew.sh
[10] https://blog.kolide.com/managing-osquery-with-kolide-launcher-and-fleet-b33b4536acb4 [11] https://osquery.readthedocs.io/en/2.1.1/installation/custom-packages/ [12] https://osquery.readthedocs.io/en/stable/installation/install-osx/
[13] https://github.com/kolide/fleet/blob/master/docs/cli/setup-guide.md
[14] https://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/
[15] https://github.com/kolide/fleet/tree/master/docs/api#file-integrity-monitoring
[16] https://github.com/facebook/osquery/issues/4703
[17] https://github.com/facebook/osquery/issues/2815
[18] https://community.hortonworks.com/articles/79842/ingesting-osquery-into-apache-phoenix-using-apache.html
[19] https://www.uptycs.com/blog