thoughts/data/osquery.md

In another post I wrote about how telemetry is a challenge [1] of
a changing and more diverse and modern landscape. Recently I have
reviewed some device inventory and endpoint detection tools that
will add to the solution. In the future I will get back to my view
on Mozilla InvestiGator (MIG) [2], but this post will focus on a
telemetry collection tool that I have grown fond of: osquery [3].

osquery was originally developed by Facebook for the purpose of
[4]:

> Maintaining real-time insight into the current state of your infrastructure[...]

With osquery data is abstracted, in the operating system in which
the agent runs, to a SQL-based interface. It contains a
near-infinite amount of available data, which is perfect to a
network defender. osquery can even parse native sqlite-databases,
which there are lots of in macOS. It also works in a distributed
mode like GRR and MiG. In practical terms this means that queries
are distributed. On the other hand, events can be streamed as well
when considering operational security.

![Example of the hardware_events table when plugging in and then detaching a Yubikey](/static/img/data/osquery_hardware_events.png)

Since 2014 osquery has been open sourced and now has a large
community developing about every aspect of the tool. According to
the briefs that's online several major institutions, including
Facebook, now uses osquery in service networks.

osquery is cross-platform, and now supports: Linux, FreeBSD,
Windows and macOS. That is also some of what separates it from its
alternatives, like sysmon.

Posts about osquery that you should review before moving on:

* Doug Wilson's excellent presentation on FIRST 2018
  (security-usage focused) [5]
* Managing osquery with Kolide (an osquery tls server) [6]
* Another post on applying osquery for security [7]
* Palantir on osquery [8]

So that was a couple of links to get you started. The next section 
shows you how to quickly get a lab environment up and running.

## Setup and Configuration

### Prerequisites

There's only two things that you need setup for the rest of this
article if you are on macOS, which can both be easily installed
using Homebrew [9]:

    brew install go yarn

Also you need to configure your Go-path, which can basically be:

    echo "export GOPATH=$HOME/go" >> ~/.bash_profile

### Server Setup

Setup Docker image of Kolide Fleet [10]:

     mkdir -p $GOPATH/src/github.com/kolide
     cd $GOPATH/src/github.com/kolide
     git clone git@github.com:kolide/fleet.git
     cd fleet
     make deps && make generate && make
     docker-compose up

Populate the database:

    ./build/fleet prepare db
    
You are now ready to boot up the web UI and API server:

    ./build/fleet serve --auth_jwt_key=3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP

Get enrollment secret and certificate from the Kolide UI at
``https://localhost:8080`` after doing the registration process.

![Kolide enrollment](/static/img/data/kolide-enrollment.png)

### Client Setup

Make the API-token (enrollment secret) persistent at the
end-point:

    export {enrollment-secret} > /etc/osquery/enrollment.secret

Define flags file in ``/private/var/osquery/osquery.flags``. This
one the client uses to apply the centralised tls logging method,
which is the API Kolide has implemented. It is also certificate
pinned, so all is good.

     --enroll_secret_path=/etc/osquery/enrollment.secret
     --tls_server_certs=/etc/osquery/kolide.crt
     --tls_hostname=localhost:8080
     --host_identifier=uuid
     --enroll_tls_endpoint=/api/v1/osquery/enroll
     --config_plugin=tls
     --config_tls_endpoint=/api/v1/osquery/config
     --config_tls_refresh=10
     --disable_distributed=false
     --distributed_plugin=tls
     --distributed_interval=10
     --distributed_tls_max_attempts=3
     --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
     --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
     --logger_plugin=tls
     --logger_tls_endpoint=/api/v1/osquery/log
     --logger_tls_period=10

You can start the osquery daemon on the client by using the
following command. At this point you should start thinking about
packaging, which is detailed in the osquery docs [11].

    /usr/local/bin/osqueryd --disable_events=false \
                            --flagfile=/private/var/osquery/osquery.flags

osquery also has an interactive mode if you would like to test the
local instance, based on a local configuration file:

    sudo osqueryi --disable_events=false \
                  --config_path=/etc/osquery/osquery.conf \
                  --config_path=/etc/osquery/osquery.conf

To make the client persistent on macOS, use the following
documentation from osquery [12].

### Managing the Kolide Configuration

For this part I found what worked best was using the Kolide CLI
client [13]:

    ./build/fleetctl config set --address https://localhost:8080
    ./build/fleetctl login
    ./build/fleetctl apply -f ./options.yaml

The ``options.yaml`` I used for testing was the following. This
setup also involves setting up the osquery File Integrity
Monitoring (FIM) [14], which I wasn't able to get working by the
patching curl command [15] in the docs. The config monitors
changes in files under ``/etc`` and a test directory at
``/var/tmp/filetest``.
 
    apiVersion: v1
    kind: options
    spec:
      config:
        decorators:
          load:
          - SELECT uuid AS host_uuid FROM system_info;
          - SELECT hostname AS hostname FROM system_info;
        file_paths:
          etc:
            - /etc/%%
          test:
            - /var/tmp/filetest/%%
        options:
          disable_distributed: false
          distributed_interval: 10
          distributed_plugin: tls
          distributed_tls_max_attempts: 3
          distributed_tls_read_endpoint: /api/v1/osquery/distributed/read
          distributed_tls_write_endpoint: /api/v1/osquery/distributed/write
          logger_plugin: tls
          logger_tls_endpoint: /api/v1/osquery/log
          logger_tls_period: 10
          pack_delimiter: /
      overrides: {}

## Next Steps

Through this article we've reviewed some of the basic capabilities
of osquery and also had a compact view on a lab-setup
demonstrating centralised logging, to Kolide, using the tls API of
osquery.

A couple of things that I would have liked to see was support for
OpenBSD [16], Android and Ios [17].

The local setup obviously does not scale beyond your own
computer. I briefly toyed with the idea that this would be a
perfect fit for ingesting into a Hadoop environment, and not
surprising there's a nice starting point over at the Hortonworks
forums [18].

There's a lot of open source information on osquery. I also found
the Uptycs blog useful [19].

[1] https://secdiary.com/2018-02-25-telemetry.html  
[2] https://mig.mozilla.org  
[3] https://osquery.io  
[4] https://code.fb.com/security/introducing-osquery/  
[5]
https://www.first.org/resources/papers/conf2018/Wilson-Doug_FIRST_20180629.pdf  
[6]
https://blog.kolide.com/managing-osquery-with-kolide-launcher-and-fleet-b33b4536acb4  
[7] https://medium.com/@clong/osquery-for-security-part-2-2e03de4d3721  
[8] https://github.com/palantir/osquery-configuration  
[9] https://brew.sh  
[10]
https://blog.kolide.com/managing-osquery-with-kolide-launcher-and-fleet-b33b4536acb4
[11] https://osquery.readthedocs.io/en/2.1.1/installation/custom-packages/
[12] https://osquery.readthedocs.io/en/stable/installation/install-osx/  
[13]
https://github.com/kolide/fleet/blob/master/docs/cli/setup-guide.md  
[14]
https://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/  
[15]
https://github.com/kolide/fleet/tree/master/docs/api#file-integrity-monitoring  
[16] https://github.com/facebook/osquery/issues/4703  
[17] https://github.com/facebook/osquery/issues/2815  
[18]
https://community.hortonworks.com/articles/79842/ingesting-osquery-into-apache-phoenix-using-apache.html  
[19] https://www.uptycs.com/blog