thoughts/data/remote-forensics.md

Like everything else in information security, forensics is constantly evolving. One matter of special interest for practitioners is doing forensics on remote computers, not that it's entirely new.

The use-case is self-explanatory to those working in the field, but for the beginners I'll give a brief introduction.

When you get a case on your desk and it lights up as something interesting, what do you do? Probably your first step is searching for known malicious indicators in network logs. Finding something interesting on some of the clients, let's say ten in this case, you decide to put some more effort into explaining the nature of the activity. None of the clients is nearby, multiple of them are even on locations with 1Mbps upload speeds.

The next phase would probably be a search in open sources, perhaps turning out in support of something fishy going on. Now you'd like to examine some of the client logs for known hashes and strings you found, and the traditional way to go is acquiring disk and memory images physically. Or is it? That would have easily taken weeks for ten clients. In this case you are lucky and you have a tool for performing remote forensics at hand. The tool was a major roll-out for your organization after a larger breach.

What's new in remote forensics is that the tools begin to get more mature, and by that I would like to introduce two products of which I find most relevant to the purpose:

• Google Rapid Response (GRR) [1]
• Mandiant for Incident Response (MIR) [2]

Actually I haven't put the latter option to the test (MIR supports OpenIOC which is an advantage) - but I have chosen to take GRR for a spin for some time now. There are also other tools which may be of interest to you such as Sourcefire FireAmp which I've heard performs well for end-point-protection. I've chosen to leave that out this presentation since this is about a different concept. Surprisingly the following will use GRR as a basis.

For this post there are two prerequisites for you to follow in which I highly recommend to get the feel with GRR:

• Setup a GRR server [3]. In this post I've used the current beta 3.0-2, running all services on the same machine, including the web server and client roll-in interface. There is one install script for the beloved Ubuntu here, but I couldn't get it easily working on other systems. One exception is Debian which only needed minor changes. If you have difficulties with the latter, please give me a heads-up.
• Sacrifice one client (it won't brick a production system as far as I can tell either though) to be monitored. You will find binaries after packing the clients in the GRR Server setup. See the screenshot below for details. The client will automatically report in to the server.

You can find the binaries by browsing from the home screen in the GRR web GUI. Download and install the one of choice.

A word warning before you read the rest of this post: The GRR website is was a little messy and not entirely intuitive. I found, after a lot of searching, that the best way to go about it is reading the code usage examples in the web GUI, especially when it comes to what Google named flows. Flows are little plugins in GRR that may for instance help you task GRR to fetch a file on a specific path.

Notice the call spec. This can be transferred directly to the iPython console. Before I started off I watched a couple of presentations that Google have delivered at LISA. I think you should too if you'd like to see where GRR is going and why it came to be. The one here gives a thorough introduction on how Google makes sure they are able to respond to breaches in their infrastructure [4].

I would also like to recommend an presentation by Greg Castle on BlackHat for reference [5]. For usage and examples Marley Jaffe at Champlain College have put up a great paper. Have a look at the exercises at the end of it.

What is good with GRR is that it supports the most relevant platforms: Linux, Windows and OS X. This is also fully supported platforms at Google, so expect development to have a practical and long-term perspective.

While GRR is relevant, it is also fully open source, and extensible. It's written in Python with all the niceness that comes with it. GRR have direct memory access by custom built drivers. You will find support for Volatility in there. Well they forked it into a new project named Rekall which is more suited for scale. Anyways it provides support for plugins such as Yara.

If you are like me and got introduced to forensics through academia, you will like that GRR builds on Sleuthkit through pytsk for disk forensics (actually you may choose what layer you'd like to stay on). When you've retrieved an item, I just love that it gets placed in a virtual file system in GRR with complete versioning.

The virtual filesystem where all the stuff you've retrieved or queried the client about is stored with versioning for you pleasure. In addition to having a way-to-go console application GRR provides a good web GUI which provides an intuitive way of browsing about everything you can do in the console. I think the console is where Google would like you to live though.

An so I ended up on the grr_console which is a purpose-build iPython shell, writing scripts for doing what I needed it to do. Remember that call spec that I mentioned initially, here is where it gets into play. Below you see an example using the GetFile call spec (notice that the pathspec in the flow statement says OS, this might as well have been REGISTRY or TSK):

token = access_control.ACLToken(username="someone", reason="Why")

flows=[]
path="/home/someone/nohup.out"

for client in SearchClients('host:Webserver'):
  id=client[0].client_id
  o=flow.GRRFlow.StartFlow(client_id=str(id),
  flow_name="GetFile", pathspec=rdfvalue.PathSpec(path=path, 
                       pathtype=rdfvalue.PathSpec.PathType.OS))
  flows.append(o)

files=[]
while len(flows)>0:
  for o in flows:
    f=aff4.FACTORY.Open(o)
    r = f.GetRunner()
    if not r.IsRunning():
      fd=aff4.FACTORY.Open(str(id)+"/fs/os%s"%path, token=token)
      files.append(str(fd.Read(10000)))
      flows.remove(o)

If interested in Mandiant IR (MIR) and its concept, I'd like to recommend another Youtube video by Douglas Wilson, which is quite awesome as well [7].

Update 2020: Today I wouldn't recommend MIR/FireEye HX, but rather something like LimaCharlie [8] due to the lack of hunting capabilities in the HX platform.

[1] https://github.com/google/grr

[2] http://www.fireeye.com/products-and-solutions/endpoint-forensics.html

[3] https://grr-doc.readthedocs.io/en/latest/installing-grr-server/index.html

[4] https://2459d6dc103cb5933875-c0245c5c937c5dedcca3f1764ecc9b2f.ssl.cf2.rackcdn.com/lisa13/castle.mp4

[5] GRR: Find All The Badness - https://docs.google.com/file/d/0B1wsLqFoT7i2Z2pxM0wycS1lcjg/edit?pli=1

[6] Jaffe, Marley. GRR Capstone Final Paper

[7] NoVA Hackers Doug Wilson - Lessons Learned from using OpenIOC: https://www.youtube.com/watch?v=L-J5DDG_SQ8

[8] https://www.limacharlie.io/