thoughts/data/graph-experiment.md

I currently maintain this threat database, and up until now I've generated the graph data for d3 using queries, and a lot of logic, in a MySQL-database. That is going to change pretty soon. You might also remember when we did Social Network Analysis and Object Attribution with Maltego 3 [1].

In my seeking for understanding the Apache Hadoop ecosystem I all of a sudden got a brutal meeting with Java (Eclipse huh..). I also discovered that there are a world of libraries and applications previously unknown to me. One of them is the über-awesome Neo4j, which is a graph database originally built for Java - but guess what: It's got a REST API as well. As usual you don't have to write the Python code yourself, someone already wrote it for you. Note that it only does Python 2 for now [2,3].

The coolest thing about Neo4j is Cypher [5]: Cypher is a "graph query language" as they put it themselves. With Cypher you can express what you look for in an entirely other way than you would do in a relational database, it's actually easy.

And: You of course need the database running as well. If you use a Debian system like me your in luck since they have an experimental version out there [5].

Enough talk, here is a simple example of how you could go about it in regard to scripting the relations considering threat intelligence in order to connect groups to incidents. The goal would be to find peripherally connected groups.

from GraphConn.Connect import Graph
g = Graph()

# create groups
g.cGroup("ThreatA")
g.cGroup("ThreatB")
g.cGroup("ThreatC")

# create incidents
g.cIncident("IncA")
g.cIncident("IncB")
g.cIncident("IncC")

# relate groups in some way to each other through incidents
g.link("ThreatA","IncA")
g.link("ThreatA","IncB")
g.link("ThreatB","IncC")
g.link("ThreatC","IncA")
g.link("ThreatB","IncB")

# find all threats related to Threat A through incidents
print g.fRelated("ThreatA")

You might find this simple, but if you've ever tried to do it in SQL you know why you'll need it. Also, remember that this scales indefinite to other entity types as well.

Here's the class used to generate the graph, for reference (feel free to copy it, produce something cool and post it back in the comment field):

from neo4jrestclient import client
from neo4jrestclient.client import GraphDatabase
from neo4jrestclient.query import Q

class Graph:
    def __init__(self):
        self.gdb = GraphDatabase("http://localhost:7474/db/data/")
        self.nodes = []

    def cGroup(self,name):
        n = self.gdb.nodes.create(name=name, type='Group')
        self.nodes.append(n)

    def cIncident(self,name):
        n = self.gdb.nodes.create(name=name, type='Incident')
        self.nodes.append(n)

    def link(self,n1,n2):
        try:
            l = (Q("name", iexact=n1)); n1 = self.gdb.nodes.filter(l)[0];
            l = (Q("name", iexact=n2)); n2 = self.gdb.nodes.filter(l)[0];
            return n1.relationships.create("Executed", n2)
        except:
            return False

    def fRelated(self,query):
        l = (Q("name", iexact=query))
        n = self.gdb.nodes.filter(l)[0]
        r = n.traverse()
        for n2 in r:
            for e in n2.traverse():
                r.append(e)
        return list(r)

I really hope you enjoy this as much as me right now. The Facebook Graph Search for the rest of us.

[1] gopher://secdiary.com/0/post/sna-oa-maltego/index.txt [2] https://pypi.python.org/pypi/neo4jrestclient/ [3] https://neo4j-rest-client.readthedocs.org/en/latest/elements.html [4] http://www.neo4j.org/learn/cypher [5] http://debian.neo4j.org/