Collaborative Research Into Threats (CRITs)July 3, 2014
Recently, MITRE announced the open sourcing of CRITs: Collaborative Research Into Threats. Releasing CRITs in open source is a way to give back to the security community because it is important to have freely available tools in an industry where every organization, big or small, needs to do all they can to protect themselves against threats. Being able to communicate and share threats between organizations, government, and the public, opens up a more collaborative effort towards intelligence-based active threat defense.
It Began with Malware Management
I wanted to give some background on how CRITs came to be. I've been at MITRE since the beginning as one of the lead developers and am now the project manager. Toward the end of 2010, I was asked to tackle a code-rewrite of a small project. The project was a malware database written in Python using Django and MySQL. It was growing long in the tooth and was in dire need of a more scalable design. Around this same time I was becoming frustrated with what I will call the "Data Rediscovery Problem."
Most of us while we work tend to put data where it is most convenient at the time. If we’re working on a malicious binary, we'll drop it onto a host that has the tools necessary to handle it. If we're writing quick scripts, we'll develop them on whichever box winds up being the first terminal we find available. This is all to save time and effort, to get the job done quickly and easily.
Even worse, what do we name those scripts? Usually things like foo.py, foo2.py, etc. What happens in a few weeks when we find ourselves in need of that data or those scripts again? We have to remember where we put them. A lot of times we'll eventually rediscover them. Other times we'll forget and wind up re-generating the data or re-writing the script. Even worse is when someone else is duplicating the work we did because our stuff wasn't readily available. This is a huge waste of time and resources. It also makes it frustratingly difficult for new hires and teammates to find data they need to do their job.
While I was rewriting the malware database, I thought it would be neat if it could be extended to solve the data rediscovery problem. I wanted to provide a way for people to quickly upload content and make it as efficient as possible to save time and be a reference point for all of their data. That would solve half of the rediscovery issue, which is a good start.
Adding Incident Response
The first thing I did was extend past storing samples (malicious binaries) and allow for the storage of PCAPs (packet captures). Network traffic is such a huge help when trying to dig towards "ground truth." I always had problems finding the network traffic I needed, so adding PCAP support was an easy decision. I was approached by an analyst and asked if I could extend the project to support emails. It seemed like a reasonable request, so I added email support. Tracking domains and IPs seemed like the next logical step given their presence in binaries, PCAPs, and emails, so these were added.
Adding Cyber Intelligence
Then I was asked about supporting IoCs (indicators of compromise). This was sort of a game-changer. IoCs are amazingly important for an organization looking to defend its network. They need to be handled carefully. Even though they might be simple in concept, the response to them can sometimes be extremely complex. Our development team spent some time thinking about what handling IoCs in the project meant, but eventually settled in on a design and implemented it.
At this time it became obvious that we needed a name for this project. It started out as a critical tool for our malware analysts, but was now being used heavily by our intel analysts, as well as our incident response team. They all began collaborating in a way that they were never able to before. Sharing data was as simple as uploading it to the same place as everyone else. This made analysis and response a much more efficient and effective process. It moved the groups to a more intelligence-based active threat defense. With all of us researching these threats, we quickly settled on the name Collaborative Research Into Threats.
Over the last three years, the project has grown to support campaigns, certificates, events, raw data, and targets (with more in the works!). Through word-of-mouth, hundreds of organizations asked to get their hands on CRITs so they could use it to improve their security posture. Many developers have jumped on board and contributed hundreds of features for managing and working with threat data. One of the biggest additions was the services framework. This allowed people to develop extensions to the project and enhance functionality without the need to modify core code.
To date we have 25 services that have been contributed back to the project, giving intel and malware analysts a way to work with threat data without having to download it and a way to integrate CRITs with third-party and homegrown software, as well as visualize and discover threat data in ways that were previously unavailable. This solved the second half of the data rediscovery problem. Now someone can write a tool once, create a service out of it, and make it available for all of their users and the entire community if they wish. The ability to work with threat data without always needing to download it to another machine has proven time and time again to be invaluable.
Another major enhancement to the project was the inclusion of basic support for structured data exchange formats (CybOX, STIX, and TAXII). Being able to collaborate with groups internally is a huge win. But being able to collaborate with other trusted organizations in real time changes the security industry completely. Where organizations used to keep this information to themselves, they are now able to use these exchange formats to communicate threat data and position other organizations to protect their network well in advance of the threat making its way to them. By incorporating these formats into CRITs we've given all of the users of the platform the capability to share threat data quickly when time matters the most.
Like any other tool in the industry, CRITs isn't the end-all and be-all solution to defending your network. It is simply an enabler. It gives analysts a better way of storing, enriching, and discovering threat data necessary to protect their organization. It doesn't replace analysts. If anything it speaks to the necessity for all organizations to have smart, capable analysts who can leverage a platform like CRITs to put good data in, and get good data out. Too long has our industry worked in isolation to solve the same problem. It's time we all work together and change our mindsets to a more open, intelligence-based active threat defense.