Using WHOIS and Passive DNS for IntelligenceFebruary 5, 2015
In this post I talk about pyDat , a tool we released on open source that will let your cyber threat intelligence analysts perform WHOIS lookups without having to query external sites. This keeps intelligence in-house and also builds views into the data that an external site might not provide.
pyDat provides an interface to search and pivot on WHOIS data and find historical information about domains. The WHOIS data must be purchased from a vendor and loaded into the pyDat database (MongoDB). For passive DNS (PDNS), we use an API key from Farsight Security (https://www.dnsdb.info/) to query its external site. When considering sources for PDNS data, note that the quality, completeness, and timeliness differs by vendor. Regardless of which source you choose, PDNS databases are inherently incomplete, so care must be taken when analyzing results.
A simple example of using pyDat is to search on a spear-phishing link domain. Suppose multiple results are returned. A cyber intelligence analyst would then continue to pivot on each result to find other pieces of information that could help fill out a picture of the particular adversary's infrastructure.
Gathering intelligence about an adversary infrastructure could be methodically achieved just by using WHOIS information, making note of missing or incorrect information as you traverse and retrace each finding.
Any time you pivot on a domain, pyDat automatically performs a passive DNS lookup for that domain, including wildcards, and displays the results in the PDNS tab. Wildcards broaden the returned results, providing yet more data for analysis and correlation. If researching an IP address, pyDat will perform "inverse" passive DNS lookups, returning all domains that have resolved to the particular IP. This may reveal additional correlations to the domain with which you first started.
Cyber Threat Intelligence Value
Cyber threat intelligence analysts would use the results to investigate historic and active malicious behavior on their network, as well as to develop a map of and track an adversary's infrastructure.
Fleshing out the threat landscape faced by your organization, and industry, is a time consuming process, but the rewards are invaluable. Check out FireEye's report on "Operation Saffron Rose" for their findings of an adversary's infrastructure pieced together by their cyber analysts from similar type research on a spear-phishing email.
A Note about pyDat
We released pyDat to the Open Source community to provide a common tool for performing this type of cyber intelligence analysis. We separated the tooling from the data to make it easier for you to adapt your own data sources. To use pyDat, you must provide the data and API keys. If you would like more information on pyDat you can find it on our GitHub page.
pyDat is a spiritual descendent of WhoDat, which was originally written and openly shared by Chris Clark. As we began to look at WhoDat to serve our needs, we ended up rewriting it, so Chris decided to donate his WhoDat repository to us. (Thank you, Chris). You can find WhoDat on our GitHub page, but the original WhoDat portion is available but no longer maintained. We realize that MongoDB is not the best storage model for WHOIS data, so our objective is to eventually move pyDat to an ElasticSearch backend for storage.
If you are considering using pyDat for your organization, we'd love to hear from you. We have considerable experience with it and are willing to share some details about how best to import and store the data. Also, like all our Open Source tools, we welcome your contributions.