Hot off the success of ShmooCon 2013, where the Open Security Training community rallied colleagues to become instructors, we're talking with Nate Adams. Nate is one of the instructors of a 3-day class, Introduction to Vulnerability Assessment, which he co-instructs with his MITRE colleagues Chriss Koch and Jose Cintron. The class content was also developed with the assistance of many of our cyber security engineers.
Editor: Nate, thank you for talking to us today about your introductory class on vulnerability assessments, clearly an essential element of engineering cyber defenses. How does someone get started learning this area of expertise?
Nate Adams: In our class, we cover a general methodology that defines, shapes, and makes repeatable the process of system vulnerability assessment. We do this by building on the commonalities of computing technologies, really just the basics, so that some groundwork is established for developing quality assessments. After presenting a methodology, we review common vulnerabilities in various computing technologies such as operating systems, network devices, databases, and applications. The class also dives a bit into the use of a number of basic security tools, like vulnerability and port scanners, client-side web proxies, network protocol analyzers, and some hands-on testing techniques for using these tools to discover vulnerabilities in systems.
[Ed.]: We know vulnerability assessments are an important part of managing risk, but what do you think would make this area of knowledge appealing to new staff?
[NA]: I can ask different people across academia, industry, or government to describe the differences between specific types of security testing, such as vulnerability assessment or penetration testing, and I will likely get different answers. The security community needs a common understanding in this area, and our class helps do that.
Everyone who does any type of security testing, including me, likes to break things, but that's not really the best approach to securing a system. Although penetration testing seems the more interesting path for many, our view is that it's the quality of testing outcomes that counts. Unfortunately the quality of test reporting can widely differ depending on who performs the testing and their skillset. A good vulnerability assessment is just as valuable as a good penetration test. Both have an effect on security when done well.
Plus, being able to find vulnerabilities before the bad guys do will better protect our systems. And it will allow organizations to implement systems that are as secure as possible, within the acceptable risk profile they've determined.
On the first day of the class, we define what we believe are key differences between types of security testing, and we attempt to persuade our students that the different types of security testing are complementary, each with a specific purpose.
Before any type of security testing can occur we need to ask: what goal(s) is the organization trying to accomplish with security testing? It is the specific goals that determine the choice of testing. Most security testing uses similar methodologies, techniques, and processes. The difference is in the outcome.
In the class, we define vulnerability assessment as a cooperative effort to find all of the potential vulnerabilities in a system within the time, funding, and resource constraints of the assessment. On the other hand, penetration testing is meant to emulate a specific attack in order to demonstrate the full impact of specific vulnerabilities or to determine the maturity of an organization's incident response process to that attack. The bottom line is that penetration testing should drive organizational change not just remediate risk.
[Ed.]: So vulnerability assessment is an area that provides a lot of opportunity, wouldn't you say?
[NA]: Yes, I would. Most any system or security engineer who supports a system during its development lifecycle can not only benefit from this class, but also are likely to become a valuable resource in the system engineering lifecycle. By being aware of quality security testing methodologies, processes, and techniques for performing vulnerability assessments, or any other security testing for that matter, as well as when they can be applied in that lifecycle, you can identify security issues sooner rather than later.
Traditionally, many organizations don't perform vulnerability assessments until after a system is deployed, which is too late. With a number of projects we support, we have been able to perform assessments on systems and code during development or right after integration. That's real value to cyber defense. Getting people educated about common vulnerabilities at the network, host, application, and data layers, will go far in securing systems for our government sponsors.
[Ed.]: So, you're actually doing this this type of work for MITRE?
[NA]: Yes, I do get to perform hands-on security testing, primarily for our customers. But I'm also doing quite a bit in the development of this area at MITRE. Along with Jose Cintron, I facilitate a MITRE Community of Practice (CoP) for Security Testing, which is helping define security testing terms of reference so that we can all speak the same language. I'm also involved in coordinating efforts to standardize processes that will produce quality deliverables.
[Ed.]: Quality deliverables sounds like a differentiator for interpreting and sharing results. How are you achieving this?
[NA]: Well, we've created templates for deliverables, like test plans and test reports, pre- and post-assessment briefings, and even custom analysis tools that can be adapted to the specific scope of the assessment. Using similar formats for the presentation of test data helps us correlate vulnerability and mitigation data across multiple systems within an organization and even among different organizations.
[Ed.]: This seems advantageous for achieving comparable confidence levels for cyber defense. But the tools and processes are only as good as the people using them, so how does someone get to your level of expertise? Your class is an introduction.
[NA]: Well, it doesn't happen over night, of course. It takes practice. The class was originally five days and had a lot more technical depth, but we found that it was difficult to keep everyone fully engaged because of the variety of skill levels that would enroll. So, we shortened the class to three days.
As you mentioned, the intro is just the first step, but does establish the foundation necessary for performing vulnerability assessments. As a follow on to this base, we currently have two advanced courses, one in Windows operating systems and the other in Web applications, useful for engineers that need more focused skills and practice.
I can't say it enough—practice is important, and labs, such as those in our class, help with that, but if you can, jump right in to doing vulnerability assessments once you're back on the job. It's the only way to retain and hone your skills. Just make sure you have the authority or permission to practice or test on a system.
At MITRE, after the classes end, we pair up interested engineers with experienced assessors to work on actual projects. This gives them the experience of performing real testing and finding real vulnerabilities that aren't easily replicated in a lab environment. This is especially important when it comes to custom-developed applications. Although many developers use common frameworks like Java, each application can be very different from other applications because they each have a different business purpose. You have to jump in to the real thing to get that experience.
[Ed.]: What's next?
[NA]: We expect to develop some classes that will do deep dives in vulnerability identification in non-Windows operating systems, network protocols, databases, and non-Web-based application security.
There are also vulnerability issues around management and operations that should be addressed, such as Employee Awareness and Training. An organization that fails to provide adequate awareness and training for things like social engineering attacks are perhaps just as vulnerable as a Web application that allows a SQL injection attack. We're trying to decide how best to cover these management and operations topics, perhaps either as a separate class or integrated into the current classes, since not all vulnerabilities are technical in nature.
[Ed.]: Thanks, Nate, for sharing your views on how vulnerability assessments are a valuable part of reducing risks, especially before systems and apps become mission critical. The intro class, and your follow-on ones, will help develop the expertise we need in this area. Check out this class at the Open Security Training site.
