People sometimes have a tendency to conflate terminology when referring to non-identical but related concepts. It's human nature to do this, and if you're looking for it, you can hear it happening all the time. For example, some people refer to any paper copy as a "Xerox," any sparkling wine as "champagne" and any overnight parcel as "FedEx."
This often happens in the compliance sphere, both in the language we use to describe technical points like "control implementations" and in the language technology partners use to describe those implementations. Compliance, however, is unforgiving when it comes to precision in language, and subtle differences in words can mean the difference between compliance vs. non-compliance. Varying interpretations of terms can lead to inappropriate control selection, unclear or inaccurate management responses, the misrepresentation of controls to auditors, and many other ills. Put simply, these differences can result in the very violations we are striving to avoid.
A common area of confusion is a vulnerability assessment vs. a penetration test. These terms are increasingly relevant in a regulatory context, either as a specific line-item requirement in certain regulations (e.g., Payment Card Industry requirements 11.2 and 11.3, National Institute of Standards and Technology SP800-53 RA-5), or as supporting controls selected by our technical peers in meeting non-prescriptive control objectives (e.g., Health Insurance Portability and Accountability Act 45 CFR 164.308(a)(8), ISO/IEC 27001:2005 15.2.2). Vulnerability assessments and penetration tests are often used interchangeably by technology peers, auditors and sometimes even regulators themselves.
From a regulatory standpoint, the selection of a vulnerability assessment or a penetration test will depend on the specific control objective you are trying to meet.
In fact, a penetration test and a vulnerability assessment are very different exercises. As a result, it behooves compliance professionals to become educated on their differences to make sure they're appropriately satisfying compliance objectives, their controls do what they expect, and they're representing control implementations accurately to others.
Vulnerability assessment vs. penetration test
A vulnerability assessment is, at a high level, any activity designed to investigate and subsequently enumerate potential attack points (i.e. the vulnerabilities) within a given context. In a corporate information technology context, for example, it usually comprises a focused investigation of information systems (applications, systems, network devices, etc.) in order to locate issues that impact the security of that environment. These can include missing patches, insecure configurations and/or weak passwords.
Vulnerability assessments can be general in nature, or can focus on a particular level of the technology "stack," such as an application-level vulnerability assessment. In practice, it can include technical scanning activities, such as running an automated vulnerability scanning product to query the scope of systems and devices to produce a report on what security issues might be present. Again, the purpose of the activity is to enumerate all -- or as many as possible -- of the potential issues that could negatively influence the environment's security.
On the other hand, a penetration test is more specialized than a vulnerability assessment. Instead of trying to enumerate all possible issues, a penetration test instead attempts to simulate real-world attack scenarios the system or environment might be expected to encounter in the field. The goal of a penetration test is to evaluate the overall security profile and, in turn, the efficacy of the specific security measures in place.
A penetration test might be structured to simulate a determined attacker attempting to leverage technical vulnerabilities to steal data. Like a vulnerability assessment, the test might start with itemizing vulnerabilities, but that's just a stepping stone to an end goal that usually includes exploiting those vulnerabilities.
More on IT security and compliance
Compliance's influence on risk management and security
The harsh realities of information security and compliance
As you can probably imagine, what differentiates these two activities is the purpose and intent of the exercise. Under the hood, the methods can be -- and often are -- quite similar. That is part of the reason the terminology sometimes overlaps.
For instance, a vulnerability assessment will very often include an automated scan of the environment using a specialized scanning tool. A penetration testing exercise might start with running these exact same tools. Further, both types of activities can include vulnerability scoring and prioritization. In the case of a vulnerability assessment, the purpose is to provide information about the relative severity and remediation priority for the located issues. The penetration test, on the other hand, is designed to avoid providing attackers with information that gives them ideas on fruitful attack avenues.
From a regulatory compliance standpoint, the selection of a vulnerability assessment or a penetration test will depend on the specific control objective you are trying to meet. Because these terms are often used interchangeably, compliance professionals may find it productive to ask some key questions rather than assume a technical process or control implementation includes particular characteristics. Some key questions to ask to determine if a vendor/partner/employee is referring to vulnerability assessments or penetration testing include:
- What is the process output or report format?
- Will it include a vulnerability list, activity report or something else?
- Is there a manual component or just automated scanning?
- Will testers attempt to gain access to sensitive resources?
Don't take it for granted that your partners mean what you think they mean when you're discussing vulnerability assessments and penetration testing. By making incorrect assumptions, you may not be getting the controls and processes you wanted -- and it could lead to more trouble than one might anticipate.
Ed Moyle is a founding partner at New Hampshire-based information security and compliance consulting firm SecurityCurve. Moyle previously worked as a senior manager with Computer Task Group Inc.'s global security practice and prior to that served as a vice president and information security officer at Merrill Lynch Investment Managers.