A security evaluation should help you to understand your risks. Risks are situations or properties of a system that expose things that you care about to danger or loss. For example, security problems in your product may expose you to a financial liability or cause your customers to stop trusting you and hence buying your products. Both of these can put your company, job or reputation in danger.
Click image to enlarge
Table 1 summarizes the properties of the common approaches to security evaluations.
Penetration Testing
The aim of Penetration Testing is simply to see if defenses can be broken. This means that a Penetration Tester will look for the easiest method of attack and is likely to rely heavily on automated tools targeting known vulnerabilities (e.g. security problems in software libraries). A Penetration Test will stop once a security problem has been found, as the aim will have been achieved or (in the case of the usual business model) continue for a fixed period of time or budget to try and find a security problem.
Penetration Tests therefore have limited value to help you to understand your risks as the coverage (vulnerabilities found versus vulnerabilities likely to be present) is small and there are likely to be very large blind spots. Even if the particular security problem discovered is fixed, there may be many more.
Vulnerability Assessment
A Vulnerability Assessment is very different – it actually IS a risk assessment and its aim is to determine ALL the ways in which your system can be attacked and the assets (the things you care about) accessed or damaged.
A Vulnerability Assessment considers all the assets (what you need to protect), the threats against them and all the ways that the system can be attacked. It can then propose mitigations (ways to prevent or reduce the damage) and assign risk. This then allows a fully risk-prioritized action plan to be created, resulting in the best security improvements for a given financial budget to be made.
Unlike Penetration Testing, Vulnerability Assessments rely much less on automated tools and generally require much higher levels of skill and manual effort to really understand the system being evaluated. This results in them being initially more expensive. However, the benefits are a complete understanding of the risks and the ability to get the best return of investment when fixing the problems discovered.
Evaluation methodologies
Evaluations are typically performed in a “White Box” or “Black Box” manner.
White Box evaluations are where the people performing the evaluation are provided with full information about the system. This may include things like circuit diagrams, hardware information, software with source code, network configurations, etc.
Black Box evaluations are where no additional information about the system under evaluation are provided to those performing the evaluation. This means that whoever is evaluating the product security will have to spend significant time in figuring out how the system works.
The Black Box method is supposed to simulate more of a “real-world” situation for the attack. However, with enough time a capable attacker will probably reverse engineer many parts of the system anyway. It also shifts the emphasis from the target of the evaluation to how good the evaluator is at reverse engineering!
Similar to Penetration Testing, a Black Box methodology may result in blind spots in the understanding of the risk. Just because one evaluation engineer does not manage to work out how a part of the system works it doesn’t mean that another engineer could not do this. Even if no security vulnerabilities are found it doesn’t mean that none exist.
Recommendations
If you don’t understand the security risk associated with your system, then a Penetration Test will do little to help. It can in fact lead to both a false sense of security or a quick fix and re-test attitude which will be expensive and not have a significant improvement on the real security and reduction of your risk.
If you don’t understand your risk, then a White Box Vulnerability Assessment will provide the best real understanding of risk and will be most cost effective in the long term for fixing any security issues discovered.
PDF
