In my opinion, quality of knowledge base is the most important characteristic of Vulnerability Management (VM) product. Maybe it’s because I have spent significant amount of time making different security content for vulnerability scanners and this is some form of professional deformation. 🙂 The fact is that nowadays we have dozens of VM solutions on the market, which have very different knowledge bases and thus different abilities for detecting vulnerabilities. And really nobody talk about this. I can recommend related post “Tenable doesn’t want to be Tenable anymore” and especially HD Moore’s comment to that post. It describes the reason why nobody interested now in quality of detection. Maximum what we, end-users, can hear from the vendor about it’s knowledge base is an amount of vulnerability checks: 40000-80000 and approximate list of supported systems. There is a massive false belief that detection quality of the products is approximately the same and it’s better talk about dashboards, reports, SIEM-like capabilities. To demonstrate that the difference actually exists I made a pretty primitive comparison of Nessus and OpenVAS knowledge bases.
I chose these two products, mainly because information on their NASL plugins is available at Vulners.com. As I also wrote earlier how you can use easily parse Vulners archives in python, so you can repeat it for yourself. I talked about this topic at Pentestit webinar about Vulners. If you are familiar with Russian, you can also check this out. 😉 The slides for this presentation are available here.
Why I call this comparison fast and primitive? I don’t define the structure of KBs for this product and don’t carefully map one nasl script to another. I suppose it may be a theme for another posts. Instead I am looking at the CVE links. If two scanners detect can the same vulnerabilities, they should have the same CVE links in all the NASL scripts, right? In reality we have a great difference between the products and more than a half of the CVEs can’t be detected by using both of them.
All CVEs: 80196
OpenVAS CVE links: 29240
Nessus CVE links: 35032
OpenVAS vs. Nessus: 3787;25453;9579
We can get group of the NASL scripts, “connected” with the links to the same CVEs. There are also thousands of NASL scripts in OpenVAS and Nessus that have some CVE links and can’t be mapped anyhow to the script in different KB.
All NASL plugins:
Mapped plugins: 38207 OpenVAS and 50896 Nessus
Not mapped OpenVAS plugins: 2673
Not mapped Nessus plugins: 6639
I find the last part the most valuable and interesting. You definitely can use it to clarify the weak sides of your vendor’s solutions. What are the reasons of such difference? Well, it of course may be an error. Vendor haven’t added a link to the CVE id, but the check actually exists. That’s why this method of comparison is far from ideal.
Other reasons why vendor may ignore vulnerabilities:
- “Old” software and vulnerabilities. Vendor may think, that keeping vulnerabilities for some old software is useless. Well, for some vulnerabilities in non-supported Linux distributions, like Mandrake Linux, it can make sense.
- Vulnerabilities in plugins for some software. E.g. OpenVAS detects “WordPress VideoWhisper Live Streaming Integration Multiple Vulnerabilities“, Nessus not.
- “Local” software. E.g. OpenVAS detects vulnerabilities for French project “openMairie“, Nessus not.
- Non-enterprise software and devices. E.g. OpenVAS detects “D-Link DIR-100 Router Multiple Vulnerabilities“, Nessus not.
- Stopped adding new vulnerabilities for the software. E.g. OpenVAS detects “vBulletin 3.6.x to 4.2.2/4.2.3 Forumrunner ‘request.php’ SQL Injection“, Nessus not. Nessus detects Solaris vulnerabilities since 2010, OpenVAS not.
In other words:
- Vulnerability Scanner is a necessity
- However, don’t depend too much on them
- If Vulnerability Scanner does not detect some vulnerability — it’s YOUR problem not your VM vendor
- Choose VM solution you can control
- Have alternative sources of Vulnerability Data (vulners.com, vFeed)
So, once again. The reason of this post is not to say, that one vendor is better than another. Both Openvas and Nessus have their great sides. However, there are certain gaps in knowledge bases of vulnerability management products and I believe they can be fixed in dialogue of regulators, VM vendors, Security Content developers, and independent security practitioners.
In this high level comparison of Nessus, Nexpose and OpenVAS I have made no attempt to do a detailed metric based analysis. The primary reason for this is that it would be time consuming and difficult to get a conclusive result. This is due to the large differences in not only detection but also categorization of vulnerabilities by the different solutions.
What I have done is targeted the 3 different vulnerability scanners in a "black box" test against a Metasploitable version 2 Virtualbox.
In 2010 I planned on doing an OpenVAS vs Nessus review, well it seems time got away and now its the middle of 2012. There is now a new high profile vulnerability scanner on the block; Nexpose from Rapid 7 has gained attention in recent years due to the adoption of its rock star big brother Metasploit.
In the testing I am deliberately focusing on the network vulnerability scanning capabilities rather than looking at the web application vulnerability detection in detail. It is my belief that a network vulnerability scanner should be capable of identifying poorly configured services, default services that have poor security and software with known security vulnerabilities.
Notes on the Vulnerability Scanner Testing
- External tools that OpenVAS can use have not been installed (apart from Nmap), these external tools being mostly web application vulnerability detection tools including wapiti, Arachni, Nikto and Dirb.
- OpenVAS version 5 has been tested with the full scan profile (ports were all TCP ports scanned with Nmap and top 100 UDP ports).
- Nessus version 5 was launched using the External network scan profile (also tested with Internal Network Scan however results were similar).
- The Nexpose scanner was executed with the Full audit profile.
- No tweaking of default scan profiles was undertaken.
- No credentials were used during the scan, it was an external network service focused scan.
These results are only a quick overview I have not followed up every discovered vulnerability to determine false positives and false negatives.
Edit 1st of September 2012 (clarification of scanner versions and plugins used)
Nessus : The home feed was used for the Nessus testing. According to the Tenable website The Nessus HomeFeed gives you the ability to scan your personal home network (up to 16 IP addresses) with the same high-speed, in-depth assessments and agentless scanning convenience that ProfessionalFeed subscribers enjoy.. Note when using the Nessus scanner with the home feed it cannot be used in a professional or commercial environment.
OpenVAS : The default OpenVAS 5 open source signatures and software was used. This is free to use under the GNU General Public License (GNU GPL).
Nexpose : The community version of Nexpose was tested. According to the Rapid7 website " Nexpose Community Edition is powered by the same scan engine as award-winning Nexpose Enterprise Edition and offers many of the same features." With this version you can scan up to 32 IP addresses.
And now for the results.....
External Network Profile
Full Audit Scan Profile
Full Audit Scan Profile
These total numbers without any context around the categorization of findings or the accuracy of the results provides us little value, except to highlight the wide variation in results from the different scanners.
Analysing a specific sample of Security Issues
In order to look at some more meaningful results I have examined a sample set of exploitable and mis-configured services on the Metasploitable system.
This is only a sample of exploitable services on the target host. There are many more vulnerabilities present on the system; both network services and web application security holes.
At the last minute I decided to include Nmap with its NSE scripts against the Metasploitable host. The results were interesting to say the least, while not a full blown vulnerability scanner the development of the NSE scripting ability in Nmap makes this powerful tool even more capable.
the numbers get interesting...
These are the numbers of vulnerabilities correctly discovered and rated by each vulnerability scanner; from the sample set of exploitable services.
7 out of 15 security holes identified
Notes about the sample set of tests
- All of the above vulnerabilities and mis-configurations with the exception of Anonymous FTP can be exploited to gain shells on the system (in most cases with root privileges) using Metasploit or other methods.
- There are a number of examples where the scanners do not detect weak or default credentials. While we were not specifically testing passwords, if MySQL is being checked for weak credentials why not other services?
- Items such as the INGRESLOCK backdoor and the Unreal IRCd vulnerability are fairly obscure, however this makes them good examples for testing overall capability.
- The Metasploitable version 2 release page has good examples of exploiting many of the mis-configurations in this list. This highlights not only how a poorly configured service can lead to a root shell but also the fact that vulnerability scanners need to be able to detect these types of security related mis-configurations.
These scans were conducted in a black box manner, when running internal scans it is recommended to perform credential supplied scanning. This means providing the vulnerability scanning tool with valid Windows domain, SSH or other valid authorisation so that it is able to perform checks against the local system. This is of most value when looking for missing patches in an operating system or third party software and detecting installed applications.
Vulnerability scanning is an important security control that should be implemented by any organisation wishing to secure their IT infrastructure. It is recommended by the SANS Institute as a Critical Control and by the US based NIST as a Security Management Control.
The results show significant variation in discovered security vulnerabilities by the different tools. It may be helpful to compare vulnerability scanners to anti-virus solutions; they are both an important security control that can enhance an organisations security posture. However as with anti-virus, a vulnerability scanner will not find all the bad things.
This will be common knowledge for most in the security industry who have performed network vulnerability testing. When performing vulnerability scanning, it is necessary to check the results for accuracy (false positives) and to actively look for things that were missed (false negatives).
My recommended approach to vulnerability scanning is to:
- tune the vulnerability scan profiles to suit your requirements
- perform detailed analysis of the results
- run secondary tools (nmap, a secondary vulnerability scanning solution and / or specialised tools). The use of multiple tools will provide a greater level of coverage and assist in confirming discovered vulnerabilities.
Feedback and corrections are most welcome, drop me a mail - peter (at) hackertarget.com or use the comments below.
If you have not visited HackerTarget.com before take a look at our Online OpenVAS scanner and other tools, it is my belief that performing internal focused testing in conjunction with external facing vulnerability scans adds value when working to secure Internet connected networks or servers.