What to Look For When Choosing a Vulnerability Scanner

Like any product-purchasing decision, before answering the question of which product is right, you first need to decide your specific requirements. For example, if plotting vulnerability- remediation progress over time is something you want automated, then a product's capability to log and plot multiple scan sets is a feature you need to look for. If you have a large NetWare environment, you might want to make sure that the scanner has NetWare-specific checks. If you have to scan 50–100 hosts, efficiency might not be an issue. However, if you need to scan thousands at a time, you'll want to make sure the scanner can scale to that range. Again, many of these issues are specific to what you'll need your vulnerability scanner to do.

There are also some common areas of concern that all products need to address. A few of the issues that you will come into contact with in choosing a vulnerability scanner include

· Completeness of the vulnerability checks. I don't recommend falling into the trap of playing the numbers game when picking a scanner. However, the number of vulnerabilities a scanner looks for is still important. At a bare minimum, a scanner should look for the known critical vulnerabilities that allow for root/administrator-level compromises.

· Accuracy of the vulnerability checks. It's important that scanners have a good set of vulnerability checks. However, a scanner's capability to accurately identify those vulnerabilities is also important. Missing a bunch of holes is as equally undesirable as being forced to sift through a report identifying hundreds of non-existent vulnerabilities. Like intrusion detection systems, some scanning products still have problems with false positives.

· Scope of the vulnerability checks. It should be noted that most of the vulnerability scanners are designed to discover remote vulnerabilities, not local (host-level) ones. However, a few products like ISS and Webtrends have system-level agents that will also look for local vulnerabilities—vulnerabilities that would otherwise be undetectable by remote scans. While these system agents often address a greater range of vulnerabilities, they also require installation, making them a management nightmare for large environments.

· Timely updates. Although scanners will always be one step behind the vulnerability announcements, they should be updated at a fairly regular (once per month or more) interval. You'll want to look for a scanner that has a significant R&D team behind it that is consistently updating the product.

· Reporting capabilities. Finding vulnerabilities is important, but properly describing the problems and their subsequent fixes is also important. So is the accurate ranking of the vulnerabilities. This is of particular concern for larger organizations because they usually rely on system administrators to remediate the discovered problem.

· Licensing and pricing issues. Some of these products are licensed per node, some per server scanned, and some are free. Some of them have an easy licensing system (like NAI); others (like ISS) require a convoluted key-cutting system. However, it should be noted that licensing issues should be thoroughly investigated before purchasing decisions are made, as some of these pricing schemes are just downright obnoxious. When in doubt, however, there is always Nessus, which is free.

No scanner that I know of has addressed all these issues well, but Nessus and ISS Internet Security Scanner come pretty close.