If you have a website, then chances are that your website is vulnerable to all sorts of attacks. And the bigger and more complex your site is, it's harder to keep track of potential security glitches.

This is why you need a Web Application Security Scanner, or just a Scanner. At first sight, the Scanner's job is easy. It scans your web app and generates a report highlighting the security complications it found so that you can mitigate them.

The problem is that since custom web apps can be put together in a million different ways, finding security issues with a Scanner is not always reliable. Sometimes the Scanner will find and report a vulnerability within your web app, and sometimes it won't.

So just how good are some of the Scanners out there - and how can you know if you're getting your money's worth?

To find out, I created a web app that has three big security flaws. These imperfections should ideally be detected and reported by all Scanners.

The three flaws in the web app are:

One - Sensitive information as source code comment:

<!--the username for our most sensitive data is Summer2018 with the password MegaSecret-->

Two-Sensitive data exposure:

The root folder has a file called passwd with the following content:
tom:x:1000:1000:Vivek Gite:/home/vivek:/bin/bash

Three-Code injection:

The website has a form where the user can fill in her e-mail address and press submit - and the e-mail address will be echoed below the submit button like so:

The problem is that the user input is not sanitized. So if anyone was to enter

Hello!!!

instead of their e-mail address we would get the rendered HTML like so:

And it's also possible to insert JavaScript and all sorts of other unpleasant stuff.

Meet the scanners

Patronus.io: They claim to be an -"automated security solution for your website". Sounds good to me but let's see if it's true shall we?

Detectify: Say that they "scan your website for security issues crowdsourced by 150+ white-hat hackers". Sounds pretty underground and cool doesn't it? But is it all talk?

Tenable: -They "…safely, accurately and automatically scan your web applications, providing deep visibility into vulnerabilities and valuable context to prioritize remediation". Great, but does it work?

Round one - Discovering sensitive information as source code comment

No -Patronus.io
Yes - Detectify
No -Tenable

Round two - Discovering sensitive data exposure

No - Patronus.io
No - Detectify
Yes - Tenable

Round three - Discovering Code injection

No - Patronus.io
Yes - Detectify
No - Tenable

So who won?

The winner

The Detectify scan took by far the longest to complete. But it sure was worth the wait. Detectify managed to find both the code injection vulnerability and the source code comment containing the username and the password. Detectify failed to find the passwd file but I can live with that shortcomming

Runner-up

Tenable had no problem finding the passwd file in the root directory. However, the code injection vulnerability flew under the radar as did the username and password exposed in the HTML source code

Dead last

Patronus on the other hand is surprisingly week. Their scanner did not find any of the three vulnerabilities. I suggest you take your money elsewhere as this product is only going to give you a false sense of security. Security tools that are this weak should be banned altogether - we simply deserve better.

Conclusion

Automated scanner aren't very good. Sure, they can catch a low hanging fruit or two - but they are still way behind manual penetration testing. Scanners should therefore only be seen as a small piece of your security puzzle. Because if you rely on scanners alone for your web app security posture you might be in for a nasty surprise.