A quick comparison of Security Static Code Analyzers for C#

Intro

Static Application Security Testing (SAST) has become an integral part of any Secure Development Lifecycle. It is relatively fast and can catch security issues before committing to the codebase. You may hear Veracode, Coverity, and Checkmarx. These players are a long time in market. Their analyzers are slow and expensive but mark all checkboxes for a typical "enterpise".

I will test new players for a single language C# and a single vulnerability type - SQL injection. My focus area will be quality of analysis (though it is wrong to make a decision based on a single issue type), vulnerability review (devs can quickly validate findings), and remediation guidance (devs know how to fix it). I would probably try "enterprise" stuff as well if they provide a free tier for open-source projects, but I could not find this option.

So, here are the generic requirements for the SAST tool:

• Free to try for public repositories
• Supports C#
• Provides security rules, can find security issues
• Supports Taint Analysis (data flow analysis from input (source) to vulnerable invocation (sink))
• Provides a proper description and how to fix it

Bonus:

• Fast enough
• Small number of false positives
• IDE plugin (so devs can view results locally)
• Supports SARIF export (so results can be viewed by other tools, e.g., Github Security Scanning)

Candidates:

• CodeQL aka Github Security Scanning (GHAS) - a full-sized security solution from Github (formerly Semmle LGTM)
• SonarCloud - affordable but powerful, a cloud version of SonarQube Dev Edition
• Semgrep - regex on steroids, you don't need to compile anything, and you can create your own rules
• Security Code Scan - Roslyn-based C#/VB only analyzers with minimal integration options, but super fast
• Snyk Code (formerly DeepCode) - "Dark magic" AI-based, you don't need to compile the code

They are quite different on purpose. Let's see them in action.

Honorable mentions:

• Sonarlint for C#, Codacy for C#, SonarQube Community Edition - provided rules are great for quality checks only; they lack Taint Analysis, so no Injections can be found!
• Official Roslyn Analyzers from MS (they have a minimal set of sinks meaning they can catch just a few methods).
• Puma Security - commercial version of Security Code Scan. The community edition could have been better, but the Pro version sounds interesting.

Testing methodology

I'm not planning to use Juliet Test Suite (just >28000 test cases for C#) or OWASP Benchmark. Juliet Test Suite is very synthetic, it is a good idea to validate your SAST against it, but nobody shares the results. OWASP Benchmark is a bit easier to run, but companies also do not publicly share results. You can ask for them, and maybe you will be lucky.

So, I created my project SharpSaster - https://github.com/dbalikhin/SharpSaster. It is .NET 6 Web Application, fully compilable but without an actual database. Just enough for our SAST folks.

There are 5 controllers (uses ControllerBase class):

• SqlBasicController: one safe method EfCoreInjectionSafe (with multiple sinks), and 8 vulnerable methods - EfCoreInjectionVulnerableX. We access a database with EntityFrameworkCore here. Test new stuff!
• SqlBasicSinkController: 3 safe and 5 vulnerable methods + extra vulnerable method SqlClientVulnerable10 where we have a parameterized stored procedure but user controllable stored procedure name. We use old-school SqlCommand here. Test old stuff!
• SqlAdvancedController: 2 safe (integer conversion), 2 vulnerable (1 or many vulnerable sources).
• SqlFlowController: 4 vulnerable flows with a single sink! 1 unreachable safe flow that easy to detect.
• SqlFlowSingleSourceController: Similar to the previous class, but for each source, we will have our own sink (I found that some tools combine findings be a sink. If you fix a sink (e.g. with parameterization), you will fix all sources. Although I prefer to see a group of sources combined by a sink, see Results section).

All input (sources) will come from an action of the controller.

Bonus: A typical SAST tool without customization cannot catch conditional cases. I included a couple of them, and all of our candidates marked them vulnerable, as expected. Therefore, I excluded conditional methods from the quality assessment.
An example of the conditional case is below:

var localCondition = false;

if (localCondition)
{
    ExecuteSQLCommandWithVulnerableInput(string source);
}

You should not expect that your SAST analyzer will understand conditions/validation. However, just a few tools support user-controllable customization.

Taint Analysis 101

There is input (user-controllable source) - a parameter in the API controller, main method of console applications, etc.
There is a method that may execute vulnerability (sink) - for SQL injections, it will be any method sending a SQL command to a server.
The source is marked as tainted. If it reaches the sink without sanitization, we have a vulnerability.
Sanitization - any known method that will "untaint" (make it safe) a user-controllable value between the source and the sink.
"Tainting" - any assignment / referencing to the tainted variable. I just created this term, but it is what is happening in reality.

public IActionResult Controller(string source)
{
    // source - I'm tainted
    var otherVariable = source + "stuff";
    // otherVariable - Now I'm tainted too :(
}

An analyzer needs to perform data-flow analysis from the source to the sink (+ from any tainted value in between) "untainting" variables with sanitizers. This is called Taint Analysis.
Sanitizer creates a new safe value as an output. XSS Encoding method or SQL parameterization would be examples of SAST sanitizers, or at least in my interpretation 😉.

Validators do not change "tainted" status. They do not produce output that will be used later. That's why all conditional cases are problematic for analyzers.

A good SAST tool should know as many sinks and sanitizers as possible (sources are usually very limited). And, of course, it should have a solid taint analysis engine.

Categories of results

True Positives / False Positives

How many findings were detected correctly (true positives) or declared safe code as vulnerable (false negatives)?
"Conditional" issues are ignored!

CodeQL (100% of true positives with 0% false positives, conditional stuff is out of scope):

• SqlBasicController: 8 of 8 found, 0 false positives.
• SqlBasicSinkController: 6 of 6 found, 0 false positives, combining results by a sink if there are 2 sources.
• SqlAdvancedController: 2 of 2 found, 0 false positives.
• SqlFlowController: 4 of 4 found but in a single issue (combining results by a sink), 0 false positives
• SqlFlowSingleSourceController: 4 of 4 found similar to SqlFlowController but cannot combine results because we have unique sinks, 0 false positives

SonarCloud (100% good findings for EFCore, almost 100% bad for SQLCommand. It is an unpleasant "surprise" for me + it is weird to see some SQL Injections in the Hotspots area):

• SqlBasicController: 8 of 8 found, 0 false positives.
• SqlBasicSinkController: 0 of 6 found. Checking Security Hotspots.
• SqlAdvancedController: 2 of 2 found, 0 false positives.
• SqlFlowController: 1 of 4 found. I could not find a way to display all sources. A single sink => a single source.
• SqlFlowSingleSourceController: 4 of 4 found, 0 false positives

SonarCloud has 2 types of security findings: vulnerabilities (confirmed) and security hotspots (maybe context dependant, requires manual review). I found a single issue for SqlBasicSinkController in the security hotspots area. Technicality, 0% false positives, just missed vulnerabilities.

For some reason, SonarCloud created 22 security hotspots, some for safe items, and other for confirmed vulnerabilities. Unfortunately, I cannot explain why it is doing that.

Semgrep (terrible findings - basically 0% true positives, I had to add an example from the official ruleset to make sure I configured it correctly):

• SqlBasicController: 0 of 8 found, 0 false positives.
• SqlBasicSinkController: 1 of 6 found, 0 false positives.
• SqlAdvancedController: 0 of 2 found, 0 false positives.
• SqlFlowController: 0 of 4 found, 0 false positives.
• SqlFlowSingleSourceController: 0 of 4 found, 0 false positives.

I didn't have any findings, so I went to the rule to check how it worked. I am confident that my security policy is properly configured, it is not a great ruleset for SQL Injections provided by Semgrep.

Security Code Scan (100% of true positives with 0% false positives. It does not group findings by the same sink, so more issues).

• SqlBasicController: 8 of 8 found, 0 false positives.
• SqlBasicSinkController: 6 of 6 found (technically 7 of 7, I do have 2 sources with a single sink in one of the cases), 0 false positives.
• SqlAdvancedController: 2 of 2 found (+ extra finding from 2 sources with a single sink), 0 false positives.
• SqlFlowController: 4 of 4 found (not grouping by the same sink), 0 false positives.
• SqlFlowSingleSourceController: 4 of 4 found, 0 false positives.

Snyk Code (it found 100% of true positives. However, it failed significantly on safe EFCore statements).

• SqlBasicController: 8 of 8 found, 4 false positives.
• SqlBasicSinkController: 6 of 6 found, 0 false positives.
• SqlAdvancedController: 2 of 2 found, 0 false positives.
• SqlFlowController: 1 of 4 found (I could not find any grouping, so a single sink detected), 0 false positives.
• SqlFlowSingleSourceController: 4 of 4 found, 0 false positives.

True Positives / False Positives Summary

CodeQL is a leader along with completely free Security Code Scan. Snyk takes a second place, SonarCloud - third. Semgrep is hardly usable for security needs in this testing scenario.

Snyk shows inconsistent results, but overall, 100% of true positives with some false positives are better than missed issues.

In real life, I saw much better results produced by SonarCloud (SonarQube Dev Edition), but not in this run. I'm disappointed.

Taint Analysis and Visualization

After a scanner finds an issue, a developer/security specialist will need to validate the finding. For SQL Injection issue type, we will need to review the code flow and make sure that tainted input (source) goes all the way to the SQL invocation (sink) without having any known sanitizing function in between. It is called taint analysis (a form of data flow analysis). Not all security rules require taint analysis, but if you want meaningful results for SQL Injections, you will need proper taint analysis.

A scanner should produce a trace in a consumable format so a human can identify the correctness of the findings. Let's check the output of the tools and compare them. I will mostly use "Flow" and "Advanced" controllers.

Note: If a tool supports Github Annotation and allows you to upload results to Github in the SARIF format, you can see annotated code within Github Checks.

CodeQL

CodeQL combines findings by the same sink in the UI. So it makes sense - you need to fix a single sink, e.g., by adding parameters to your SQL statement.
It shows the found sink; to get more details, you need to click on Show Paths to review all available sources.

Remember, we have 4 issues here + 2 conditional "safe" flows, that no tool can detect; you need some old fashion manual code review for that.

CodeQL can properly identify the source parameter - string input.

You need to scroll down to review the sink. The flow is easy to understand, and a short comment is provided for each step. The execution order is shown.

The only possible improvement is to include a variable assignment in the last block, so we can see how concatSql is crafted - I selected this line manually, see below.

Now let's check multiple sources on a single line.

In this scenario, we have 2 vulnerable inputs, CodeQL created 2 paths for them, and each one clearly identifies the vulnerable source.

Overall, the taint analysis looks great. A bit narrow window, so you need to scroll horizontally sometimes.

SonarCloud

SonarCloud doesn't combine findings by a sink; it will only report "first" finding. But it is doing it beautifully.

Execution order can be easily followed by numbers and comments on the left side. It marks all required code, nothing to add or remove. Unless we are talking about conditional execution 😊
My only concern, it doesn't show the exact source parameter.

Now let's check multiple sources on a single line.

Oh well, it simply highlights the method with the source but not the parameter. In our case, user is not vulnerable, but name. It needs to be clarified from the screenshot.

The tool supports SARIF upload, but you will still need to go to SonarCloud website; it is a simple link on the Code Scanning page with basic remediation examples.

Overall, the taint analysis looks great too. The UI requires fewer actions to take than CodeQL. However, a scanner needs to identify a source more specifically.

Semgrep

I have a single finding. Here it is.

I'm not sure if I can say Semgrep supports Taint Analysis. The rule is not simple, but not complicated as well. Semgrep stores just a finding with reference to the Github (permalink).

It supports SARIF upload, so you can use Github Security Code Scanning page to view results.

Security Code Scan

The tool supports SARIF upload. The message format is from the Visual Studio Error List window, a single line with minimal required information that is not easy to read but provides some value. We can get the source, the accessing method, and the sink. I would love to see improvements.

Snyk Code

Snyk Code doesn't combine findings by a sink. Taint analysis visualization is easy to understand, and execution (data) flow provides more steps. It displays almost all tainted data, where CodeQL and SonarCloud prefer to show only method invocation and assignments related to reaching the sink. It may be helpful if you need to validate "validators" (by reviewing the whole data flow) manually - different goals.

Sometimes UI is not responsive, but you can switch between the Data flow and the Fix analysis tabs to refresh it.

So you will get results after all.
It can properly identify what origin of the source used. No need to guess if one or more parameters are vulnerable in the same method.

You can navigate through the steps on the left side. Overall, it is a decent solution.

Note: I didn't check if Snyk Action allows you to upload to Code Scanning in SARIF format.

Taint Analysis and Visualization Summary

CodeQL, Snyk, SonarCloud provide great data flow visualization.
Only CodeQL can show different paths for the same sink, so I will give it 1st place.
SonarCloud visualization is more intuitive, but it doesn't show what origin of the source was used. Snyk provides much more steps but can properly identify the source. They will share 2nd place.
Security Code Scan and Sempgrep do not really visualize anything, so they are in 3rd place.

Remediation Guidance

CodeQL

You will be redirected to Code Scanning Page to view details. Don't forget to expand "Show More" section. Recommendations, samples, and references are useful. First-party integration makes it look beautiful.

SonarCloud

It provides enough information if you click the "Why is this an issue" link.

It is even better for Security Hotspots, so I wonder if they decided to mark my SQL injections as a Hotspot for this reason 😊
You can navigate between multiple tabs to get more insights about the vulnerability. Pretty cool.

Code Scanning Page

Semgrep
Semgrep could be more generous in providing details. It is a bare minimum, and I need more!

Code Scanning Page

Security Code Scan
This is Sparta! It will provide a link where you can read about the vulnerability type, bad/good code, and find other references. Still useful.

Code Scanning Page

Snyk Code
They made some progress from the last year, but it can be better.
"Details" and "Best Practices" sections are too generic; samples don't make sense most of the time.

How can I fix a SQL injection with this?

Remediation Guidance Summary

CodeQL, SonarCloud, Security Code Scan provide a proper description and compliant/non-compliant samples - 1st place.
Snyk Code and Semgrep - 2nd.

Other features

This is a light assessment of the tools.
I can also compare some other valuable features of these tools but without providing any reference 😉

Performance
It depends on technology, the aggressiveness of the scan (e.g., the depth of analysis), and how well-optimized are preparation steps.
CodeQL, SonarCloud, and Security Code Scan will need to compile the code first, so they will also need to fetch the required dependencies. It slows the process, but still, it will be minutes only. On small codebases, expect the time to be between 3-5 minutes. Then it grows this way:

• for Security Code Scan - extra seconds
• for SonarCloud - extra minute or 2
• for CodeQL - extra minutes

Semgrep and Snyk will complete a scan for a small project within a minute. Simply remove building time from the equation. For bigger projects, analysis time increase is somewhat linear.

IDE support
The most mature will be SonarCloud with its SonarLint extension in the connected mode. However, you cannot perform taint analysis on the client; results will be fetched from the server and displayed in your IDE.
Snyk Code made great progress from the last year, more IDEs supported, and you will get a similar to the portal (app.snyk.io) experience. The plugin will need to upload all files to be scanned on the server, an initial scan will be longer, but then it can handle only delta.
Semgrep supports a limited number of IDEs, but scanning is happening locally.
Security Code Scan is nothing more than a NuGet package with Roslyn analyzers. It will populate your error list, and you can utilize Visual Studio's built-in features. It works in entirely disconnected mode, and scanning is local.
CodeQL doesn't support any IDE, as far as I know. Technically, you can try to export to SARIF format in your Github action and review results with SARIF viewer extension in your IDE.

Price
All options are free for open-source projects.

If you are developing a closed-source application, you will still pay nothing for Security Code Scan.

SonarCloud is excellent for small and medium businesses. Its "line of code" price model is pretty cheap. Enterprises may prefer SonarCloud because of the cost/value and quality/performance ratio.

Semgrep has a free community tier (up to 20 developers). Snyk Code has a Free tier with a hard limit of scans. If we compare Team / Enterprise editions, they will be about the same. Even the same as CodeQL. Expect to spend at least 40 dollars per user per month. Well, your negotiating skills can definitely lower this number, all enterprise-oriented companies are quite flexible 😊

Don't forget that the Software Composition Analysis add-on is "free" for Sempgrep and Github Advanced Security if you use their SAST solution. However, it costs extra for Snyk because it is an entirely separate product.

Summary

I will remind you of my focus area first:

• quality of analysis
• vulnerability review
• remediation guidance

CodeQL is a mature and solid solution with excellent results in any focus area. Just remember that it doesn't support IDE and will be the slowest one in our group.
If CI/CD checks are enough for you, go for it.

SonarCloud is also a very mature solution. It supports more programming languages, including very exotic ones. Sometimes it will produce not ideal results, but it is easy to use and costs little. It definitely helps to improve security in your organization.

Sempgrep is still in the beta phase in my opinion. You can easily customize rules for your need, and speed is great, but security checks are limited to the underlying engine. I encourage you to check available rules and decide if it is worth it.

Security Code Scan can be your "second" scanner for C#/VBA. It works in an IDE ultra-fast (literally a couple of seconds at most) and produces solid results. CI/CD integration is very basic, but it is entirely free.

Snyk Code is close to being a solid product. They need to slightly improve quality of analysis and their remediation guidance. Will it justify a high price? Well, maybe, the speed is great.