For many years now, it has been impossible to imagine building solutions not relying on open source. In fact, every project I've worked on has benefited more or less from community development. This trend doesn’t apply only to product companies and start-ups. Large financial institutions and other critical sectors are also reaping the benefits of open source. The State of Open Source report by OpenLogic & Open Source Initiative, among others, confirms this statement.
The 2022 State of Open Source Report Open Source Usage, Market Trends, & Analysis by OpenLogic & Open Source Initiative
Such extensive use of open source can lead to some problems. Anyone who has worked with npm or maven based applications knows this. Developers, including myself, are often tempted to rely on external libraries and tools. This makes list of dependencies grow and grow, and it isn't easy to keep track of it. This raises the issue of trust in these dependencies. At some point, our project/product will need to generate reports with vulnerabilities and licenses of dependencies.
As software engineers, we should care about the quality and security of our solutions. For this reason, we should address this topic as early as possible in the Software development lifecycle (SDLC).
I have recently been researching and evaluating the most popular commercial products for open source security scanning - Sonatype, JFrog, Snyk. I have decided to bring all my outcomes together in this article. The comparison will be subjective and will refer to aspects that were crucial in my use case.
The technology stack of my project - a key aspect for further comparison:
- Java/Kotlin microservices
- Docker images as artifacts
- GitHub as code repositories & CI/CD
- Kubernetes
- IaaC with Terraform
Under this Miro Link you will find the same table colored and better formatted than in limited Markdown.
| Sonatype | JFrog | Snyk | |
|---|---|---|---|
| SaaS | No SaaS option. Planned for next years. | Available. | Available. |
| Self-hosted | Available. | Available. | Not available. |
| Free plan | Not available. | Available and sufficient for small projects but only for SaaS. | Sufficient for PoC and private home projects. |
| Pricing | Very high even for medium-sized teams due to minimum number of licenses required. | Reasonable for medium-sized teams. High cost with full package and more licenses. | Reasonable on any scale. |
| Licensing | Very complicated. A large number of tools and licenses create a confusing ecosystem. It's hard to find clear information in the documentation. Without contacting sales team, it is practically impossible to estimate the final price.Not possible to test anything without a trial license. To get it you have to arrange a series of meetings with sales (3 in my case) to starts with PoC - it costs a lot of time. | More complicated than Snyk due to several installations options (SaaS, Self-hosted) and a list of additional sub-products. Possibility to try a paid version after contacting the sales team. | The most transparent pricing. Possibility to test a paid plan for free without providing credit card and contact the sales team. |
| Docker images support | Not directly supported. In order to scan Docker image, it must be saved as tar archive. Link | Supported. | Supported. |
| CI/CD integrations | Not too good support. A lot of samples and instruction for old-fashioned Jenkins in offcial docs. For GitHub only not mantained community action. However, the list of supported CI/CD is growing, so I hope this will change in the future. | limited list of integrations. As for GitHub, you can use the official GitHub Action with the built-in JFrog CLI. | Support for most fo modern CI/CD’s including GitHub offcial Action and AWS Code Build. |
| IDE integrations | Provided for IntelliJ, VSC, and Eclipse with few shortcomings like worse dependency filtering for gradle projects. | Provided for VSC, most of JetBrains IDEs, and others niche players. For IntelliJ it has some nice features. | Provided for VSC, VS, JetBrains, and Eclipse. In IntelliJ very convenient and clear. Support also static code analysis. |
| CLI support | Available with few shortcomings. You have to always pass credentials to each command. Output returns a link to rich report in IQ Server but can’t generate well-formatted summary in CLI output. | Very powerful JFrog CLI. Great well-formatted summary of scanning in console output. Poor CLI documentation on the website. | Powerful CLI. Authentication with a Snyk account. CLI scan can export result in SARIF format! |
| Vulnerabilites information | Rich and well structured information about vulnerabilities. Usefull data about risk and the attack vector. Info how to determine if you are vulnerable. Recomendations how to fix or work around issue. Flagship copmonent called Version graph. | Has all important information about risk, attack vectors, and advices how to deal with vulnerability. Information is clear. | Probably the least informative of the other tools. However, there is everything that is most important. When it comes to the presentation layer, there aren’t too many bells and whistles here. It may be less readable due to small font used. |
| Open soruce licence scanning | Extensive possibility of managing licences. Black-listing of individual licences, manual verification, etc. | Just like Soantype, a wide range of license management options. | Very similar solution as in other tested tools. It seems to be poorer and less complicated, but therefore easier to use. |
| File reports | Possibility to generate PDF Report. Too bad you can't export to HTML. I can’t find option to generate ad-hoc report for more than one artifact/repo. | Possibility to generate PDF, JSON, or CSV report. No HTML option. You can include in report more than one project/artifact and apply advanced filters. | In Snyk web UI you can only export as CSV. CLI can produced JSON or SARIF scaning results. |
| Notifications & alerting | Limited number of built-in integrations for notification - only Email and JIRA. Fortunately, it is possible to configure Webhook. | Only Email & Webhook notifications. | Possible to setup Email alerting, built-in Slack notifications, custom Webhook or even JIRA integration. |
| Static code analysis | Not supported. | Not supported. | Supported with Snyk Code. Kotlin language is still in beta. With Snyk IaC you can also scan Kubernetes and Terraform configuration files. |
| Web browser UX | Mixed feelings. On the one hand, the new UI looks clear and not hurts eyes, on the other hand, it often took me a long time to find something. I think the main tabs could be better named and grouped. Somewhere you can still enable the old UI. | My first impression was good, but I tested JFrog first. Perhaps, if I had the opportunity to test it again I would have a more reliable opinion. | Well thought-out and intuitive. My only objection is to the clarity and smallness of the fonts. |
| Documentation | Poor impressions. In the documentation, you can find a lot of old, deprecated-looking content. Googling to find something, we often end up on a marketing page rather than proper documentation page. | As with Sonatype, the documentation looks poor. It's hard to find something navigating through their pages. Many articles, but poorly organized. | A great example of how documentation should look. Truly one of the best I've dealt with. Well structured. Navigating step by step from the most important things to the details. Lots of graphics and what developers like the most - code samples. |
Summary
It is easy to see in the colored table that Snyk won my sympathy. IMHO, Snyk fits modern projects based on microservices, Docker, and CI/CD ala GitHub Actions. Sonatype turned out to be the least suitable for my project. However, I wouldn’t completely reject this tool. I think it has its advantages which you will see in projects that use Jenkins or private package repositories.
In the end, however, we didn't choose it. Instead, we decided to build our own security scanning pipeline based on open source solutions such as Trivy or Syft.
As I mentioned earlier, the above comparison is my personal assessment. My conclusions are very subjective as I did this research to find the right tool for my project. For a different tech stack, perhaps the conclusions could be different. I made the tool comparison in August 2022, so if you are reading this after a long time, know that some things may be out of date.
Top comments (0)