Cover image credit: Photo by Kevin Ku on Unsplash
Software and application development is difficult. Producing code that is flaw and error-free is even harder, sometimes finding the problems in our software can be half the battle of creating applications that are secure, reliable, and performant. We spend an enormous amount of time and money running, testing, and monitoring our code trying to find the root cause of an issue that has popped up either before or after the application went live, hoping that the next issue doesn’t rear its ugly head in production. So to prepare for the inevitable we implement tools everywhere, trying everything we can to catch it when it does. But what if we could reduce that overhead, and give ourselves a “head-start” to finding issues while we are still in the development phase? In this article, I’ll introduce runtime analysis and describe what it’s good for. I’ll also give you a tour of how AppMap uses runtime analysis to identify hard-to-find issues within your application.
What is runtime analysis?
Runtime analysis is a process of analyzing code while it is running, capturing how your code behaves when it is living and working with actual data. Traditionally, developers use static code analysis tools to examine patterns and create a model of their code and how the data flows through and application without actually executing it. Usually, if we want to watch processes and see what happening in our live running applications we employ Application Performance Monitoring Tools running in our production environment. This option is expensive and rarely shows where the problem occurred in the source code, which can make it challenging to integrate a solution back into the code. Runtime analysis gives us a “best of both worlds” scenario. When runtime analysis is done within the development environment, developers have an easy and accessible way to observe running processes, analyze code patterns responsible for those processes, see how and what data is affected by our coding patterns, and identify potential hot spots directly in the source code of that application. This in turn gives us the opportunity to resolve issues before we even submit our code for a code review.
Flaws vs. bugs
Runtime analysis identifies flaws, which are different from bugs. Bugs are usually the result of a logic error, such as forgetting to initialize a variable or causing a race condition. Flaws, on the other hand, are fundamental structural issues within the code. They pertain to problems with code design, code behavior when handling data, or deeper structural aspects of code implementation.
Benefits of runtime analysis in the dev environment
While there are monitoring tools available for the production environment, runtime analysis in the development environment offers three superior benefits:
Cost effectiveness
Production monitoring tools can be expensive, especially if your application has high traffic or complex architectures. Not to mention that it can be hard to tailor these tools for what you need and to leave out what you don’t.
Early issue detection
By analyzing your code at runtime during the development phase, developers are able to identify flaws and weaknesses early on before it goes to production or is seen by a customer. In a study published last year by Meta, they found that finding issues during the development phase makes them more likely to get fixed and makes the fix more likely
Seamless integration with the development workflow
With runtime analysis, you can deploy your application to different testing environments and stages, catching and addressing issues at various points in the development lifecycle. This enhances code quality and reduces potential risks.
Using AppMap for runtime analysis
AppMap performs runtime analysis within your IDE. Unlike static code analysis, which creates a model of your code's behavior, AppMap watches your code run, giving it the ability to reason about it much more accurately and definitively compared to static analysis.
To identify flaws, AppMap sources its default list of 12 rules from the OWASP top 10 found on the Common Weakness Enumeration (CWE) forum. CWE is a community-developed list of software and hardware weakness types governed by high-level organizations and companies with a focus on security, maintainability, and performance. Additional rules are available for use as well, and a full list of available rules and their definitions can be found in our documentation.
The cherry on top of this rule based runtime analysis is that it is fast. With static code analyzers, we have a wait time for the tool to parse code, build a model, and evaluate it. AppMap takes a slightly different approach. Since we can actually run your code we have the ability to dynamically instrument your code. Doing so provides additional insight into what is happening in your application, where and which processes our running. We flag certain events as they occur and add labels with additional information about what is actually happening in your application at the moment in time that the event occurred. AppMap’s Runtime Analysis Scanner then utilizes those flags and labels to identify potential issues or to spot patterns that may be problematic.
How to setup AppMap Runtime Analysis
After you install AppMap, record your application, and create your first AppMap we will add these default rules by creating or modifying the appmap-scanner.yml file. This extends the functionality of the scanner and allows you to include custom rules that suit your specific needs.
Our documentation also provides insights into how the rules are constructed, which labels we use in instrumentation, and how the scanner works with flags to match patterns and find flaws.
example 1: appmap-scanner.yml
checks:
- rule: authz-before-authn
# - rule: circular-dependency
- rule: deprecated-crypto-algorithm
- rule: deserialization-of-untrusted-data
- rule: exec-of-untrusted-command
- rule: http-500
# - rule: illegal-package-dependency
# - rule: incompatible-http-client-request
# - rule: insecure-compare
# - rule: job-not-cancelled
- rule: logout-without-session-reset
# - rule: missing-authentication
- rule: missing-content-type
- rule: n-plus-one-query
# - rule: query-from-invalid-package
# - rule: query-from-view
# - rule: rpc-without-circuit-breaker
# - rule: save-without-validation
- rule: secret-in-log
# - rule: slow-function-call
# - rule: slow-httpServer-request
# - rule: slow-query
- rule: too-many-joins
- rule: too-many-updates
# - rule: unbatched-materialized-query
- rule: unauthenticated-encryption
- rule: update-in-get-request
Results
Once a configuration has been created, you get all sorts of information about what possible issues may arise in your application at runtime.
Example 2: Analysis Overview
When run against an example application, we set our configuration to look for N+1 queries that are causing performance issues that are difficult to find using traditional static code analysis. We can see in the findings that not only did we find two N+1 queries, we found 69 other flaws in our application, 28 of which are potential security risks. In full transparency, this example application was built to be deliberately insecure so those security findings aren’t all that surprising, but what is surprising is the wealth of information available to us about the flaws that were found.
Understanding runtime analysis findings
When we drill down into one of these findings, we can see a link to the definition of the flaw with more information about it (ex. 3.1), the number of times it occurs (ex. 3.2), the stack trace to know how it occurred (ex.3.3), and how which “paths” a user could take to encounter that flaw (ex. 3.4). With these insights we are able to determine if a fix is necessary, where the fix needs to happen, and we can design and implement a fix before our feature ever goes into review.
Example 3: Findings
This presents us with real actionable information about just how negatively this flaw could impact the performance of our application. Imagine the cost and user interruption this would cause if it was found in production or, worse, not found at all.
Runtime analysis FTW 🏆
By leveraging runtime analysis in the development environment, you can save on costs, detect issues early, and integrate the analysis into your workflow. Using AppMap for your runtime analysis empowers developers to deliver better and more reliable software.
We love to hear what our users have to say. Let us know what you think in the comments below. Also, check out our AppMap integration for your IDE and other AppMap-related content at the links down below.
For an in depth look at the default rules check out this article: How to ensure your web APIs are Safe
Links
⬇️ Download AppMap for VSCode and JetBrains: https://appmap.io/download
⭐ Star AppMap on GitHub: https://github.com/getappmap
🐦 Follow on Twitter: https://twitter.com/getappmap
💬 Join AppMap Slack: https://appmap.io/slack
ℹ️ Read the AppMap docs: https://appmap.io/docs
📺 Watch AppMap Tutorials: https://www.youtube.com/@appmap
Top comments (0)