Container Security: Creating an Effective Security Program with Reachability Analysis
Containers have taken over the world of software development. According to Gartner analysts, “90% of global organizations will be running containerized applications in production by 2026,” up from 40% in 2021. Containerized applications provide enterprises with an agile, modern approach in the age of cloud computing; safeguarding these technologies from existing and future threats requires equally modern methods.
One significant container security challenge stems from the common use of publicly available open source images. Developers typically pull container images with known vulnerabilities, or do not update container images fast enough to remediate CVEs, leading to significant security backlog. Software companies that build and operate containers spend thousands of hours each year, if not more, on vulnerability management. However, identifying, triaging, and patching CVEs found in your container images is a lengthy, cumbersome process that introduces a lot of toil and friction for organizations that need to move fast and ship new products and services to customers.
Conducting robust and accurate CVE triage requires dedicated staff and a significant time investment, translating into higher operational costs and potential delays in addressing critical vulnerabilities. Teams often become inundated with these tasks, which distracts them from more important tasks like product and business innovation.
Reachability: A New Security Paradigm
Reachability has become a must-have feature for application security scanners because it helps separate exploitable risks from false positives. Reachability analysis takes vulnerability prioritization to the next level. It identifies relationships between software functions, so you can understand open source library vulnerabilities in their real-world context. By linking dependencies, you can trace forward through application code to see if an attacker could potentially access a given software flaw. Reachability can answer questions like:
- Is my code actually invoking this library with vulnerable code within it?
Reachability analysis lets you smoke out those vulnerability scanner findings for issues that pose minimal risk in your product or network. It also enables you to evaluate the safety of library upgrades and the elimination of unnecessary dependencies.
Effective Reachability For Container Scanning1
Before analyzing the needed reachability technology for container scanning, it’s helpful to differentiate between software composition analysis (SCA) and container scanning because vendors use the term “reachability” for both types of scanning. These are distinct but commonly confused tools.
SCA
SCA is a tool that analyzes open source code (from other repositories, which are most commonly public but can also be internal dependencies) that have been imported by a developer and used in applications. The purpose of an SCA tool is to create a Software Bill of Materials (SBOM), surface potential vulnerabilities and risks with those OSS packages, and provide remediation recommendations.
Container Scanning
Container vulnerabilities affect libraries that are installed onto the operating system of an image. Similar to an SCA tool, the container scanner looks for vulnerabilities in those imported images. However, given the nature of containers, which includes the operating system (OS) layer with the application code, the amount and the severity of the discovered vulnerabilities is significantly higher.
For example, let’s take a look at Redis, a popular open source key-value store that functions as a data structure server. Scanning Redis source code (https://github.com/redis/redis) reveals the following results:
| Packages | Dependencies | Total CVEs | Severity |
| 2 | 11 | 7 | Medium (2) Low (5) |
Scanning Redis’ official container image (https://hub.docker.com/_/redis) reveals:
| Packages | OS | Total CVEs | Severity |
| 15 | Debian 12.5 | 280 | Critical (1) High (46) Medium (91) etc. |
Examining the results reveals that almost all CVEs are coming with the Debian OS (version 12.5) and that without this layer, the accumulated CVE results are minimal (7 CVEs).
Similar results can be seen with other popular open-source container images like Postgres. When scanning its source code (https://github.com/postgres/postgres) you can’t detect any known vulnerabilities. However when scanning its docker-hub official image (https://github.com/docker-library/postgres) build on Debian OS, you find more:
| Packages | OS | Total CVEs | Severity |
| 15 | Debian 12.5 | 153 | Critical (1) High (25) Medium (25) etc. |
Here, too, all CVEs come with the OS level.
But it’s not just Debian OS that adds multiple vulnerabilities to container images. Let’s take a look at another popular CNCF project called Buildpack using Ubuntu OS. Scanning its source code (https://github.com/paketo-buildpacks/go) reveals the following results:
| Packages | Dependencies | Total CVEs | Severity |
| 4 | 2 | 4 | Critical (2) |
Scanning its official container image (https://hub.docker.com/r/paketobuildpacks/build) shows the following:
| Packages | OS | Total CVEs | Severity |
| 15 | Ubuntu 18.04 | 375 | Critical (2) High (15) Medium (270) etc. |
Adequate Reachability for Containers2
Let’s examine the different reachability technologies to assess which one has the best results for container-security scanning.
Function-Level Reachability (SCA)
An SCA tool with source-code reachability analysis scans your code to check if the vulnerable library is being used in a way that can be exploited by the detected CVE. It can trace entire application contexts to verify if the vulnerable circumstances can happen outside of simple code execution flow. With function-level reachability, if CVE exists in the code but can’t be exploited, then the CVE is a false positive that you don’t need to send for remediation.
There are some inherent challenges with this methodology, because multiple CVEs don’t contain the function-level data needed for the reachability analyzers, and some CVEs require the general application context to validate if an exploit is possible. Nevertheless, reachability in the SCA context provides a great solution for application source-code analysis.
In container images, the vulnerable libraries are installed onto the operating system. SCA reachability doesn’t cover the OS layer and consequently does not identify the vast majority of CVEs reported for container images.
Consequently, it’s not suitable for container security.
Package Used in Image (Container)
A common method to detect reachability for container image vulnerabilities utilizes runtime methods. Equipped by technologies like eBPF, modern runtime security tools extract the loaded code packages when the container is running; with runtime assessment, users can identify whether the packages (not functions) are being used/loaded and thus prioritize the CVEs of these packages.
Examining the previous Redis container example deployed in AWS EKS cluster, we see the following results:
| Packages | OS | Total CVEs | Reachable (in use) |
| 15 | Debian 12.5 | 280 | 7 (initial phase) 15 (later phase) |
A first glance shows great results: between 7-15 CVEs are in use while the container is running, no need to fix the other 265 CVEs.
However, a deeper look into the result reveals fluctuations in the number of reachable CVEs, which is based on the actual usage of the container while it’s running. This method is actually a runtime snapshot of the loaded packages, which might change based on the container usage. Hence, there’s a potential for CVEs to be missed with the “in use” methodology that captures current usage status.
There are also some practical aspects that might limit the “in use” methodology. A common argument is that when the container is deployed, it’s too late to identify which code parts need to be changed/modified due to code vulnerabilities. Additionally, the cost implications of addressing security issues in a running application, rather than earlier in the development cycle, cannot be overstated. Remediation at this stage often entails significant operational disruptions and resource allocations, as fixes may require rolling back production environments, comprehensive system audits, and extensive testing to ensure that patches do not negatively impact functionality or introduce new vulnerabilities.
Runtime Prediction — Container Images Reachability
To bridge the gap between function-level reachability and package-in-use for running images, one may use a static reachability analysis for container images. This method introduces an algorithm that analyzes the compiled code and configuration files in the container’s file system and indicates which of the packages can be used at runtime, and which are considered as “dead code”. In contrast to runtime solutions, this analysis can be used in the classical shift-left (pre-production) scanning points of containers images (e.g., CI or image registries) or in Git repos (with adequate build option). It requires no additional setup, and it is much faster and more efficient than capturing data from running environments in production, a process which usually takes a few days to get accurate.
There’s also an often-overlooked value in this case. Because SCA and container teams are sometimes separated, AppSec managers may find themselves in between SDLC stages, just to figure out way too late (in production) or too early (code without the OS context) about security risks and their actual reachability status. Looking back at the Redis example with runtime prediction analysis, we get an 84% reduction before deploying anything to our running environment.
| Packages | OS | Total CVEs | Reachable (in use) | Reduction Rate |
| 15 | Debian 12.5 | 280 | 43 CVEs (no Critical or High) | 84% |
Conclusion
Reachability has become a must-have feature for open-source vulnerability detection. It identifies the exploitable risks, separating them from false positive alerts. Reachability analysis allows efficient prioritization by identifying relationships between code components and understanding library vulnerabilities in their real-world context.
In container images, reachability-based prioritization is crucial, as the amount of detected CVEs is typically skyrocketing. However the classical reachability used in SCA isn’t adequate for container security usage, it requires a reachability analysis that addresses the OS level CVEs.
1The following security scanning results were done by Mend’s SCA and Mend’s Container solution but were compared to other commercially available and open-source scanners that reveal similar (but not identical) results.
2The reference for this section is from the remarkable research work by James Berthoty (5 types of reachability analysis: https://www.endorlabs.com/learn/5-types-of-reachability-analysis-and-which-is-right-for-you).
