Mend.io Launches Inaugural Open-Source Reliability Leaderboard

New report offers valuable resource to help organizations evaluate the safety and reliability of open-source packages

TEL AVIV, Israel and BOSTON – June 28, 2023 – Mend.io, a leader in application security, released findings today from its latest report, the Mend.io Open Source Reliability Leaderboard. Powered by data from Renovate, Mend.io’s popular open-source dependency management tool, the Leaderboard presents the top packages in terms of reliability across three of the most widely used languages.

“The Leaderboard helps shift the AppSec view from detection to prevention, a valuable perspective for reducing the risk imposed by our increasingly vulnerable software supply chain,” said Rhys Arkins, vice president of product management, Mend.io. “Success hinges on having the knowledge necessary to prevent possible open-source vulnerabilities from ever being installed in the first place. For that to happen, companies need to know not only what packages are in use at their companies, but how safe they are.”

The Leaderboard allows the Mend.io team to leverage and share a valuable resource. There is no better arbiter of package reliability than Renovate, which has gathered crowd-sourced data on over 25 million dependency updates. By analyzing what packages are consistently releasing good updates, the Leaderboard presents an accurate picture of a package’s overall reliability for software engineers trying to balance functional risk with security risk.

The full report showcases detailed rankings for npm, PyPi, and Maven.

Key findings:

Group runs bring down overall package reliability. 

Any fan of the TV show Survivor can tell you that in competition, groups are often hurt by their weakest link, and the same holds true when it comes to group updates. A group of ten packages is ten times more likely to encounter a failure.

Release frequency has no effect on average success rates.

You would think that more-frequent releases would improve reliability through faster bug fixes and an engaged maintainer community, but such was not the case.

The best of the best

Looking across the overall categories, the top three most reliable packages for each language are:

Npm:

1. prettier-eslint
2. np
3. jest-cli

Maven:

1. org.apache.maven.scm:maven-scm-provider-gitexe
2. com.github.ekryd.sortpom:sortpom-maven-plugin
3. org.apache.maven.plugins:maven-release-plugin

PyPi: 

1. Pulumi
2. Botocore-stubs
3. types-python-dateutil

About the report

The report examines data from Renovate, an automated dependency management tool that leverages crowd-sourced data on over 25 million dependency updates.