Announcing the Open-Source Reliability Leaderboard: A New Resource for Preventive AppSec

We are excited to announce the inaugural edition of the Mend.io Open-Source Reliability Leaderboard! Powered by data from Renovate, the wildly popular open-source dependency management tool, the Leaderboard presents the top packages in terms of reliability across three of the most widely used languages.

The Leaderboard allows the Mend.io team to leverage and share a valuable resource. There is no better arbiter of package reliability than Renovate, which has gathered crowd-sourced data on over 25 million dependency updates. By analyzing what packages are consistently releasing good updates, we built an accurate picture of a package’s overall reliability for software engineers trying to balance functional risk with the security risk imposed by our increasingly vulnerable software supply chain. 

“The Leaderboard helps shift the AppSec view from detection to prevention, a valuable perspective for reducing the risk imposed by our increasingly vulnerable software supply chain,” said Rhys Arkins, vice president of product management at Mend.io. “Success hinges on having the knowledge necessary to prevent possible open-source vulnerabilities from ever being installed in the first place. For that to happen, companies need to know not only what packages are in use at their companies, but how safe they are.” 

The full report showcases detailed rankings for npm, PyPi, and Maven.

Key findings:

Group runs bring down overall package reliability. 

Any fan of the TV show Survivor can tell you that in competition, groups are often hurt by their weakest link, and the same holds true when it comes to group updates. A group of ten packages is ten times more likely to encounter a failure. 

Release frequency has no effect on average success rates.

You would think that more-frequent releases would improve reliability through faster bug fixes and an engaged maintainer community, but this was not the case. 

The Best of the Best

Looking across the overall categories, the top three most reliable packages for each language are: 

Npm:

1. prettier-eslint
2. np
3. jest-cli

Maven:

1. org.apache.maven.scm:maven-scm-provider-gitexe 
2. com.github.ekryd.sortpom:sortpom-maven-plugin
3. org.apache.maven.plugins:maven-release-plugin

PyPi: 

1. Pulumi
2. Botocore-stubs
3. types-python-dateutil

Read the full report

• Check it out here