What is Community Health?
Community is important to the success of open source software. The health of a community can be measured in different ways depending on the stakeholders involved. To understand and grow a community, project founders and maintainers need visibility into various technical, social, and even financial metrics. These metrics are important because they can help projects form data driven decisions. *But what metrics should we be using? *
This post is inspired by several discussions I have seen around the failings of current open source metrics and the tools used to measure them. This post will discuss what metrics we currently use and what I believe we should be using instead.
Current Metrics
On LinkedIn’s publishing platform Pulse, William Tracz wrote about Open Source vanity metrics and the problem with using them to measure a project's success. Tracz lamented they could not currently see a better way to gather valuable data about open source projects.
What are these current metrics?
Like Tracz mentions, many of the metrics open source maintainers and contributors rely on are considered “vanity metrics”. This term does not mean these metrics are not useful but they often do not give a full picture of a communities health.
Examples of vanity metrics in open source are:
- Number of GitHub stars
- Number of active commits
- Number of new and closed issues
- Releases
and more.
Often these are metrics founders and maintainers care about as they provide a way to quickly show “success” but may not actually help when it comes to important business outcomes and decisions.
Tools You Can Use to See Metrics
Orbit
Orbit is a tool you can integrate with GitHub to see metics like:
- Pull request opened
- Pull request merged
- Issue opened
- Issue comment created
and more.
As an open source community manager I used Orbit to quickly track new comments from contributors and maintainers so I could respond in a timely manner. I also used it to see when community members were no longer active. This helped me reach out and get feedback.
CHAOSS Metrics
CHAOSS is a Linux Foundation project focused on creating analytics and metrics to help define community health.
Common Room
Common Room is a growth platform that focuses on a business’ community. The platform helps companies connect their community engagement, product usage, and customer data through the use of artificial intelligence.
Bitergia
Bitergia is an analytics tool for software development teams, open source program offices, and InnerSource programs. Open source projects can use this tool to improve decision making and reporting by analyzing software development community, activity, and performance of open source projects.
Merico
Merico provides tools to help engineering teams understand and optimize their productivity and code quality.
What Metrics Should We Use?
“Tracking different types of metrics is essential for free and open source communities. Metrics give project insights into specific efforts and help get a feel of the community’s general perception. For that, tools that can pull data from various sources and develop a visualization of this data will help projects make informed decisions.”
Linux.com Editorial Staff, Measuring the Health of Open Source Communities
In the article quoted above and in William Tracz’s article the need for a tool that can pull more valuable data is apparent. But what data should projects be looking at and what tool could they use?
Beyond Vanity Metrics
What if you could see a project’s impact, identify potential customers, and understand/showcase growth without sacrificing your end-user’s privacy?
Scarf Gateway helps you understand how users are interacting with your open-source project at every step of their journey
Instead of answering questions like “how many followers do we have”, Scarf’s data digs deeper in to a project’s community health and answers questions like:
- How many new companies tried our OSS this month?
- How many unique users did our OSS have this month compared to last month?
- Was the spike in downloads we saw last week genuine growth, or a thrashing CI pipeline from an existing user?
- Should I actually spend the time to fix the bug for that one Windows user?
We can also have visibility into how OSS software is distributed.
Instead of wondering who is using our software, Scarf’s data answers questions like:
- What percent of our user base has adopted our latest patch?
- What cloud providers are the most popular with our users?
Scarf can also help uncover community metrics like your projects discoverability, usage, churn, and commercial reliance. Community Managers can use this data to proactively connect with users and grow a healthy and vibrant OSS community.
This data compliments the metrics you can get from tools like Orbit and answers questions like:
- What is the geographical breakdown of our user-base?
- What percentage of our user-base is corporate usage vs hobby usage?
- What time zones should we cater to with our online event planning?
For a project’s documentation it can answer:
- What percent of our user-base is on Windows vs Linux vs Mac?
- Which section of our docs are users finding most confusing?
A New Way
This type of rich usage data moves beyond the typical vanity metrics the open source community has typically relied on. While those metrics can tell you a lot about a project, they do not give the full picture of a community's health.
With the addition of tools like Scarf Gateway, maintainers now have the power to make more informed decisions. When maintainers have hard data around who, where, and why companies and individuals are using their software they can better prioritize features, patches, releases and more.
Top comments (0)