Consideration of OSS Activity Metrics Based on GitHub Repository Data

Hey Devs!

Recently at ROUTE06, Inc. we published two projects, Giselle and Liam, as OSS.

• https://github.com/giselles-ai/giselle
• https://github.com/liam-hq/liam

As we move forward with our OSS activities, it is important to define and measure key metrics in order to understand project growth, user engagement, and identify areas for improvement.

This article will explore metrics used to measure results in OSS marketing, how they correlate, and the range of metrics that can be obtained.

• Type of Metrics
  • 1. Awareness
  • 2. Interest
  • 3. Consideration
  • 4. Action
  • 5. Adoption
• Funnel structure and close but different correlations
• Range of metrics that can be obtained
  • 1. Awareness
  • 2. Interest
  • 3. Consideration
  • 4. Action
  • 5. Adoption
• Conclusion
• Supplement: About Giselle and Liam as OSS

Type of Metrics

We have identified the following five categories of metrics for evaluating the success of OSS marketing efforts on GitHub. Through these metrics, we will track user behavior from project awareness to usage and determine the effectiveness of our marketing efforts.

1. Awareness

These metrics are used to determine how well a project is perceived by its users.

• Views
  • The total number of visits to the repository, counted as multiple visits by a single user
• Unique views
  • The total number of unique visits to the repository, counted as one visit, even if one user visits multiple times

2. Interest

These metrics show the number of users interested in the project.

• Stars
  • The total number of stars users have given the repository. Each user can give only one star
  • The number of prospective users, indicating support for and interest in the repository
• Watchers
  • The total number of users watching the repository
  • The number of prospective users interested in repository updates

3. Consideration

This metrics represents users who are specifically considering using or participating in the project.

• Clones
  • The total number of times users have cloned the repository
  • The number of prospective users who will review and use the contents of the repository in detail
• Forks
  • The total number of repositories forked by users on GitHub
  • The number of prospective users who will make their own changes to the repository, create pull requests, learn, and use for reference

4. Action

These metrics are indicators of actual project participation and contribution.

• Issues by outside collaborators
  • The total number of prospective users who would like to contribute to the improvement and development of the repository by reporting bugs, requesting new features, etc.
• Pull Requests by outside collaborators
  • The final step of the action phase. The number of prospective users who are highly motivated to grow and improve the quality of the repository and have the technical skills and knowledge to do so

5. Adoption

This metric shows the actual usage of the project. Only for projects such as npm packages.

• Number of repository downloads
  • The total number of potential users actually using the repository

Funnel structure and close but different correlations

The above indicators represent user behavior from awareness to usage, a flow similar to a funnel structure.

Awareness
↓
Interest
↓
Consideration
↓
Action
↓
Adoption

However, due to the nature of OSS projects, they have a different correlation to the general marketing funnel.

• The progression of stages is not linear
  • Users do not necessarily progress from awareness to usage in a sequential manner. For example, some users may download and use a product immediately after learning about it in the awareness phase, and sometimes they may skip steps in the funnel
• Cyclical relationships
  • OSS projects are more cyclical than traditional funnel structures, as users may return to the interest and action phases after using the project. In many cases, users provide feedback to the project and add new stars, so there is a reciprocal relationship
• Interdependence of indicators
  • The indicators at each stage are interrelated. For example, as usage increases so do awareness and interest. Therefore, in OSS marketing, it is necessary to be aware not only of fluctuations in individual indicators, but also of the overall balance

Range of metrics that can be obtained

There are some limitations to the metrics that can be obtained on GitHub to measure the marketing performance of OSS projects.

1. Awareness

• Views
  • Can be obtained via the count property of the REST API Get page views (Only available for the last 2 weeks)
  • Similar figures are available from Traffic in the repository Insights tab
• Unique views
  • Can be obtained via the uniques property of the same REST API (Only available for the last 2 weeks)
  • Similar figures are available from Traffic in the repository Insights tab

💡 Visits by you and Organization members also appears to be counted.

• Does the GitHub traffic graph include your own views? - Stack Overflow

2. Interest

• Stars
  • Can be obtained via the REST API List stargazers. It is also possible to obtain who starred and when for the entire period of time.
• Watchers
  • Current Watchers can be obtained via the REST API List watchers

3. Consideration

• Clones
  • Can be obtained via the count property of the REST API Get repository clones. The number of unique clones can also be obtained from the uniques property. (Both are only available for the last 2 weeks)
  • Similar figures are available from Traffic in the repository Insights tab
• Forks
  • Can be obtained via the REST API List forks. All forks can be obtained for the entire period, including when and by whom they were forked
  • Similar figures are available from Forks in the repository Insights tab

💡 For Clones, the number of actions/checkouts, etc. executed in GitHub Actions also appear to be included.

• Do GitHub Actions contribute to a repo's Traffic stats under Insights? · community · Discussion #23806

4. Action

• Issues by outside collaborators
  • Can be obtained via the REST API List repository issues. You can also obtain who created the issue and when for the entire time period
• Pull Requests by outside collaborators
  • Can be obtained via the REST API List pull requests. Pull requests can also be obtained for the entire period, including who made the pull request and when

5. Adoption

• Number of project downloads
  • This metric is difficult to track on GitHub, so download counts from package management systems (e.g. npm and PyPI) are commonly used as a reference

Conclusion

We have looked at various indicators of OSS activity from GitHub repository data.

As mentioned above, Views, Unique views, and Clones are only available for the last two weeks. In the next issue, we will show how we solved this problem.

Supplement: About Giselle and Liam as OSS

The giselles-ai/giselle repository recently published as OSS is for Giselle, a platform that enables no-code creation of agents and workflows utilizing generative AI. Detailed information can be found on the Giselle service website: https://giselles.ai/.

The liam-hq/liam repository is for Liam, a tool that effortlessly generates beautiful and easy-to-read ER diagrams. Detailed information can be found on the Liam service website: https://liambx.com/.