Automating Bulk Transfer of Mercurial Repos to Git

With Mercurial sunsetting support, it was time to move our repos. But how do you do this en masse?

When we received the first communication about Bitbucket sunsetting Mercurial support, there was quite a lot of excitement around the office. All of our source code was stored as Mercurial repositories on Bitbucket and – being a product company – a lot of the company’s value is in the source code.

A closer look at the problem presented us with the following challenge:

• 350 Repos across 5 teams to convert
• Retaining the commit history for all projects
• All branches need to be converted

A solution was needed, but many searches and discussions later, nothing suitable was found. So, we decided to wait. There was still plenty of time and we trusted that a solution would surely present itself sooner or later.

However, when the reminder from Bitbucket arrived early in 2020, no solution had as yet presented itself and time was starting to run out. A few searches later we found the fast-export converter by frej, a python project on Github that we could use to convert our repos one-by-one. While each individual migration seems simple enough, our challenge was the fact that we had to do this in bulk.

The challenge to do this in bulk was a great use case for Linx. Linx excels at the rapid development and deployment of back-end automations. With its visual models, it accelerates the development process for developers to quickly build unattended automations without having to worry about setting up or scaling their backend infrastructure and the related services.

We decided to create an automated process that would perform the entire conversion against each of our repos without any intervention.

We decided to create an automated process that would perform the entire conversion against each of our repos without any intervention.

The Process

Within 3 days we developed and tested our Linx solution and it works great. The final solution automatically performed the following steps:

• Calling the Bitbucket API to fetch a list of hg repos and writing the info to a CSV file
• Cloning each hg repo in the file
• Converting each hg repo to git
• Creating the git repo on Bitbucket
• Pushing the git repo to Bitbucket

The only limitations we are aware of is that closed branches and issues are not migrated. It seems that the closed branches concept does not exist in Git and we don’t use the Bitbucket Issues feature. However, if you need Issues, building the migration of these into the Linx solution should be quick and easy. Bitbucket has an Issue Export API call that you can implement in Linx to accomplish this.

Results

Once ready, each of our teams would need to run this solution themselves as the conversion needed to fit into each team’s own release cycle. So, the process each team followed was:

• Everyone checked in all changes at the end of conversion day
• We ran the Linx process overnight or over a weekend
• In the morning, the teams pulled the new repos and just continued where they had left off

Some conversions took a long time to run, but for most teams have completed their conversions now and development has continued smoothly and without interruptions to date.

Some of our results were pretty impressive;

• Longest conversion process took 12 hours to run
• Oldest project had commit history going back as far as 2006
• Largest project has over 500MB of source code
• Over 300 company and over 80 personal repos converted

Interested in trying it?

The solution is freely available for anyone to use. Download the Linx Designer and use the sample provided.

The post Automating Bulk Transfer of Mercurial Repos to Git appeared first on Linx.