Perforce Source Control for game development

This is part of a series of posts on source control for game development. Read more in the Blog.

Perforce is the most widely used source control solution for game development studios.

At Darewise, we evaluated many options and finally opted for Perforce, much like the rest of the game industry, or at least those who can afford it. Pretty much all major AAA studios, and medium to large teams are using Perforce as their main source control, so there must be some valid reasons behind this choice. I have personally used Perforce in every single one of my professional projects. I have worked primarily with workflow issues and source control, so this is a product I know very well and I might be slightly biased towards it.

In the previous article, I explained why Git is not the most optimal source control solution for game development. This time, let's examine why is Perforce so popular.

Background and History

According to wikipedia, Perforce (the company) was started in 1995 with its first product being the eponymous Perforce (the source control solution). The goal was to provide a system allowing companies to keep track of changes in large projects containing both source code and binary files. This already sounds interesting to us.

Perforce went through a few branding (mis)adventures as the years went by. I still remember its slogan being "the fast source control". Later, the company was bought and more products were added over the years to compliment the source control. These days the official branding for the source control product is "Helix Core", but in my heart and for this article, we will still refer to it as Perforce.

As stated on the official website, Perforce focuses on security, managing binary artifacts, performance at scale and teams distributed globally. They also claim that 19 out of 20 game development studios are using Perforce. While this claim is hard to verify, in my experience this is true when it comes to medium to large teams. In fact, I do not know of any sizeable studio not using Perforce.

On their officially disclosed list of customers, you can find Perforce is used by major game studios. EA, Ubisoft, Epic, Capcom, Crytek, CD Projekt Red just to name a few.

Microsoft famously maintains (or maintained?) a fork of Perforce called Source Depot. This was used notably for the Windows repository, which has now been migrated to Git, but they could only achieve this incredible feat after creating VFS for Git (previously called GVFS). This shows how much effort was necessary to adapt Git to handle as much as Perforce could.

Perforce has a history of being successfully used on large projects and by reputable companies. Some are migrating away after massive engineering efforts, others like the game industry are sticking with it.

Features

Perforce is packed full of features and I'm still discovering some of them. Let's go over the main points.

Centralized source control

Game development is at its core a centralized effort. Everyone needs to collaborate on the same project and be allowed to change it and receive updates as fast as possible.

Where open source leverages the distributed model to make up for a lack of trust, game development projects are most likely comprised of people trusting each other to work in a central repository. The only really interesting part of the distributed workflow is evident for programmers: the ability to create commits and branches which are only on your local machine and are not shared with everyone. This is great for programmers working on several features at the same time, amongst other things.

Perforce is often criticized for a lack of decentralized workflow, even though it provides several tools for it. The most basic tool is shelving which is the equivalent of Git stashes. I will agree that it is mostly a workaround, and feels lackluster after using Git for a while. However, Perforce does, in fact, have a decentralized option called DVCS. This works in command-line in a similar way to Git and has recently been added to the P4V client. Unfortunately, not many people know or use DVCS, so let's hope this article gets the word out.

Changelists and per-file versioning model

Each change in Perforce is grouped together into a Changelist. A changelist is a list of files, their revision numbers and the operations performed on each file.

Each changelist is given a unique incremental number. This makes it very easy to understand which change precedes or follows another. The number is unique on the entire Perforce server, even if you have multiple projects. In fact, one changelist can theoretically contain files from multiple projects, although I would not encourage you to do this often.

The change model is also per-file. While each changelist is a collection of changes, each file has a history with incremental revision numbers. Again, this is clear and easy to follow, as opposed to git's object model and commit hashes.

What's more, your workspace does not need to be up to date with every single file when submitting a changelist. You only need to be up to date with the files you are planning to change. This means you can iterate much faster than Git, especially when dealing with many large files in a game project. However, this also means your workspace may not be in a fully working state, as some of your files are up to date while others may not be. In practice, this is not as big of an issue as it sounds, you can easily synchronize your workspace to a specific changelist, or more frequently the latest change, when a problem arises with your workspace. As you don't have to sync so often, your iteration time is increased.

The branching model of Perforce is not as simple as Git's. Since each file is versioned independently, branches are performed at a file-level and are harder to visualize when looking at the big picture.

In the past, working on several branches was cumbersome and required the creation of a new folder on your hard drive. Since the introduction of streams, you can easily switch branches within the same folder, as you do with Git. On top of that, streams formalize the branching workflow by enforcing which files belong to which branch and controlling how you can integrate or copy from one branch to another, at a granular level. This is great because your guidelines are present in the tool rather than simply by convention, which helps avoid human error.

Large project support

Perforce puts the accent on performance when handling large files. This is evident in all of their communication and can be verified in practice. Storing and retrieving files is fast, especially if you are hosting your server on-premises.

On top of that, Perforce provides multiple solutions to allow geographically distributed teams to work together more efficiently. You can use basic caching proxies and read-only replicas, all the way to a network of "edge" servers that act as your local server and ensure eventual consistency with the master. This, combined with enterprise-grade networks and bandwidth, allows large organizations to work around the world on a centralized repository. Of course, setting up such an infrastructure involves massive costs as well as constant maintenance.

Another key enterprise feature is the integration with LDAP or Active Directory for user management, as well as the myriad of access control options to restrict access in all kinds of way. Large organizations can, therefore, use a single Perforce infrastructure for all of their projects while maintaining the security and confidentiality of their data. Additionally, Perforce can be configured to use the SSL protocol for greater security.

User experience

Command-line

For programmers and advanced users out there, Perforce provides a command-line interface. Having used it for years, I can say for certain that it is very well designed, consistent, easily understood, very powerful and very well documented.

When compared to Git and the criticism I emitted in the previous article, it's night and day. Perforce's command line is almost intuitive in its design and has remained very stable over the years. This has made my job easier when integrating Perforce in our automation systems or in various game engines in my career, where updates to the Perforce client or server rarely if ever require code changes. There are a few subtle issues that experienced users will eventually run into, but they are very minor inconveniences in practice, and Perforce's command line is a joy to use.

Graphical User Interface

This is where things get interesting. Perforce ships with P4V, the official a graphical client for Perforce and the only one worth mentioning. This is one of the biggest differentiating factors, as Perforce has always put emphasis on user experience especially for less technical users. One of the major arguments for the adoption of Perforce against other solutions is P4V.

Now, let's be honest, this is source control and it can never really be trivial to use, so there is a bit of a learning curve, but it is my personal opinion that P4V is the best source control GUI bar none.

For beginner users, it intuitively presents the file tree of the local workspace and the remote repository, provides basic functionality such as showing the history of a file or a directory, organizing your changelists and of course submitting your changes. This means non-technical users will never have to see or use the command-line. The same can't be said for most competitors.

For power users, it is a fully-featured Perforce client that can perform almost all the tasks you will ever need in practice. In fact, as a very experienced Perforce user and administrator, I rarely have to use the command line. I can perform complex queries over the files and the history and visualize all of the results in very well designed tools. With the administrator GUI, I can perform the most regular administrator tasks such as creating and managing my users and groups. The only time where I really need the command line is to perform complex and unfrequent server configuration or maintenance.

Here are some of the most advanced tools which don't have equivalents elsewhere.

Revision Graph: Easily visualize and follow the changes and integrations of a file. This is similar to visualizations of a Git branching tree, except it only shows you the evolution of a file which is often more useful in practice and lacking in Git interfaces.

Time-Lapse View: The most useful of them all. This is git blame on steroids. It allows you to browse a file and see for each line who was the last person to edit it, just like git blame. However, you can also scroll through time! Using this tool, you can follow a line and see the surrounding context evolve with revisions. You can quickly and easily understand how some code came to be, and when appropriate, blame the right person for their mistakes. The attempts to replicate this tool for Git still have a long way to go before they reach the same efficiency.

P4V is certainly not perfect, and Perforce development teams are releasing improvements and updates on a regular basis. What I would like to address here is the common belief that Perforce's user interface is difficult to handle. Those complaints often come from non-technical people who don't have a lot of experience with source control and find the core concepts confusing. However, I have not encountered a better graphical tool to handle source control than P4V so far. The concepts are daunting at first, but once understood, P4V allows you to efficiently manipulate them. I would be interested, of course, in knowing your opinion on this, but ultimately this is a tool that has been proven to work in real-life situations. Even the less technical minded artists and designers end up learning and understanding enough to work with it and they do get over the initial pain. In fact, the reputation of having a bad UX probably comes from not having been presented with the alternatives. That's not to say that UX of source control software cannot be improved further, but this is almost as good as it gets in 2019.

Merge tool

If all of this wasn't enough, Perforce also maintains and distributes a free diff and merge tool: P4Merge.

This is, of course, included and well-integrated with P4V, but it is available for anyone to use. It is one of the better diff and merge tools out there, and includes most of the features you'd expect. It is easy to use and consistent with the other tools from Perforce. As an architect and today a CTO, you can trust me on the fact that I have performed a lot of very complex integrations using many different source control solutions and setups. For all of those, I have been using P4Merge and still do to this day.

I have seen some people prefer Beyond Compare or Araxis Merge over it, but those aren't free and I never really understood what makes them so much better. I'd love to hear some compelling arguments. There is, in fact, a very promising and much more interesting merge tool I'm looking at these days, but we'll get to this in another article.

Useability for game development

Handling large projects

Perforce can handle source control for game development projects of any real-life scale. It performs reasonably well even with huge projects containing terabytes of data, hundreds of thousands of files and hundreds of users.

I have some personal experience with Ubisoft's Perforce infrastructure which is probably one of the most complex in the industry, and it undeniably does the job. Teams are building on top of Perforce to improve and optimize where needs be: transferring large amounts of data across the globe can be slow, but this becomes a rich man's problem when reaching such scales. For reasonably large teams, using Perforce out of the box should cover all of your collaborative needs.

Setup and administration

Perforce is complex software to deal with as an administrator. Where you can start using Git simply by typing "git init", for Perforce you'll have to take your time and learn.

The initial investment is significant: you will need hardware to host your server and learn to set it all up. Perforce can be hosted on Windows or Linux but for many reasons Linux is preferable, and this is once again not beginner-friendly. You also need to figure out a backup and restore strategy, to plan for the worst. You do not want to lose your data, as this is your most valuable asset.

Perforce requires configuration to fine-tune to your needs. You should configure the authentication, typemap, depots and streams before you start working. Sounds like a lot, but the upside is that you can get exactly what you want out of it. For more customization, you can create triggers, the equivalent of Git hooks, and server-side extensions in Lua. On the client-side, you can create plug-ins for P4v, or use the client APIs with C++, Perl, PHP, Python, Ruby, Java, Groovy and perhaps more.

While the learning curve is steep, Perforce allows you to truly implement your desired workflow down to the specifics. This can be extremely valuable over time and especially in larger organizations.

Tricky operations

Branching and merging operations are harder than on Git. This is generally reserved for power users, even with experienced teams. Branching can be slow and error-prone, and the merge engine is not as efficient as Git's. Some specific cases are very delicate to handle: those of you who have integrated moved and modified files will know what I'm talking about.

You will also encounter limitations of P4V one day or another. It performs extremely well on regular operations, but when dealing with several thousand files, you are better off using the command-line. Again, those operations should be unfrequent in most game projects and generally reserved for power users.

Perforce will be a great tool to leverage, provided you have technical people to support it. For small teams with artists and designers only, this could be daunting. For most serious game development projects, one of the programmers will be capable of taking this responsibility.

Ecosystem

Integrations

Perforce (the company) provides an array of products on top of the source control. Each one of them has its own licensing model. While some are free, some others require a separate paid license.

One of the most relevant for us is Helix Swarm, which is a free code review tool for Perforce. This is a great addition to the toolbox, and an indispensible one if you are serious about programming. Swarm is well integrated with Perforce and provides enough features to do the job. Just be aware that it is not the best tool on the market. Here is a list of alternatives. I hear that Perforce is planning for an update of Swarm, and I can't wait to try it out.

Because it is used everywhere in the game industry, Perforce is also well integrated with game engines. Unreal, Unity, Cryengine, and every proprietary engine I've worked on had a Perforce integration of some sort.

Perforce is also integrated with IDEs, most notably Visual Studio with its official plugin P4VS.

Of course, Perforce also has integrations with all of the major Continuous Integration solutions. For instance, they maintain and distribute the official Jenkins integration. This is what we use here at Darewise.

There is a Slack integration, sadly it is part of a separate Perforce product. A little research will tell you if your favorite tool has an Integration, but don't count on it.

Interoperability

In recent years, Perforce has fully embraced Git as an attempt to increase interoperability with Perforce source control as well as their other products. They provide several tools to that effect:

• git-fusion: a legacy python script that allows two-way copies between a Git repository and Perforce server. While it is no longer listed on the website, it is still in use and I don't think there is a replacement for this particular use case. git-fusion is, for instance, how Epic Games and Cryengine provide a public Git endpoint, even though they work on Perforce internally.
• Helix4Git: a Git server available on-premises or in the cloud. The unique selling point being, of course, its performance as it is backed by a Perforce server, presumably at least the same database engine (oracle). This allows for hosting both Perforce and Git repositories on the same server.
• Helix TeamHub: a full project management solution including hosting the repositories, access control, kanban boards. It even supports SVN and Mercurial. Basically this is the equivalent of GitLab by Perforce, although it lacks the DevOps/Continuous Integration which makes GitLab so popular.

Hosting

Simple hosting solutions for Perforce are rare. Most teams choose to host on-premises using their own hardware or public cloud options. For managed solutions, it seems that Assembla is the exclusive provider of cloud hosting.

Overall, the ecosystem of Perforce is smaller than that of Git, as expected. That being said, all of the major points are covered: everything necessary for you to perform your daily tasks is available and pretty high quality. If that's not enough, there are several P4 to Git options to choose from.

Pricing

Perforce sells itself as an "enterprise" software and comes with an "enterprise" level price. In other words, Perforce is expensive.

You can use Perforce for free if you host it yourself. It is however limited to 5 users and 20 workspaces. In practice, most people will exceed this limitation. Each user needs to create several workspaces, at least one per workstation. It is recommended to create "system" users such as an administrator user, and probably one for your swarm and for your CI system. Of course, you can ignore these recommendations but the free tier remains very limiting.

If you require more, you must negotiate a licensing agreement with Perforce, with an undisclosed price and licensing model. While I am using Perforce myself, I am not sure I am at liberty to disclose our agreement. We'll just stick to public information to be safe.

For cloud-hosted servers with Assembla, the pricing comes in two tiers. The basic price starts at $44.50/user/month with a 12-month commitment. Moreover, extra features, as well as single-tenant hosting, can be negotiated with a custom pricing agreement.

We can safely assume all the undisclosed pricing is at least the same if not more expensive, which makes Perforce at least $534/user/year. This is before you factor in hardware and maintenance costs if you choose to self-host, which many studios do for greater control and data security. This is a large investment for most teams, and it is the most expensive source control solutions of all we are going to be looking at in this series.

I would love for Perforce to reconsider their licensing model to allow for more teams to access the software. This is the backbone of the game industry and it would surely benefit from a larger ecosystem. I'm sure many small teams would be inclined to switch to Perforce if they were offered a reasonable monthly model per user. The free tier limitations could also be relaxed to allow for small teams to use it properly, but it would require some development on Perforce's side due to how the license system is implemented. Let's keep out hopes up and see what the future holds.

Conclusion

Perforce is a great source control option for game development if you have the means to afford it, both financially and technically. For newer or smaller teams, it can be scary to get into, not to mention expensive.

It is still the undisputed champion of the game industry and the workhorse of all the major studios. It was adopted for good reason, as its ability to handle large projects was unrivaled, and this is the biggest issue facing game development projects, far before user experience or interconnectivity with the open-source ecosystem. However, the game industry has also accumulated enough legacy on top of Perforce that even if there was a better solution available today, the adoption would be slow due to the ever-increasing costs of migration.

Perforce is constantly improving in the meantime. There are several yearly releases of the server and the client, as well as the satellite products. A lot of these updates are going in the right direction, improving the user experience, adding features to support our workflows and interoperability with more systems. The company is clearly investing and consolidating its ecosystem to cover all grounds of software development, from source control to application lifecycle and project planning with more recent acquisitions. I believe it has a great opportunity to increase adoption by proposing new business models to attract new customers, and I will certainly keep an eye out for major updates.

In the meantime, newer teams should aim to start with the best tools available presently and within their budget. Which is the best fit is certainly not obvious, this is why I'm writing this after all. As an industry, we should never stop looking for better solutions, challenge our assumptions and established processes in order to improve our workflow. Making games is one of the most challenging tasks in software and art development of our time, and every improvement to our quality of life is worth it.

In the next article, we will look at a challenger that could rival Perforce on its own territory: Plastic SCM.