AI-Powered Pull Requests: CodiumAI vs GitHub Copilot

In the past year, AI has become an essential part of developers’ lives. Back then, it could only write simple code snippets, now it can analyze and refactor whole codebases. With the rapid development of AI, we’ve raised our expectations and we’re now waiting for something even better.

With our eyes on the horizon for the next features in Copilot X, a representative from CodiumAI reached out to me, suggesting the creation of an article showcasing features that we can try out right away. Particularly an AI copilot for writing and reviewing pull requests.

Imagine there was an AI copilot that could help in describing PR changes, reviewing them, and suggesting improvements to them. The life of many open-source contributors and maintainers could be so much easier. Well, CodiumAI PR-Agent is exactly that tool!

Overview

Alright, so here’s how it works. Add a comment to any public pull request on GitHub with the following text:

@CodiumAI-Agent /review

After ~30 seconds you’ll see a comment from CodiumAI-Agent with bullet points about the PR (such as theme, summary, type, estimated effort to review, etc.) and some feedback on it.

In the same way, you can replace /review with one of many other commands to get various descriptions, suggestions, and feedback from the agent.

Write better PR descriptions with the help of AI

Find me a developer who had dedicated a substantial amount of time to implementing a feature or resolving a challenging bug and then didn’t feel want to finally send the code to production. In that moment of exhaustion, all you want to do is hit the merge button and take a well-deserved break.

Although it’s quite understandable, this often results in PRs like:

Fixed some things

I made some changes to improve the data processing in the data_handler.py file.

It doesn’t provide any specific details about what improvements were made. The pull request lacks context about the motivation behind the changes. Neither does it mention whether the changes have been tested.

Both GitHub Copilot and CodiumAI aim to fix this problem by generating PR descriptions with a summary and a detailed list of changes.

Let’s take the previous example, where I had a pull request that improves the logic of habit status computation in a habit-tracking app. Here’s the CodiumAI Agent’s review of the PR:

First, it determines the main theme, which may serve as a title. It suggests the PR summary that indeed matches what is done. Then, it determines the type of PR and whether the relevant tests were added. In addition, it estimates the effort to review and tells if there are any security concerns.

If you choose one of the different ways to use the agent, which we’ll cover later on in the article, it will be able to automatically update the title and description and also add labels to the PR. Use /describe command for that:

CodiumAI’s PR-Agent can also write a detailed list of changes and update your CHANGELOG.md. For that, use the update_changelog command:

Among the mentioned features, GitHub Copilot’s abilities are limited to crafting titles and descriptions with the help of copilot:summary. It also can write a detailed list of changes through copilot:walkthrough command.

Get feedback and suggestions

Alongside PR analysis, the agent gave the following PR feedback upon running the /review command:

💡 General suggestions: The PR includes a significant amount of refactoring and changes in the logic of habit status computation. It would be beneficial to include unit tests to ensure the new logic works as expected and to prevent potential regressions in the future. Additionally, it would be helpful to provide more context in the PR description about why these changes were necessary and how they improve the codebase.

So what it’s just done is address two main issues that may prevent this pull request from being merged:

1. The PR does not include unit tests.
2. The motivation behind the changes is missing.

Additionally, it provided detailed code feedback with references to specific files and even lines of code:

But hold on, there is something even more impressive. You can ask follow-up questions! Check this out:

Moreover, it can also modify the code that needs to be improved with the help of the improve command. This can be something simple such as a naming inconsistency that your IDE has not been able to detect:

Rename the variable 'routineId' to 'habitId' to match the new naming convention:

Existing code:

 Kotlin
val previousStreak = streakRepository.getStreakByDate(
    routineId = habit.id!!,
    dateWithinStreak = date.minus(DatePeriod(days = 1)),
)

Improved code:

 Kotlin
val previousStreak = streakRepository.getStreakByDate(
    habitId = habit.id!!,
    dateWithinStreak = date.minus(DatePeriod(days = 1)),
)

Or it may point to a missing test case:

Isn’t that cool? It now motivates me to submit pull requests for my personal projects, where I'm the only contributor because there is no one else to review my code.

With all this automatic feedback, people can not only learn how to write good pull requests but also learn about programming practices that they might otherwise overlook. Take, for instance, one of the code feedback suggestions that recommended utilizing parameterized tests. One might be unaware that such a thing exists.

GitHub Copilot currently does not provide any of the functionality described in this section.

Even more commands!

Unlike Copilot, CodiumAI also supports commands like:

similar_issue: retrieves and presents similar issues;

generate_labels: suggests custom labels based on the PR code changes.

add_doc: adds documentation to undocumented functions/classes.

Moreover, commands can be appended with a parameter from the configuration.toml file. For example,

/describe --extra_instructions = "Emphasize that ..."

Supported platforms and different ways to use

For simple usage, it may be enough to call @CodiumAI-Agent every time. However, this approach has several limitations, such as:

• Multiple comments from the agent can potentially clutter the conversation. You may want to write everything on your behalf so that you can easily delete these comments.
• CodiumAI-Agent can’t edit comments, files, and PRs.
• It can't be used for private repositories.

Fortunately, there are solutions to all the listed cases, as well as numerous other scenarios where a more nuanced control is desired. To cater to any of these situations, you’ll need to install the agent in one of the following ways:

Locally

• Using Docker image (no installation required)
• Run from source

GitHub specific methods

• Run as a GitHub Action
• Run as a polling server
• Run as a GitHub App
• Deploy as a Lambda Function
• AWS CodeCommit

GitLab specific methods

• Run a GitLab webhook server

BitBucket specific methods

• Run as a Bitbucket Pipeline
• Run on a hosted app
• Bitbucket server and data center

BTW, you may have noted that in contrast to Copilot, CodiumAI's availability is not limited to GitHub. It also gives you a choice between different AI models, such as GPT-4, GPT-3.5, Anthropic, Cohere, and Llama2 models.

However, it’s important to note that in order to install the PR agent, an API key of one of the supported models is required. In this case, CodiumAI’s services stay free, but the requests to the model itself may require additional fees.

It’s also possible to run the PR-Agent in VS Code or any of JetBrains IDEs. This functionality is included in the CodiumAI IDE plugin. But note that it requires a CodiumAI Teams subscription.

Response time

Both Copilot and CodiumAI respond in approximately 30 seconds on average. Roughly 10 seconds after submitting a command, the 👀 emoji appears to indicate that the result is loading.

Among the things I didn’t like is that when some exception occurs, you don’t see any response at all. You’re left wondering whether it’s still loading or not. It would also be great if the generated part of the response was visible while it was being constructed.

Nevertheless, this differs when it comes to the CodiumAI PR-Agent within the IDE. It does display partial responses, so you don’t need to wait. Additionally, I noticed that its response time is a lot quicker in the IDE, typically taking 15–20 seconds.

Generating tests

One of the most common things that don’t allow PRs to get merged is the lack of necessary unit tests, which makes generating them directly in pull requests one of the most promising things AI tools can provide.

GitHub plans to solve this problem by introducing a GenTest feature that will offer suggestions on resolving issues that would analyze the PR for missing tests and then create another pull request that suggests the necessary unit tests. Unfortunately, this feature is still not released publicly.

For now, CodiumAI does not have such a feature either, although the PR-Agent does address the absence of unit tests.

However, you can use the /ask command to get detailed test suggestions and then go back to your IDE and write them there. That’s exactly what they specialize in when it comes to their IDE code assistant.

Ghost text

This one is actually where the GitHub Copilot is most likely to win. It’s about the subdued, inline suggestions that appear as you type in the editor.

When it comes to the Copilot IDE plugin, these suggestions have been proven to be one of the most convenient ways AI can integrate into the development workflow. Now they are working on bringing this UX to the pull request experience.

So what is better?

Alright, so both assistants have great potential to improve our workflow, save time, and automate tedious tasks. I’m looking toward the future where such tools integrate seamlessly into the development process.

As we are waiting for the new cool AI features, let's now determine which of these two is currently superior.

Let’s address the elephant in the room, from all the described features, GitHub Copilot only has PR summaries available at the moment. To get access to this tool, you must sign up for a waitlist for Copilot Enterprise, which costs 39$ per user/month. But to sign up, you need GitHub Enterprise Cloud, which, in turn, costs 21$ per user/month. Other GitHub Copilot features are currently not available at all! For now, they are only planned as a part of the Copilot X prototype.

But let’s not underestimate it. According to GitHub Next, the planned features are quite promising and it may be great for user experience in the future.

CodiumAI, on the other hand, lets us try the cutting-edge technology right now. It provides more commands, is open-source, and is well-documented. It also gives us ample opportunities for customization. In terms of pricing, we can use it for free, or opt for one of the paid plans that offer more features.

CodiumAI has plans on its roadmap too. For example, enforcing CONTRIBUTING.md guidelines.

Conclusion

I’m quite impressed with the power of these tools. We can now automate AI responses for our repositories however we want. I can see such PR agents operate in tandem with tools like CI/CD pipelines in the future, and serve as efficient and quick-to-respond intermediaries for creating and reviewing PRs.