TL;DR

PR-Agent looks promising for quick feedback, answering questions, and providing committable suggestions on pull requests.

Intro

Have you ever dreamed of pull requests (PRs) that are fixed and merged automatically right after they are created? With no devs and QAs spending time on code review and with no wait time on your side.

No? Me neither.

BUT. What I think would be beneficial is a tool that can provide meaningful feedback on PR changes with committable suggestions. And this is exactly what PR-Agent can do.

PR-Agent overview

PR-Agent is an open-source tool that helps in reviewing pull requests. It analyzes the pull request and provides feedback as well as suggestions on how to improve the code.

PR-Agent supports GitHub, Gitlab, and Bitbucket. You can use the free public @CodiumAI-Agent on GitHub, install PR-Agent on your premises, or subscribe to PR-Agent Pro which is a hosted version of PR-Agent, provided by CodiumAI.

PR-Agent calls OpenAI's GPT-4 under the hood. It also supports other models should you want to use one of them, e.g., GPT-3.5, Anthropic, Cohere, Llama2.

You can find more info on the official page and on GitHub.

PR-Agent commands

Okay, now let's take a look at what PR-Agent can do. I will use the free @CodiumAI-Agent available on GitHub.

Running PR-Agent is as simple as tagging it in the PR comment.

I'm going to take my repository for Advent of Code puzzle solutions and create a PR. All the samples below are run in this PR, so feel free to check it out for more details.

Also, here is the tools guide with the description of available PR-Agent commands.

All right, let's dive in.

Review a PR

Provides PR analysis and feedback on the code changes.

Usage:

@CodiumAI-Agent /review

Sample:

Code feedback under the cut:

The analysis is well-formatted and pretty clear.

It suggests adding error handling for cases like file not found or wrong file format, so it looks like it sees only the context of the PR. It cannot see the bigger picture, i.e. that the file is local to that same repository and I don't need error handling here. If the file is not found, then it means I made a typo and I will just fix the path in the code. But still, it's good to have feedback on this as a reminder.

Suggest code improvements

Suggests changes that can be committed.

Usage:

@CodiumAI-Agent /improve

Samples:

These suggestions make sense to me, I think I would apply them.

Add documentation

Generate documentation for new components introduced in the PR, again as committable changes.

Usage:

@CodiumAI-Agent /add_docs

Sample:

The doc looks verbose a bit. I would not commit it as is, but it can be used as a starting point.

Describe PR

Update the title and description of the PR based on its content.

Note that @CodiumAI-Agent is a "promotional" bot of PR-Agent, which does not have access to edit existing data in my PR. Hence, /describe won't work in this case.
What we can do here is add a command parameter, so that description is published as a PR comment:

@CodiumAI-Agent /describe --pr_description.publish_description_as_comment=true

Sample:

Changes walkthrough under the cut:

The generated description is not as verbose as the generated documentation which is good. And yet it is a bit vague in "performs some calculations". But again, it looks nice as a starting point.

Answer questions

You can ask questions about the PR. For instance:

@CodiumAI-Agent /ask How can I change the code to make it more readable?

Sample:

Continuation - example of refactored code:

The refactored code compiles and produces the same output which is good. And it indeed looks more readable. There is one thing here though that I don't personally like and try to avoid: modification of mutable map passed as a parameter. But again, this is just an example that could be used as a quick help to refactor the code.

Add changelog updates

Update the changelog based on the PR's contents. By default, PR-Agent generates a comment but then it can push it to the CHANGELOG.md file.

Usage:

@CodiumAI-Agent /update_changelog

Sample:

Nothing special here and yet the markdown formatting is good, the description is clean and concise.

Also note that it inserts the current date in the header, not the date when PR was created.

List config parameters

There are a lot of configuration parameters you can set in a config file or in the command itself (for instance, see the /describe command sample above). To list them all, use the /config command:

@CodiumAI-Agent /config

Sample:

Security concerns

Okay, this is great, but what about security? Does the PR-Agent and/or OpenAI store the code they analyze?

In a nutshell, free-tier users' data is used to train AI models. Paid subscribers' data is never used to train models and is stored for 48 hours for troubleshooting purposes only. Also, CodiumAI-hosted PR-Agent uses zero-data-retention OpenAI account.

If you still prefer zero data retention, then there is an option to email CodiumAI support and explicitly ask for this.

If you use a self-hosted PR-Agent with your own OpenAI API key, then it is between you and OpenAI.

For more information see the following blog post: CodiumAI Security: Our Commitment to Data Privacy and Security.

Things to improve

After using @CodiumAI-Agent on GitHub for some time I found a couple of things that would be nice to have:

• PR-Agent does not answer anything on both wrong and premium (PR-Agent Pro) commands. For instance:

@CodiumAI-Agent /test
@CodiumAI-Agent /analyze

It would be nice if it added a comment to PR stating that the command is wrong and providing usage info with a list of the correct commands.

• I don't know if it's even possible with comments in PRs but it would be really cool to have an autocompletion for commands available. So that when I'm typing @CodiumAI-Agent /, the list of possible commands appears. Then, once I picked the command, the list of possible configuration parameters for this command is presented.

One more feature that I think would be beneficial is integration with static code analysis tools such as SonarQube. You can set your own profile and rules in SonarQube. When you open a PR, SonarQube analyzes changes, and based on issues found PR-Agent could automatically suggest committable changes in the PR to fix issues.

Conclusion

PR-Agent looks promising for quick feedback and suggestions on PRs. Of course, you cannot blindly accept everything it produces (yet), and review is still needed. However, this looks like a big help in speeding up the PR review and improving code quality.

Dream your code, code your dream.