Copybara: A Tool for Transforming and Moving Code between Repositories

Copybara is an internal tool developed and used by Google to facilitate the transformation and movement of code between repositories. It addresses the need for code to exist in multiple repositories and provides a solution for keeping them in sync. One common use case is maintaining a confidential repository alongside a public repository.

To ensure there is always one source of truth, Copybara requires the selection of an authoritative repository. However, contributions can be made to any repository, and any repository can be used to cut a release. The tool simplifies the repetitive movement of code between repositories, and it can also be used for one-time code transfers to a new repository.

The main features of Copybara include its stateless nature and the ability to store state in the destination repository as a label in the commit message. This allows multiple users or services to utilize Copybara with the same configuration and repositories, ensuring consistent results.

Currently, Copybara supports Git repositories as the primary repository type. It also has experimental support for reading from Mercurial repositories. The extensible architecture of Copybara enables the addition of custom origins and destinations for various use cases. Official support for other repository types is planned for future releases.

Example

Here is an example of how Copybara is used in a Python workflow:

core.workflow(
    name = "default",
    origin = git.github_origin(
      url = "https://github.com/google/copybara.git",
      ref = "master",
    ),
    destination = git.destination(
        url = "file:///tmp/foo",
    ),

    # Copy everything but don't remove a README_INTERNAL.txt file if it exists.
    destination_files = glob(["third_party/copybara/**"], exclude = ["README_INTERNAL.txt"]),

    authoring = authoring.pass_thru("Default email <default@default.com>"),
    transformations = [
        core.replace(
                before = "//third_party/bazel/bashunit",
                after = "//another/path:bashunit",
                paths = glob(["**/BUILD"])),
        core.move("", "third_party/copybara")
    ],
)

To run the Copybara workflow:

$ (mkdir /tmp/foo ; cd /tmp/foo ; git init --bare)
$ copybara copy.bara.sky

Getting Started with Copybara

As Copybara doesn't have an official release process yet, you need to compile it from the latest source code. Follow these steps to get started:

1. Install JDK 11 from the Oracle website.
2. Install Bazel by following the official installation guide.
3. Clone the Copybara source code repository locally using the command: git clone https://github.com/google/copybara.git.
4. Build Copybara:
   • Run bazel build //java/com/google/copybara to build the project.
   • Run bazel build //java/com/google/copybara:copybara_deploy.jar to create an executable uberjar.
5. Optionally, you can run tests to ensure the integrity of the codebase using the command: bazel test //.... Note that some tests may require additional tools such as Mercurial or Quilt.

System Packages

You can install system packages required by Copybara using the package manager for your operating system. Here's an example for Arch Linux using the AUR:

• Install the package copybara-git from the AUR.

Using Intellij with Bazel Plugin

If you're using IntelliJ with the Bazel plugin, you can configure the

project as follows:

directories:
  copybara/integration
  java/com/google/copybara
  javatests/com/google/copybara
  third_party

targets:
  //copybara/integration/...
  //java/com/google/copybara/...
  //javatests/com/google/copybara/...
  //third_party/...

Note that configuration files can be stored anywhere, but it is recommended to treat them as source code and store them in a version control system like Git.

Building Copybara in an External Bazel Workspace

To build Copybara in an external Bazel workspace, you can define convenience macros for its dependencies. Add the following code to your WORKSPACE file, replacing {{ sha256sum }} and {{ commit }} with the appropriate values:

http_archive(
  name = "com_github_google_copybara",
  sha256 = "{{ sha256sum }}",
  strip_prefix = "copybara-{{ commit }}",
  url = "https://github.com/google/copybara/archive/{{ commit }}.zip",
)

load("@com_github_google_copybara//:repositories.bzl", "copybara_repositories")
copybara_repositories()

load("@com_github_google_copybara//:repositories.maven.bzl", "copybara_maven_repositories")
copybara_maven_repositories()

load("@com_github_google_copybara//:repositories.go.bzl", "copybara_go_repositories")
copybara_go_repositories()

You can then build and run the Copybara tool within your workspace using the command:

bazel run @com_github_google_copybara//java/com/google/copybara -- <args...>

Using Docker to Build and Run Copybara

Please note that Docker usage with Copybara is currently experimental. To build Copybara using Docker, follow these steps:

docker build --rm -t copybara .

Once the build process is complete, you can run the Copybara image from the root of the code you want to use Copybara on:

docker run -it -v "$(pwd)":/usr/src/app copybara copybara

Several environment variables are available to customize the execution of Copybara within the Docker container. Here's an example with custom options:

docker run \
    -e COPYBARA_CONFIG='other.config.sky' \
    -e COPYBARA_SUBCOMMAND='validate' \
    -v "$(pwd)":/usr/src/app \
    -it copybara copybara

Git Config and Credentials

To share your Git configuration and SSH credentials with the Docker container, you can mount the necessary directories. Here's an example for macOS:

docker run \
    -v ~/.ssh:/root/.ssh \
    -v ~/.gitconfig:/root/.gitconfig \
    -v "$(pwd)":/usr/src/app \
    -it copybara copybara

Documentation

While the official documentation for Copybara is still a work in progress, you can find some resources to get started:

• Reference documentation
• Examples
• Tutorial on how to get started

Contact

If you have any questions about how Copybara works, you can reach out to Copybara team through our mailing list.

Optional Tips

Here's an optional tip for Bazel users who want to see test errors directly in the Bazel output:

1. Open your ~/.bazelrc file.
2. Add the

following line:

test --test_output=streamed

With this configuration, test errors will be displayed directly in the Bazel output instead of having to manually inspect the logs.

These instructions should help you get started with Copybara and explore its capabilities. If you have any further questions or need assistance, don't hesitate to reach out to the Copybara community or the developers working on the project.