Working with Files in Cloud Run Jobs: Introducing GCS Fuse

When it comes to processing files within your Cloud Run Jobs, having a familiar filesystem interface can make things a whole lot easier. That's where GCS Fuse comes in! It bridges the gap between Google Cloud Storage (GCS) and your Cloud Run Job's environment, allowing you to mount GCS buckets as if they were local directories.

Why GCS Fuse?

• Simplified File Access: Read, write, and list files using standard commands and libraries.
• Performance: GCS Fuse caches frequently accessed files, making subsequent reads faster.
• Flexibility: Integrate with your existing file-based workflows and tools effortlessly.

Cloud Storage Volume Mounts

Before it was a bit of a hassle to set up GCS Fuse in either Cloud run or Cloud run jobs, you had to install it manually in a Docker container and start it, as you could see in Google samples repo.

When Google announced managed support for it:

It was great news! Made my job and the job of many folks that leverage "serverless" solutions in many different parts of their architectures much easier!

Now, what are those cloud storage volume mounts, you may ask?

The managed version of GCS Fuse leverages a Cloud Run feature called Cloud Storage volume mounts. Essentially, this allows you to specify a GCS bucket in your Cloud Run Job's configuration, and the job will have direct access to the files within that bucket.

Setting it up

All you need is to include a volumes section to define the mount point and the GCS bucket you want to access. docs

Python library:

    container = run_v2.Container()
    container.volume_mounts = [
        run_v2.VolumeMount(
            name=volume_name,
            mount_path=my_local_dir_path,
        ),
    ]

    job = run_v2.Job()
    job.template.template.volumes = [
        run_v2.Volume(
            name=volume_name,
            gcs=run_v2.GCSVolumeSource(
                bucket=my_bucket_path,
            ),
        ),
    ]

To use any files that lives inside the bucket, the beauty about it, is you abstract away all the GCS code, and only need to deal with local files.

Really simple example:

f = open(f"{my_local_dir_path}/sample-logfile.txt", "a")

Under the hood the GCS Fuse config will be doing all the necessary list and read operations, same for writing.

Tips and Considerations

• Caching: Keep in mind that GCS Fuse uses caching, so changes you make to files in the mounted directory might not immediately propagate back to GCS.
• Concurrency: For multi-worker jobs, be aware of potential concurrency issues if multiple workers try to modify the same file simultaneously.
• File Locking: GCS Fuse doesn't provide file locking, so consider how your job handles concurrent writes.

That's it!

GCS Fuse and now Cloud Storage volume mounts provide a powerful way to deal with file operations in your Cloud Run Jobs. I use this feature extensively in production, make sure you dive into the official documentation for more details and start leveraging this feature to enhance your cloud-based workflows.