Scheduled Cloud Functions and Secrets - A Step By Step Tutorial - Google Cloud Platform 🤠

In this post I will be taking you through how to access secrets in google cloud - allowing you to make use of sensitive information such as api keys, while minimizing the risk of such data falling into the wrong hands. By following this tutorial, you will not only learn how to access secrets, but also learn how to setup and deploy a scheduled cloud function that will be triggered via cloud scheduler.

Whilst most things can be done via the command line in google cloud, for starting out it's best to use the UI as things make much more sense when you are trying to wrap your head around the ins and outs of this platform and join the dots (at least for me anyway). For that reason I will be performing and detailing this process using the UI.

Prerequisite

To allow us to implement the features that we are learning about in this tutorial, we need to setup a Firebase 'Cloud Firestore' database. This database will be used to demonstrate that our cloud function has gained access to our secret in Secret Manager. To get started, the first step is to log in to Firebase or sign up if you haven't already, and then create a new project. Once the project is created, we will need to initialize the 'Cloud Firestore' database and create a new collection called 'myCollection.' You can find details on how to set this up here, which covers steps 1 to 5:

After you have set up the Firebase 'Cloud Firestore' database and created the necessary collection, the next step is to obtain your Firebase access key or credentials. You can do this by navigating to your project settings and then to the service accounts section, where you can generate a private key. It is important to save this key in a secure location as you will need to transfer it to Google Secret Manager in the next step of the process.

Secret Manager

Assuming you already have a Google Cloud Platform account setup, go into your project - you need to create a project if you have not already. Locate the secret manager by doing a quick search and click 'Create Secret'. If you cannot access this service then you will need to enable it.

Give your secret a name of 'FIRESTORE_SECRET' and for the secret value copy the access key we gained in our previous step and paste directly as JSON into the value field. You will see there are a number of options for encryption and rotation etc - I will leave these all blank as I am happy with the default settings here, although it is recommended to look into these settings for a production setup. For the purpose of simplicity in this tutorial we go with default. Click done and your secret is created.

Cloud Function

Now are secret is created we are ready for the next step which is creating the cloud function. In the cloud console, do a search for 'Cloud Functions' and hit 'Create Function'. For the purpose of this tutorial we are going to create a 1st generation cloud function with a pub/sub trigger:

Choose 1st generation for the environment and give the function a suitable name, then select a region where your function will be hosted. For the type of trigger choose cloud Pub/Sub then click create a topic from select a topic dropdown.

Give the topic a suitable topic id. For the purpose of the tutorial leave 'use schema' and 'enable message retention' unchecked, and leave 'google managed encryption key' checked. Hit create.

Continuing with the google function configuration click save and move onto the runtime section.

In some cases your function may need run for some time, so its important to set a timeout that suits your function. Here we can be happy with the default of 60 seconds. Similarly with memory allocation - you need to ensure that your function has enough memory to handle itself. For our case the default is fine.

In the Runtime service account section we want to create a new identity for our cloud function. This identity will be given permissions to access secrets later on, which will in turn give our cloud function associated with the service account, access to those secrets. Click create new service account in the Runtime service account dropdown and give the service account an appropriate name. Click Create Service Account.

Our function configuration is now setup and we are ready to move on to writing some code. Click next.

Code

The js file we're about to use in this tutorial incorporates the firebase-admin npm module. This package allows us to execute firebase functionality on the server rather than via a front end, where it's commonly used. Additionally, we make use of the SecretManagerServiceClient from Google Cloud Secret Manager. This client will enable us to access secrets during runtime without needing to store them as environment variables at compile time, increasing the security of our secrets.

If you're here, I'm assuming you have a good understanding of JavaScript, so I won't go into too much detail about how the code works. However, it's worth noting that in the code, the cloud function entry point name should be inserted as the first argument in your cloudEvent function to avoid errors.

We also need to ensure that the correct secret name is given when we use accessSecretVersion. The name of your secret can be retrieved from google secret manager. Usually it looks something like this:

projects/<projectNumber>/secrets/<secretName>/versions/latest

Code:

const functions = require('@google-cloud/functions-framework');
const { initializeApp, cert } = require('firebase-admin/app');
const { getFirestore } = require('firebase-admin/firestore')
const {SecretManagerServiceClient} = require('@google-cloud/secret-manager');
const client = new SecretManagerServiceClient();

const getSecretValue = async (secretName) => {
    const [version] = await client.accessSecretVersion({ name: secretName})
    const secretValue = version.payload.data.toString('utf8');

    if(secretValue)
      return JSON.parse(secretValue)
}

functions.cloudEvent('helloPubSub', async cloudEvent => { // remember to change the name of your cloud function to the entry point

  const firestoreConfig = await getSecretValue('projects/<projectNumber>/secrets/FIRESTORE_SECRET/versions/latest');

  if(!firestoreConfig){
    console.log('no firestore config')
    return
  }

  getFirestore(
    initializeApp({
        credential: cert({
          type: firestoreConfig['type'],
          project_id: firestoreConfig['project_id'],
          private_key_id: firestoreConfig['private_key_id'],
          private_key: firestoreConfig['private_key'],
          client_email: firestoreConfig['client_email'],
          client_id: firestoreConfig['client_id'],
          auth_uri: firestoreConfig['auth_uri'],
          token_uri: firestoreConfig['token_uri'],
          auth_provider_x509_cert_url: firestoreConfig['auth_provider_x509_cert_url'],
          client_x509_cert_url: firestoreConfig['client_x509_cert_url']
        })
    })
  );

  const db = getFirestore()

  db.collection('myCollection').add({ name: 'John Doe', date: new Date().toString() })


});

And of course the package.json:

{
  "name": "sample-pubsub",
  "version": "0.0.1",
  "dependencies": {
    "@google-cloud/secret-manager": "^4.2.1",
    "firebase-admin": "^11.4.0",
    "@google-cloud/functions-framework": "^3.1.3"
  }
}

To complete the deployment process, simply copy and paste the above code, ensuring that you have made any necessary amendments for the secret name and cloud function entry point. Once you're satisfied that everything is in order, hit the deploy button and your cloud function will be deployed.

If any issues arise, be sure to check the logs for any error messages.

Granting Permissions

Next up we need to give your cloud function service account you made earlier permission to access your firebase secret..
Go to the secret manager, click on the secret you made earlier, then click on the permission tab.

Click grant access and in the new principals input type the service account that you made earlier for the cloud function identity. If you cant remember it you can find it in the service account section.

Once you have added the principal in the role dropdown, search for secret manager secret accessor. Select it, then hit save.

Your cloud function is now setup to have access to the secrets you specified.

Now we need to be able to fire our cloud function... To do this we will setup a chron job in cloud scheduler that will fire our topic at a certain time.

Cloud Scheduler

To create a new job in Cloud Scheduler, first navigate to the Cloud Scheduler section by using the search bar and click on the "Create job" button. Give your job a descriptive and suitable name. Ensure that you select the same region as you used earlier for your cloud function to avoid any unexpected errors.

Now, in the "Frequency" input field, you can enter a chron time that determines the schedule of your job. It's important to note that the frequency setting can have a significant impact on your costs. For example, a very frequent schedule could result in a larger bill from Google. I recommend using a sensible and suitable frequency, for example, "0 10 * * *", which will trigger the job at 10 AM every day. Remember that you can always adjust the frequency later if needed.

After selecting the time zone, hit continue to proceed with the schedule configuration. Choose the target type of pub/sub and select the topic you created earlier. The system will then request a message body, which is not relevant in this case. However, it is important to fill out this field with any suitable message to proceed. Once done, click on create to finish the configuration process.

Finally, click on the ellipses for the job you just created and click force run:

Completion

Your entry should now appear in Firestore and an entry will be made every day at 10 am via the schedule, which you will see via the timestamp in your Firestore entry.

If for any reason the record isn't made in Firestore, remember to check your logs in the Cloud Function section of the console for errors.