Persist data in Vercel(NextJS) serverless backends

Photo by Aron Visuals on Pexels, chosen for its similarity to Vercel's logo

TL:DR;

• This is not an introduction however, here is a good intro to serverless backends on Vercel by Vishal Yadav
• Persisting data by storage in global objects doesn't work in Vercel functions because they are ephemeral.
• Writing files to the directory of the project throws a server crashing error while reading files doesn't
• The workaround used involves writing and reading files with unique filenames to a temporary directory; /tmp, allowed by Vercel but absent in the docs

I recently migrated my web app from Railway to Vercel only to encounter bugs in my Node.js server due to the short-lived nature of Vercel functions.
I had used an object to store intermediate data for the url parameter of each GET request. The said object reverts back to {} everytime the serverless function is hit.

The following comprises the steps I took in working up a solution as well a link to the web app to see a working version of the code and logic below.

Figuring out likely problems and their solutions

Unique filenames, unique data

Filenames for stored data should be unique to avoid mixing them for every request.
Say the exported function in api/root.js is reached by requests to https://<unique-url-to-app>/api/root. It may at first suffice to generate a unique string from the parameters of the GET or POST requests however, serializing request URLs into unique filenames leaves room for duplication not to mention, frustrations with invalid strings for filenames.

One solution to generating UUIDs is require('crypto').randomUUID() if it is supported by the current node version you choose to work with.
Otherwise, here is a useful function for generating UUIDs gotten from the 1loc repo

const uuid = (a) =>a
  ? (a ^ ((Math.random() * 16) >> (a / 4))).toString(16)
  : ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, uuid);

Making reading/writing data similar to using objects: {"data":"data"}

The code design is done in a way that is asynchronous yet ergonomic in the way getting and setting properties on objects are.

• Writing data is done after which a callback is supplied the generated UUID wherein it may be sent to the client as part of the response

/* api/store.js */
let fs   = require('fs'),
    path = require('path'),
    dir  = '/tmp/',
    uuid = a =>a
      ? (a ^ ((Math.random() * 16) >> (a / 4))).toString(16)
      : ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, uuid);

module.exports = {
  write: (data, cb, id)=>fs.writeFile(data, path.join(dir, id=uuid()),
    function(err) {
      if(err) throw err;
      else cb(id)
  })
}

Using it is as follows; reading the data comes after writing it

let {write} = require('./api/store'),
    data;

write(data, function(uuid){
  /*the uuid is sent to the client which then uses it as a sort of unique string to retrieve its associated data by sending it along with its requests at a later time to be read by store.read */
  response.send(JSON.stringify({uuid})/*may be sent as either JSON or text*/)
});

• Reading of stored files is wrapped with a Promise to await the result of its callback as follows:

/* api/store.js */
let fs   = require('fs'),
    path = require('path'),
    dir  = '/tmp/';

module.exports = {
  read: uuid=>new Promise((resolve, reject)=>{
    fs.readFile(path.join(dir, uuid), (err, buffer)=>{
      if(err) reject(err);
       else resolve(buffer.toString('utf-8'))
    })
  }),
  rm: id=>fs.unlinkSync(path.join(dir, id))
}

Which can then be used as follows

let {read, rm} = require('./api/store'),
    data;

if(data = await store.read(/*uuid*/)) {

  response.send(data);
  rm(/*uuid*/) //may be removed or left alone
}

Final and working draft of the code in api/store.js

/* store.js */
let fs     = require('fs'),
    uuid   = (a) => (a ? (a ^ ((Math.random() * 16) >> (a / 4))).toString(16) : ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, uuid)),
    /* may be made asynchronous */
    rm     = id=>fs.unlinkSync(path.join(dir, id)),
    dir    = '/tmp/',
    path   = require('path');

/*writes/reads data as text/plain, open to modifications*/
module.exports = {
  read: id =>new Promise((resolve, reject)=>fs.readFile(path.join(dir, id), (err, buffer)=>{
      if(err) reject(err);
      else resolve(buffer.toString())
    })
  ),
  write: function(data, cb, id) {
    fs.writeFile(path.join(dir, (id=uuid())), data, _=>cb(id))
  },
  rm 
}

Using it in the wild

Here is how I used it to store and retrieve results generated for each valid request in this webapp

let store  = require('./store');
module.exports = async function(request, response, uuid) {
  let {url, id} = request.query;
  if(uuid = await store.read(id)) {
    /* exists */
    response.send(/* data */),
    /* you may remove the file or leave it since the contents of the /tmp/ directory are likely temporary */
    store.rm(uuid)
  } else {
    store.write('data', function(id) {
       /* send the unique id for the stored data in some way as JSON or text */
       response.send(id)
    })
  }
}

Conclusion

The /tmp/ directory, from its name, may be susceptible to changes from other programs on the server and therefore is not advisable to persist data in it in the manner you would a database.

Vercel provides object blobs for persistent storage but it is not used in this code base since its need for storage is temporary and only throughout the short runtime of the serverless function which is expected to send a reponse within a maximum allowable time of 25s if its runtime is edge.

This maximum duration is configurable if the runtime of the function is set to something aside edge like nodejs as follows

export const runtime = 'nodejs';
export const maxDuration = 15;
/*rest of the code in the file goes here*/

Serveless functions are a great way to start building the backend for a website or webapp with less worrying over edge networks, scaling, server distribution, IP-based content delivery, etcetera