Read All Files of Directory and Subdirectories with Recursive Generators in JavaScript

Reading all files from a directory and its subdirectories is a common requirement in many projects. While there are several ways to accomplish this, one approach is to use a recursive generator function to create a sequence of file paths.

Let's start by taking a look at the code:

import fs from 'fs';
import path from 'path';

export function* readAllFiles(dir: string): Generator<string> {
  const files = fs.readdirSync(dir, { withFileTypes: true });

  for (const file of files) {
    if (file.isDirectory()) {
      yield* readAllFiles(path.join(dir, file.name));
    } else {
      yield path.join(dir, file.name);
    }
  }
}

The function readAllFiles takes a directory path as input and returns a generator object that creates a sequence of file paths.

Inside the function, we first use fs.readdirSync to read all files and directories in the specified directory. We pass the withFileTypes option to get fs.Dirent objects instead of filenames.

Then we loop through each file or directory with a for..of loop. If the current item is a directory, we recursively call the readAllFiles function with the new directory path using the yield* syntax. This creates a nested sequence of file paths for each subdirectory.

If the current item is a file, we use the path.join function to construct the full path of the file and yield it.

Now that we have our generator function, let's see how we can use it to read all the files in a directory and its subdirectories.

for (const file of readAllFiles('/path/to/directory')) {
  console.log(file);
}

We can simply use a for..of loop to iterate through the generator object and print each file path to the console.

Using a recursive generator function to read all the files in a directory and its subdirectories has several benefits over other approaches. Instead of pushing all file paths into an array, the generator yields each path one at a time. This means that only one file path is kept in memory at a time, making it more memory efficient. Moreover, the generator only generates file paths when they are requested by the consuming code. This means that if we only need to process a subset of the files, we don't waste time and resources generating paths for files we don't need.