The Magic of Using TypeScript at Runtime

Ever wanted to pass the path to a TypeScript file to some other library or module maybe to run it in a separate process or in a worker thread.
You think you can just write the file in JavaScript, but you found out you will need to import another one or two of your TypeScript files beacuse it needs a function from it. Then I guess you have to refactor your whole codebase to use JavaScript instead, or not, if you read through this.

The problem is, using require with a TypeScript file won't work, because Node.js does not and cannot handle .ts files. The extensions handled by the require function by default are .js, .mjs, .node, .json. The libaray or module you are passing the file path to would eventually require it at runtime and even if you add .ts to require.extensions, it would only resolve properly, but you get a syntax error on execution. This means sending it a TypeScript .ts file won't work, require will choke on this.

import { Worker } from 'worker_threads'

const worker = new Worker('./path/to/typescript/worker.ts')

At runtime in the worker_threads module it would probably look somewhat like this

class Worker {
  constructor(filename) {
    const mod = require(filename)
  }
}

Well, the implementation is quite different, but my point is; filename will eventually go through require maybe not in this same process.

The magic

The only option is to precompile your TypeScript files, know where the compiled file would be output to before it would be compiled, then pass it the path. But what if you use a runtime like ts-node, which compiles on the fly and run the compiled files in-memory without emitting? There's no way to do this, except:

File 1: worker.js

A base JavaScript file to portal every worker file written in TypeScript to.

// worker.js
const { workerData } = require('worker_threads')

require('ts-node').register()
require(workerData.aliasModule)

File 2: worker.ts

The module containing the code to be run on a worker thread, which is actually written in TypeScript.

// worker.ts
const { parentPort, workerData } = require('worker_threads')

parentPort.postMessage(`Post back: ${workerData.whatever}`)

File 3: index.ts

This is the main file that needs to run a job on a worker thread. It begins the whole worker thread thing.

// index.ts
import path from 'path'
import { Worker } from 'worker_threads'

const worker = new Worker('./worker.js', {
  workerData: {
    aliasModule: path.resolve(__dirname, 'worker.ts'),
    whatever: 'Hello, worker bee! The Queen greets you.',
  },
})

worker.on('message', (message: string) => {
  console.log(message) // Post back: Hello, worker bee! The Queen greets you.
})

If you are setting a timeout to terminate the worker so you don't wait for too long when it hangs, bear in mind the worker may take quite long to post a message, because it will first do some TypeScript compilation and type-checking. You could make it fatser by setting transpileOnly option to true on register.

Most of the magic is done by ts-node using the require('ts-node').register() which registers the loader for future requires. The most beautiful thing about this magic is that you can dynamically set the module to load, because of the way the modules are structured. Therefore uisng worker.js for future workers but running different code in it is possible.

Re-creating the magic with a job queue like Bull

If you ever used a job queue in a Node.js application or more specifically, Bull, you will know you sometimes have to run a job in a different process (child process) from the main one (parent process). Bull lets you specify the path to the file or filename that conatins the code to process the job. Whenever you pass a file to queue.process, Bull knows to process that job in a different process.

In the case where a job processor is CPU intensive, it could stall the Node.js event loop and this could lead to double processing a job. Processing jobs on a separate process could prevent double processing it. Processing the job on a separate process would also make sure the main process doesn't terminate even when the job process terminates maybe due to a runtime error.

The same problem as with worker threads happens here again if we are using TypeScript. We can't do:

queue.process('./path/to/typescript/process-job.ts')

As we did with the worker thread example, although may not be as dynamic as that, we could do the same here also.

We create the queue and add a job to be processed to it. We then specify the code file that processes the job off of the queue.
Bull will run this code file in a separate process, but it can't handle TypeScript files.

// index.ts
import Bull from 'bull'

const queue = new Bull<IData>('job-queue', options)

queue.add('job-name', data)

queue.process('job-name', './path/to/processor.js')

Using the ts-node register method as before, we register a loader to be used for future require, then load the TypeScript code file, compile it and run it. Bull picks the top level export (default export or unnamed export) from module.exports and invokes it with the job object containing info sepcific to the job and the data, sent from queue.add, to be processed.

// processor.js
require('ts-node').register()
require('./processor.ts')

The processor.ts file is the file containing the original code to process the job.

// processor.ts
export default async function (job: Bull.Job<IData>) {
  // do something with job.data
}