Composable HTTP Client for NodeJS

#functional #javascript #node

So I made this HTTP client for NodeJS:

var compose = require('request-compose')

BTW if you want to skip ahead and immerse yourself in the FP goodness right away - look no further

And how is it used?

var {res, body} = await compose.client({
  url: 'https://api.github.com/users/simov',
  headers: {
    'user-agent': 'request-compose'
  }
})

Oh WOW! REALLY!?
Yet another HTTP Client for NodeJS!
A M A Z I N G ! ! !

.. APIs, APIs .. Everywhere

As an end user, what if I want something fixed, changed or added in someone else's module? What are my options?

Open up an issue on GitHub and ask for it
Implement it myself and submit a pull request
Search for alternative module that have what I need
Repeat

The reason why is because the module authors present you with an API about what you can do, and what you cannot. You are essentially locked in. The authors also fiercely guard the scope of their project from something unrelated creeping in.

But what if we had more powerful primitives allowing us to step one layer below, and elegantly compose our own thing. Just for ourselves, completely bypassing the API and scope bottleneck presented in the other's solution.

Composition

Luckily there is such primitive called Functional Composition:

In computer science, function composition (not to be confused with object composition) is an act or mechanism to combine simple functions to build more complicated ones. Like the usual composition of functions in mathematics, the result of each function is passed as the argument of the next, and the result of the last one is the result of the whole.

In fact, what request-compose exposes is exactly that:

var compose = (...fns) => (args) =>
  fns.reduce((p, f) => p.then(f), Promise.resolve(args))

At its core request-compose is not even a client, it's a Functional Programming pattern, an idea, a simple one-liner to help you compose your own thing.

Whit it you can compose any function, asynchronous or not:

var sum = compose(
  (x) => x + 1,
  (x) => new Promise((resolve) => setTimeout(() => resolve(x + 2), 1000)),
  (x) => x + 3,
  async (x) => (await x) + 4
)
await sum(5) // 15 (after one second)

Or being slightly more on the topic - compose your own HTTP client:

var compose = require('request-compose')
var https = require('https')

var request = compose(
  (options) => {
    options.headers = options.headers || {}
    options.headers['user-agent'] = 'request-compose'
    return options
  },
  (options) => new Promise((resolve, reject) => {
    https.request(options)
      .on('response', resolve)
      .on('error', reject)
      .end()
  }),
  (res) => new Promise((resolve, reject) => {
    var body = ''
    res
      .on('data', (chunk) => body += chunk)
      .on('end', () => resolve({res, body}))
      .on('error', reject)
  }),
  ({res, body}) => ({res, body: JSON.parse(body)}),
)

var {res, body} = await request({
  protocol: 'https:',
  hostname: 'api.github.com',
  path: '/users/simov',
})

Can you spot the API?
There is none.
It's all yours, your own Promise based HTTP Client.
Congratulations!

Practicality

That's cool and all but not very practical. After all we usually try to extract code into modules, not coding everything up in one place.

And why you'll even bother using request-compose if you have to do all the work by yourself.

The answer is simple:

You can choose what you want to use, extend it however you want to, or don't use it all - compose your own thing from scratch.

There are a bunch of functions, however, cleverly named middlewares that encapsulate pieces of HTTP client logic that you may find useful:

var compose = require('request-compose')
var Request = compose.Request
var Response = compose.Response

var request = compose(
  Request.defaults({headers: {'user-agent': 'request-compose'}}),
  Request.url('https://api.github.com/users/simov'),
  Request.send(),
  Response.buffer(),
  Response.string(),
  Response.parse(),
)

var {res, body} = await request()

It's important to note that these middlewares are just an example of a possible implementation. My own implementation. But you are not locked into it, because it's not hidden behind API walls.

You are free to compose your own thing:

var compose = require('request-compose')
var Request = compose.Request
var Response = compose.Response

var request = (options) => compose(
  Request.defaults(),
  // my own stuff here - yay!
  ({options}) => {
    options.headers['user-agent'] = 'request-compose'
    options.headers['accept'] = 'application/vnd.github.v3+json'
    return {options}
  },
  // base URL? - no problem!
  Request.url(`https://api.github.com/${options.url}`),
  Request.send(),
  Response.buffer(),
  Response.string(),
  Response.parse(),
)(options)

var {res, body} = await request({url: 'users/simov'})

Put that in a module on NPM and call it a day.

Full Circle

Having separate middlewares that we can arrange and extend however we want to is great, but can our code be even more expressive and less verbose?

Well, that's the sole purpose of the compose.client interface to exist:

var {res, body} = await compose.client({
  url: 'https://api.github.com/users/simov',
  headers: {
    'user-agent': 'request-compose'
  }
})

And as you may have guessed the options passed to compose.client are merely composing the HTTP client under the hood using the exact same built-in middlewares.

Going BIG

Lets take a look at the other side of the coin - instead of laser focusing on the HTTP internals - we can ask ourselves:

How can we use the Functional Composition to build something bigger?

How about composing a Higher-Order HTTP Client:

var compose = require('request-compose')

var search = ((
  github = compose(
    ({query}) => compose.client({
      url: 'https://api.github.com/search/repositories',
      qs: {q: query},
      headers: {'user-agent': 'request-compose'},
    }),
    ({body}) => body.items.slice(0, 3)
      .map(({full_name, html_url}) => ({name: full_name, url: html_url})),
  ),
  gitlab = compose(
    ({query, token}) => compose.client({
      url: 'https://gitlab.com/api/v4/search',
      qs: {scope: 'projects', search: query},
      headers: {'authorization': `Bearer ${token}`},
    }),
    ({body}) => body.slice(0, 3)
      .map(({path_with_namespace, web_url}) =>
        ({name: path_with_namespace, url: web_url})),
  ),
  bitbucket = compose(
    ({query}) => compose.client({
      url: 'https://bitbucket.org/repo/all',
      qs: {name: query},
    }),
    ({body}) => body.match(/repo-link" href="[^"]+"/gi).slice(0, 3)
      .map((match) => match.replace(/repo-link" href="\/([^"]+)"/i, '$1'))
      .map((path) => ({name: path, url: `https://bitbucket.org/${path}`})),
  ),
  search = compose(
    ({query, cred}) => Promise.all([
      github({query}),
      gitlab({query, token: cred.gitlab}),
      bitbucket({query}),
    ]),
    (results) => results.reduce((all, results) => all.concat(results)),
  )) =>
    Object.assign(search, {github, gitlab, bitbucket})
)()

var results = await search({query: 'request', {gitlab: '[TOKEN]'}})

Now you have an HTTP Client that simultaneously searches for repositories in GitHub, GitLab and BitBucket. It also returns the results neatly packed into Array, ready to be consumed by your frontend app.

Wrap it up in a Lambda and deploy it on the Cloud.
That's your Serverless!

Conclusion

What if we had modules that doesn't lock us in? What if there is no API, or one that's completely optional and extendable. What if we had tools that empower us to be the author ourselves, and built our own thing that's best for us.

The idea behind request-compose is exactly that, plus it is a fully featured, and functional functional (get it?) HTTP Client for NodeJS. Or rather should I say: it contains an opinionated HTTP client bundled in it. It covers most of the use cases that you may encounter, and it's far from a toy project, nor is my first HTTP Client.