CloudPress — Part 1: How to reinvent a better wheel!

Foreword:

I've decided to move my articles from Medium to dev.to. I just discovered this platform and immediately fell in love with it. I believe that this is a better medium (pun intended) to publish articles in the tech industry. (It's also not blocked in my country, so there is that burden off my shoulder.)

You can find the old article here.

CloudPress is a new content management system that I’ve been working on intermittently for the past year or so. It went through multiple evolutions so far, the biggest of which was the migration from plain JavaScript to TypeScript, which took about a month considering that — as it stands — the project is approximately 56,000 lines of TypeScript code. (Not counting comments and miscellaneous files.)

You might be saying “Hey, look, another foolish attempt at a CMS, let’s see how long this one lasts.” Or you might be asking, “why do we need yet another CMS at all?” But hear me out please. Okay?

Over the past decade, we’ve made amazing and huge strides in technology. We now have React.js and another hundred virtual DOM libraries out there that make it possible to run isomorphic/universal JavaScript on the client and the server. We have GraphQL, an amazingly strict API for data loading and processing. We have CSS modules. We have all kinds of new toys, but where does the web stand?

We have PHP running 82.9% of the entire web according to this source.

We have WordPress leading the CMS with a 58.9% market share.

Even Facebook itself (the inventors of React and GraphQL) are still using PHP.

So, who’s using the amazing technologies we’ve seen coming out in the past few years?

Few, fragmented projects. For instance: there’s Vulcan.js attempting to bring GraphQL and isomorphic server rendering to the Meteor flagship and offering CMS-like ease of development, if not outright CMS functionality out of the box.
There are quite a number of emerging content management systems built with Node.js and those cool technologies. Although all of them are in the early stages of development and lack maturity in my opinion; and some are more opinionated than I’d like.

But the problem I saw. The problem that still plagues me, is that all of that is meaningless to the average Joe. There’s a real barrier between your end user and the cool technologies we developers can easily deploy.

We have a load of rocket parts but no rocket scientist is building the rocket. Meanwhile, the end user is forced to use dated tech and with limited choices at that.

Unless, of course, they dished out enough money to develop a custom solution from scratch. Which is quite the undertaking, actually, considering the average wages of node developers nowadays.

So I sat down and thought: I have multiple project ideas that share a common denominator: they all require a CMS that scales, and one that’s obscenely dynamic and versatile.

Something infinitely extensible.

And just like that, a plan came to mind; and after looking at my options I decided that I would build one from scratch to suit my needs.
I worked for a while as a WordPress developer in the past, and I really liked some things about WordPress’ design philosophy. Namely how the filter and action systems make it very extensible and straightforward. So I decided to start by emulating that in JavaScript with node.

Now let’s get technical.

The current system is an amalgamation of the WordPress way of doing things and my own vision on the subject. Instead of a global filter/action system, CloudPress is component-based. Meaning that all plugins inherit a base class: Channel.

A Channel is an event-based object that supports filters and actions. If you’re not familiar with the filter system in WordPress: it’s a system where a single value (or what’s called a Payload in CloudPress) is forwarded through a chain of middleware. Each middleware (handler) can make modifications to the value or overwrite it altogether, then call the next handler in line.

As a matter of fact, since the middleware in CloudPress is essentially an async function, it can call the rest of the chain first, then modify the value. The system is versatile like that.

Here’s an example of how a plugin (here the Renderer plugin, responsible for rendering the page) might apply filters:

/**
   * Renders and returns the HTML for a page.
   * @param renderInfo Information for the renderer.
   * @param renderContext Context data.
   * @returns {string}
   */
  async render(renderInfo: RenderInfo, renderContext: RenderContext) {
    const filterContext = { renderInfo, renderContext };
    const meta = await this.applyFilter('meta', { ...renderInfo.metaTags }, filterContext);
    const { graphql, bundle } = this.imports as any;
    const scripts: string[] = [], stylesheets: string[] = [];
    const bundles = new Set(renderInfo.bundles);
    if (bundles) {
      for (let name of bundles) {
        const item: IBundle = await bundle.regenerate(name);
        if (item.script)
          scripts.push(item.script);
        if (item.stylesheet)
          stylesheets.push(item.stylesheet);
      }
    }
    /**
     * The `browser-state` filter can be used to alter the state object on the client.
     */
    const state: any = await this.applyFilter('browser-state', {
      graphql: { endpoint: await graphql.endpoint() },
      initialState: {}
    }, filterContext);
    const component = React.createElement(
      ApolloProvider, renderContext, React.createElement(HtmlContainer, {
        title: renderInfo.title || await this.engine.configManager.readKey('site.title',
          'Welcome to cloudpress!'),
        meta,
        state,
        scripts: await this.applyFilter('page-scripts', scripts, filterContext),
        stylesheets: await this.applyFilter('page-stylesheets', stylesheets, filterContext)
      } as any, React.createElement(renderInfo.component, renderInfo.props))
    );
    try {
      await getDataFromTree(component);
    } catch(e) {
      if (e.queryErrors) {
        for(let error of e.queryErrors) {
          await this.log([error.message, error.stack], Severity.Error);
        }
      } else
        await this.log([e.message, e.stack], Severity.Error);
    }
    state.initialState = renderContext.store.getState();
    cleanupApolloState(state.initialState.apollo);
    /**
     * Plugins can use the `stream` filter to alter the final HTML stream.
     */
    return this.applyFilter('stream', ReactDOM.renderToNodeStream(component), filterContext);
  }

And here’s how the browser plugin adds the viewport meta tag:

  await rendererPlugin.useFilter('meta', async (payload: Payload<any>, next: Function) => {
    // TODO: Make this configurable!
    payload.value.viewport = "width=device-width, initial-scale=1, maximum-scale=1";
    return await next();
  });

In addition to the payload’s value, the middleware can access payload.arguments to access the named arguments for the original function. This allows CloudPress plugins to modify each other’s behaviour quite easily.

Another thing to note here is how plugin inter-dependencies are handled. Each plugin offers a factory as its main module’s export.

import { IEngine, IPlugin, IPluginImports, IPluginFactory } from '@cloudpress/interfaces-core';
import { BrowserPlugin } from './browser';

const pkg = require('../package.json');

export default class BrowserFactory implements IPluginFactory {

  get name(): string { return 'Browser'; }
  get version(): string { return pkg.version; }
  get provides(): string { return 'browser'; }
  get consumes(): string[] { return ['bundle', 'router', 'redux', 'renderer', 'subscriptions']; }

  async createInstance(engine: IEngine, imports: IPluginImports): Promise<IPlugin> {
    return new BrowserPlugin(engine, imports);
  }

}

The factory lets the system know of that plugin’s requirements and what service it provides, and the system will instantiate the plugin with its imported dependencies ready and activated. For instance, in the case of the renderer plugin, it depends on bundle, graphql and redux services. It provides the renderer service which is used in turn by the router service to serve requests. In short, a plugin can provide a single service, and may consume any number of services.

What’s more (and was not shown here) is that the Channel base-class inherits yet another. It inherits a special promise-based EventEmitter that’s completely asynchronous. Which means that it will execute all event handlers in parallel and await any promises returned from them before returning. This provides functionality akin to WordPress’ actions.

And just like filters, you can broadcast and subscribe to events on any object that inherits Channel.

  async installLoadingScreen() {
    this.imports.server.on('install-middleware', async (app: Koa) => {
      await this.log('installing bundle routes');
      const outputDir = path.join(__dirname, '../assets');
      const endpoint = '/plugin-bootstrap/assets/';
      app.use(async (ctx: Koa.Context, next: Function) => {
        if (ctx.path.startsWith(endpoint)) {
          const filePath = ctx.path.substring(endpoint.length);
          return await send(ctx, filePath, { root: outputDir });
        } else
          return await next();
      });
      app.use(async (ctx: Koa.Context, next: Function) => {
        if (!this._ready && ctx.path == '/' && ctx.method == 'GET')
          await send(ctx, 'loader.html', { root: outputDir });
        else
          return await next();
      });
    });
  }

This is how all system components communicate and extend each other. At this point in time, there are 18 plugins that I’ve implemented or am in the process of implementing. The installer works. The database connection works (you can use any database that TypeORM supports), and I’m in the process of implementing the front-end, dashboard, and authentication modules.

The project is currently licensed under GPL v3 (I’m a fan of GPL), but I might switch or dual license it under MIT as well.

In this series, I’ll hopefully discuss more of the technical aspects of the project and the challenges I face. I’ll also try and post regular updates, future plans, and repeatedly and shamelessly beseech people to contribute to the project.
If you’re interested in contributing (I really could use the help), don’t hesitate to contact me here or on Twitter.

Until next time!

Part 2