Engines Evolution

Once upon a time, many years ago in the grad school days of 2010, I frequented a forum for Mac game developers ("iDevGames") that still has an active website (https://www.idevgames.com/) and even (given that it's 2022) a Discord channel. It was a formative time in my development trajectory, when I was full of (not entirely well-advised) ideas of writing entire game engines from scratch.

One of the more interesting articles I wrote up on that forum, which I've looked for (to no avail) on the Internet Archive / Wayback Machine, reviewed a brief summary of game engine architecture approaches, from the super-elementary starter engine structure up to something approaching what I thought the professional stuff might look like. It was a great thought exercise.

Much to my surprise, I did rediscover the diagrams I created. I've done a lot of development since then, and learned a lot about what works; what doesn't; and what "real" (modern, AAA) engines look like "under the hood". So, I thought interesting to revisit the trajectory of increasing-complexity game engine designs I grew through, and point out a few of the more interesting fallacies.

So, sit back and pour a cold one while I attempt to recreate the article that (I fear) at this point may very well be lost to the digital sands of unarchived internet history--warts, jorts, and all.

Alphonse

The first iteration at engine architecting I named "Alphonse".

It is deliberately basic, and looks a lot like what you might put together from a first basic proof-of-concept program while you're first learning how to interface with OpenGL, or replicating a NeHe or LazyFoo tutorial.

https://lazyfoo.net/tutorials/SDL/51_SDL_and_modern_opengl/index.php

There's not a lot going on here, so we might as well focus on the nomenclature:

• Everything is within the global scope, with no namespaces; object/modeling; or library constructs around which APIs can be designed.

• There's a single entry point, likely your standard C-family "main()", that probably has some variation on the basic "while true {}" loop.

• There might be a collection of reusable functions, but there's no real state control and probably a lot of manual interfacing with things like your window management, event handling, custom rendering routines, etc.

• All state is likely handled in global variables, including pointers to a graphics context; program variable addresses; mesh geometry and transforms; texture data; game state flags / enumerations; etc.

But let's not knock it more than we have to! This is a great, flexible model with which we can go any direction we want. It's great for sandboxing, probably super-easy to build, and helps us practice the more problematic technologies in a transparent setting before we start juggling resource allocation and hiding behind multiple abstraction layers.

Bartelby

The second architecture iteration I named "Bartelby".

Hey, we've got some object structure! We can start enforcing a basic degree of namespace organization, including separation of concerns into some basic objects like "App" and "Graphics". With the introdution of some object-oriented techniques, we gain some nice advantages in reusability and can start abstracting some of the more significant headaches away, including:

• Construction and deconstruction can be used to help manage resources (like graphics contexts) and complex initialization routines (especially for those complex library dependencies you're likely starting to rely on, like SDL).

• By pulling out "Graphics" into its own namespace/object, we can focus other parts of the program on non-graphics problems (which are difficult in their own right, and easy to discount because they're--literally--less visible).

• While we still have a healthy collection of "random functions" (and likely still perform a lot of our rendering pass outside the core app and graphics behaviors), there's significant organization and streamlining we can start enforcing for state-dependent behaviors (like initialization) vs. game logic.

Of course, the majority of events (which will always be application-specific) are likely still defined by handlers and functions in the global scope. But, there's no doubt we're making significant improvements over the everything-is-global-namespace hodgepodge of Alphonse. Even within the new constraints, we're still in a place where we can explore with significant flexibility and transparency.

Christina

The third architecture iteration I named "Christina".

Now we're getting somewhere! "Christina" is well-reflected in a consolidated and reusable (across a half-dozen different personal projects, at least) engine that represented the peak of my game architecting efforts around this time. I called the reusable engine Artemis, and recently dusted off the source code to a buildable status and posted it on GitHub:

https://github.com/Tythos/ArtemisLib

I even wrote an article about how the engine architecture is structured and reused in some greater detail, which you can find here:

https://dev.to/tythos/old-school-is-new-again-sdl2-grad-school-engines-and-pacman-clones-32la

I'll still summarize this approach here, though, with some basic observations on key pros and cons:

• We have a real "App" object here that represents an authoritative engine singleton. There's real separation of concerns, including both behavior and state, in a way that identifies and interfaces key reusable subsystems.

• We have clear modeling of reusable interface elements ("Panels", as I called 2D UI models); auxilliary engine systems for sound and music; somewhat-abstracted 3d objects (though not really part of a full-up scene graph yet); and the "Graphics" object from Bartelby that abstracts our GL interface (including initialization and, to a degree, rendering).

• One key advance here is that we've completely grown beyond the need for random global functions for fleshing out application behavior. The "App" object maintains a list of "condition" and "response" function pointers that define game logic. The first function pointer returns a bool value that can be evaluated each frame; if it returns "true", the second function pointer is invoked.

It's easy to underestimate how big an accomplishment that last item represented. Growing an engine beyond the point where it needs custom bindings and spaghetti code is a big milestone. Of course, you have to ask, where do all of those function pointers come from? Well, they're still define in the global scope (ha ha ha). But, there's a clean interface between them and the engine doesn't really care, so long as it can call them.

Once game logic is encapsulated in this way, we've completely moved beyond the need for any special behaviors in the entry point. All it does now is instantiate the "App" object (an engine singleton) and enter it's main loop after all of the behaviors have been populated. Pretty awesome, actually.

Dorothy

The fourth architecture iteration I named "Dorothy".

Dorothy was, in hindsight, a classic example of "second system syndrome" and a strong counterproof of the Hegelian fallacy--not everything new or changed is necessarily better, progress is not in fact inevitable, and sometimes change can in fact make things substantially worse.

https://en.wikipedia.org/wiki/Second-system_effect

Some interesting ideas, specifically organized around MVC-style software patterns, seemed (at the time) to clearly be a "better" way to program. Ooops. Let's be generous, though, and point out some of the advantages an approach like this might actually offer:

• The clear organization of a "control" and a "model" family of classes separates, entirely, a set of states (models) from a set of stateless behaviors ("controls"). Actually, in hindsight, this has some (very vague) parallels with the ECS-style (entity/component/system) engine architecture that would come into vogue a few years later:

  https://en.wikipedia.org/wiki/Entity_component_system

• The remaining software elements are somewhat straightforward: A top-level "App" singleton class to define the engine instance, of course, and a "View" class family that would define rendering behaviors and properties across the relevant sets of models.

And... that's about all the positive things I can say. Negative things are legion--don't ever fall in love too much with your own ideas. What's the "data" class family doing? I have no idea. Why is the view family both stateless and stateful, and how is it related to the nice "Graphics" abstraction/encapsulation from earlier architectures? What made me think I could define such a strong (as in formal) typing and dependency hierarchy across something that needs to be as modular and reusable as an engine architecture?

But the real killer here are events--specifically, those that define the unique bindings and behaviors that make up an application's unique logic. Where are they? Nowhere, because they can't fit into a strong hierarchy without significant (especially in C-family languages) circular-dependency issues.

I get that I was probably "disgusted" (or at least discouraged) by how messy the function-pointer approach of Christina might have felt after a while. But, with this "Dorothy" approach: How are events supposed to be triggered? How are they supposed to "reach into" (or resolve) relevant engine states (move this over here when that button is pressed, etc.)? They can't--not without some sort of (for example) rigorous query-and-update syntax within the engine itself.

Ultimately, Dorothy was a dead end that never made it off the drawing board. This was particularly disappointing after the relative success of Christina, whose pattern would successfully contribute the core of multiple side projects. But, there's always lessons-learned, even (maybe even especially) in failures.

Ebeneezer

The fifth architecture iteration I named "Ebeneezer".

Ebeneezer took a lot of the "lessons learned" from Christina's success, combined with a significant amount of follow-up research I did on more modern engine architectures (where I could find them; not a lot of documentation was "in the open" at the time, beyond what you might glean from GDC presentations). In fact, there's quite a few lesson and mini-patterns from Ebeneezer I still use on full-time software projects to this day.

There's a lot of interesting conclusions you can take away from this architecture, and across multiple dimensions, too. I'll start with the individual layers, from which we'll dive into some of the more interesting specific groups of architecture elements.

• "User" layer elements define I/O for user-centric behaviors, including keyboard & mouse (input) and monitor & sound (output) abstractions for specific hardware considerations (like windowing contexts). The idea here is that most of these behaviors are largely stateless and more hardware-oriented in a way that shouldn't (and isn't) really specific to an application implementation. You press a key, you get a key. Etc.

• "App" layer elements comprise the "core" of the engine. This includes the "manager", which is largely an evolution of the "App" singleton from earlier architectures. But, in this case, we're making significant effort to "push" as much application-specific behavior off to other modules and let the manager focus on managing the orchestration of individual engine elements across each level. This "app" also includes 2d and 3d scene modeling, but largely as an abstraction of relationships; the actual content (meshes, textures, shaders, etc.) are referenced from elsewhere as managed resources.

• "Module" layer elements are meant to be interchangable extensions of application behavior that can be "plugged into" a specific application when those requirements are needed. Specifically, "modules" are meant to be interfaces into "external" (e.g., beyond immediate memory space) resources.

• Those "resources" could be networked resources like databases and multiplayer servers, for which modules will largely comprise wrappers around (at the lowest level) some sort of socket behavior, or just isolated elsewhere on the filesystem (like textures, shader programs, etc.). Most critically here, though, is the "interpreter" module that leads off into "scripts" resources.

The "interpreter" and "scripts" elements, here, are the real magic sauce. Events and other game logic behaviors would, in this engine architecture, largely be defined "outside" the engine space by custom scripts (in, say, Lua) that could interface into the engine manager by way of a nicely-sandboxed interpreter module. This is largely inspired by, for example, the design approach you see in the "World of Warcraft" add-on and macro infrastructure. There's no reason to give the mod community direct access into the engine!

The Past, The Future, and Conclusions

These five stages of engine evolution go from, roughly speaking, beginner to intermediate levels of software engineering and game development. I would, in subsequent years, go on to learn a lot more about many different engine architectures, including those underlying multiple AAA games.

The profession as a whole would also see a significant pivot towards more open and standardized approaches to fast & efficient engine patterns that would be openly documented, implemented, and presented at highly-visible events like GDC and SIGGRAPH. ECS is one particularly good example. Similar approaches would realize great success in taking maximum advantages of insane parallelized hardware developments in both CPU and GPU spaces.

There would also be much better consolidation and evolution of key game engine architecture reference texts, like Jason Gregory's excellent "Game Engine Architecture", that would largely supplant the lower-level ubiquitous "green books" of Premier Press. (Remember Andre LaMothe's "OpenGL Game Programming"? Ah, yes... Still on my shelf, for some reason.)

Jason Gregory's excellent "Game Engine Architecture"

Two other key forces pushing the community in this direction would be:

• The rise of profitable licensed engines (in the form of Unity, Unreal, and others) that would expose quite a bit of their underlying architecture to facilitate development.

• An increasing enthusiasm for supporting third-party content creation (like mods, custom maps/scenarios, addons/macros, and other community-enabled play mechanisms). When you give players a map editor, for example, you can't help but expose a significant amount of game logic and engine architecture.

Related to both of these points: It's very interesting, for example, to compare the scripting support in the StarCraft; WarCraft III; and StarCraft II campaign editors--all of which were developed by the same studio, but with remarkably different approaches (and impacts) on customization and scripting.

Which Brings Us To Today

For me, one of the biggest influences on my approach to engine development would be the formalization of the WebGL standard shortly after the engines defined in this article were explored (that is, following my grad school years).

(The other influence would be the advent of JSON-driven pub/sub message-passing interfaces. Remarkable that engines could be programmed without them, really.)

At first, it wasn't obvious how WebGL could be extended into a reusable engine pattern for web-based JavaScript applications, but since then, we've seen the advent of THREE.js, really one of the most remarkable software projects in recent years:

https://threejs.org/

It really is amazing to look at what is possible today to render within a browser tab. (Combine it with Electron and Cordova for cross-platform standalone executable and mobile app builds, and DAMN, it's a potent combination.) But perhaps the most remarkable thing about THREE.js is the engine pattern it exposes, and how perfectly it balances several competing concerns:

• "Hiding" (that is, automating/facilitating) the "ugly" parts of engine development, like transformations and graphics API calls, so developers can focus on the "fun" (and unique) parts of their own software projects.

• Exposing, in a transparent way, low-level engine behaviors like graphics shaders (despite the above) in a manner where, if you want to write your own GLSL code, you absolutely can--and it's easy to drop in and get it working (or share other people's interesting shader projects) with minimal headache.

• Providing a minimal engine pattern and graphics pipeline/lifecycle that is both minimalistic/reusable and simultaneously extensible and a joy to use. Just look at the considerable plugin and extension ecosystem around the THREE.js community (not to mention the technical demos) and you'll see exactly I mean.

I've seriously considered taking some of the lessons I've learned from THREE.js, for example, and revisiting my old SDL-based engine code to see if there's a new evolution of these engine patterns I could squeeze life out of. "SDL-THREE"? "Forrester"? Who knows. Stay tuned, I suppose, because if it happens, I'll be sure to write about it here.

Tythos out. And off to pour a drink.