At the start of 2020, I wrote this twitter thread on recommendations for engineers that are self-studying to get into graphics engineering. I decided to extract the contents of that thread here, with additional exposition for the dev.to community.
What is a graphics/rendering engineer?
Most of my followers on twitter are already graphics practitioners or on their way to becoming one, but the demographic here isn't skewed in such a way, so it makes sense to describe graphics engineering a bit more.
Graphics engineers are typically found in the games and film industry, although you'll find them in all sorts of nooks within the industry (automotive, CAD software, research labs, etc). If I had to summarize a graphics programmer's job, it's to take a scene description and produce a render. By a "scene description," what I mean is a (potentially huge) data structure describing all the geometry, skeletons, particles, terrain, atmospheric properties, in the environment in addition to all the "material data" which describes how all the aforementioned objects respond to light. Furthermore, the scene description will contain a bunch of lighting information (how many lights where, which ones cast shadows, what is the range of each light's influence, what does the skybox look like, etc). These two chunks of data (materials in the scene, and lights in the scene) dictate what is "seen" by a virtual camera used to render the scene to a monitor, image, VR goggles, or what have you.
To produce an image like the above, a ton of things are happening. First, every object pose (aka, position + orientation) must be transposed to the same "view space" of the camera. This information tells us where in the frame the object should be, and how far away from the camera it is. This same transformation needs to occur for every light in the scene as well. The crucial piece of information produced by these transforms is "occlusion" information. When an object A is between a point and another object B, we say that B is occluded by A. If the occluder is occluding light, the object behind the occluder is in shadow. If the occluder is between the object and the camera, the object occluded will not be directly visible in full.
After the geometrical information is processed (either via ray tracing, rasterization, or some other technique), the next step is lighting. Lighting is a huge topic, and it would be impossible to cover every aspect of lighting. In short however, every graphics engineer is trying to solve or approximate the equation below:
That integral above needs to be solved for every pixel in the frame. What makes matters worse is that because light "bounces" off surfaces (not just mirrors, any material that isn't vantablack or a black hole will reflect light to some degree) or "refracts" through transmissive surfaces (e.g. light bending through water), the equation above is a recursive one. Solving for all inbound light impinging on a surface requires that exitant light from nearby surfaces must be solved for as well. This includes surfaces occluded from the camera! This summarizes what is known as the global illumination problem.
Finally, many graphics engineers are also concerned with the performance of their renderer. Rendering frames accurately requires a lot of computation, so rendering typically occurs on an accelerator (most commonly, the GPU). On top of having SIMD/SIMT architectures exhibiting hierarchical parallelism, GPUs have dedicated hardware for quickly performing various operations (rasterization, texture sampling, screen-space fragment derivatives, vertex attribute interpolation, etc). Using a GPU effectively also requires a sound understanding of properly utilizing a CPU. The CPU needs to "stay ahead" of the GPU to ensure the GPU always has work to do. As a result, many renderers are heavily multi-threaded and pipelined. Film rendering engineers are typically more concerned with accuracy, while game rendering engineers are typically more concerned with latency. Either way, the rendering problem is sufficiently complicated and computationally heavy that performant code is always the objective.
Why would I want to be a graphics engineer?
This is a fair question! You'll get a different answer from every graphics engineer, but my favorite aspects are the following:
- Highly interdisciplinary. Graphics engineers draw wisdom and knowledge from the following fields
- Art
- Photography
- Math (Linear Algebra, Geometry, Analysis, and more)
- Computer Science
- Electrical Engineering (in particular, signals processing)
- Physics
- Finding solutions to problems in graphics typically requires some degree of creativity and sophistication
- What you ultimately produce is usually pretty awesome
- Colleagues in the field of graphics are generally passionate about their discipline, and exceedingly fun to talk with
In addition to the above, because graphics engineers are usually hard to find, they tend to be in high demand as well.
What are important topics a graphics engineer should study?
I find I draw from the following subjects the most consistently:
- Geometry (and Geometric Algebra)
- Trigonometry
- Quaternions/Dual-quaternions
- Linear transformations
- Analysis
- Calculus/vector-calculus
- Approximation theory
- Statistics (Monte-Carlo, moment analysis, etc.)
- Linear Algebra
- Vector spaces
- Orthogonalization
- Various decompositions (SVD, QR)
- Computer Science
- Memory hierarchy (CPU and GPU cache architecture)
- Algorithms and data structures
- Compression/codecs
- Programming language that's "close to the metal" (C, C++, Rust, etc)
- Specialized knowledge on tools and APIs
- (Some degree of) Knowledge of how Art tools work
- Depending on the specialization within graphics you choose, it can be helpful to understand how artists are producing the data you will ultimately use (and in some cases, you're the one that builds the tools). Such tools include:
- 3D Modeling software (Maya, 3DS Max, ZBrush, etc)
- Material editors (Substance, various game engines, Nvidia's MDL)
- Scene compositors (Blender, game engines, various data formats such as GLTF, Universal Scene Description, etc)
- Heightmap editors
- Animation/motion-capture (mocap) tools
YOU DO NOT NEED TO KNOW ALL OF THE ABOVE TO GET STARTED. I can't emphasize this enough. As you can see, graphics engineers will typically get exposed to reams of tools, technologies, techniques, and concepts over the course of their careers. That's part of the fun!
OK, fine, but how do I get started?
There are two general approaches to learning anything, top-down, and bottom-up. I'll address these two approaches separately, with the caveat that most people will do a combination of both simultaneously. The degree that you go top-down vs bottom-up will depend on your learning style. Personally, as an example, I tend to go as bottom-up as possible, switching over to top-down only when I get stuck or if a concept gets lost on me.
Top-down
When learning top-down, the idea is to get the broad strokes of the concept and understand what's possible, without necessarily grasping all the details about implementation as you go. Taking this route, you will generally start with an off-the-shelf engine, such as Unreal Engine, Godot, or Unity. For learning purposes, I actually suggest engines that provide source code (i.e. not Unity) because while learning with a closed source engine is possible, it's a strict disadvantage compared to learning with source available. If C++ or C# aren't your cup of tea, there are plenty of other engines to get started in your programming language of choice. Be advised, however, that most graphics engineers in the wild are C++ programmers.
After picking an engine, you should try to accomplish the following tasks:
- Draw an object to the screen
- Assign a material to the object to change its appearance
- Change various draw parameters to affect the object's "draw mode" (e.g. transparency, emissive)
- Make a custom material to color the object something simple with a custom shader
- Implement a custom material that responds to a directional light (e.g. Phong Shading)
- Implement a custom material that samples from a texture
After doing the above, you should at least have a sense for how more complicated lighting models might work, without necessarily understanding the underlying math and techniques associated with it.
Bottom Up
If you take the bottom up approach, this means you want to control everything from the data, to how the data is fed to the GPU as draw calls, to what the GPU does with the draw calls, to how the draws are ultimately composited to a screen or offscreen raster (e.g. image or PDF).
To go this route, I recommend starting with either D3D11 or OpenGL (or WebGL on the web, be sure not to use Safari which has a crippled WebGL implementation). There are more modern APIs, namely D3D12 and Vulkan, but these APIs are notoriously difficult to use and designed for professionals looking to eke out as much performance as possible. The higher-level APIs such as D3D11 and OpenGL provide abstractions over memory barriers, resource ref counting, and generally take care of any potential data hazards that would otherwise crash your program. To summarize the problem, imagine you allocate memory on the GPU for a texture, and then submit a draw call that uses that texture. Later, you realize you need that memory for something else. In D3D12 and Vulkan, you must ensure that the draw call that reads from that memory has finished executing on the GPU before modifying that memory. There are many ways to do this, but in D3D11/OpenGL, such concerns are off the table because the API handles the memory management for you behind the scenes.
A great tutorial for OpenGL in particular is the Learn OpenGL tutorial, which covers the entire construction of a basic renderer. When I say "basic" here, I don't mean that all the techniques are basic, just that the architecture itself is relatively basic. If you follow it end to end, you'll learn a lot of techniques such as SSAO, bloom, and more. The still below, for example, shows the image you can create with specular image based lighting (Spec IBL).
Moving past the beginner stage
You've seen homogeneous coordinates, you've seen linear algebra, perspective transforms, shadow mapping, and a number of techniques. Hopefully, at this point, you've also taken some interviews. I'm a big believer that graphics engineers will learn most of that they know on the job, so do interview early and often to try to get a foot in the door. Generally, aptitude is more heavily weighed over experience in a graphics engineering interview, especially for a more entry level position.
From there, how do you progress to be a senior or lead graphics engineer? Here's a list of some tips I've compiled over the years.
- Start avoiding commercial engines and frameworks in your own experimentation. This gets you away from the "magic" of how things happen and forces you to learn it from the ground up.
- Expect to have 3 projects just generally floating in the background at once. A prototyping engine for trying out new ideas, an unbiased path-tracing engine (follow the PBR book to get started here) useful for validating results, and a "production engine" where you try to push perf to the limits.
- Continue studying linear algebra, geometric algebra, calculus, fourier/harmonic analysis, and any other math topics as they arise in your work
- Pay attention to data structures, particularly acceleration structures but also understand the importance of proper I-cache/D-cache utilization, branching effects, and have a good feeling for the latencies involved
- From time to time, pick a paper from ACM, Arxiv, etc that interests you and try to implement it from start to finish
- If you can, publish!
- Learn about compression and aliasing. Maybe write a BCn codec or a DCT or wavelet codec. Experiment with the various forms of AA and analyze their tradeoffs (MSAA, MLAA, FXAA, TXAA, etc).
- Stay up to date on the tools available and what they are capable of (shader compilers, shader assembly analyzers, frame dumps, GPU crash analyzers, mesh optimizers, texture compressors, etc).
- Shadow the artists you work with to understand their workflows and generate ideas on how to make things better.
- Study other engines, especially from studios like idTech, Ubisoft, Blizzard, Rockstar, etc (sorry if I didn't mention your studio!). The point is to study engines that achieve something you're looking to achieve as well.
Of paramount importance is to recognize that like other highly specialized disciplines, graphics engineering isn't something you can learn in a fortnight. In fact, it's likely something you'll never stop learning about. Staying humble throughout your career, never being able to admit when you're wrong, and constantly seeking to improve are surefire ways to advance and find fulfillment while doing so.
Closing Thoughts
This post was a bit difficult to write because I had to tread the line between offering too much (which gets overwhelming fast), or offering too little, in which case a concept/term is lost on the reader. Also, there's a thin line between showing innumerable possibilities that inspire, vs showing an impossibly high mountain that crushes spirits. To all who venture down this path, welcome! It's a challenging road to be sure, but you're in very good company. Never forget that if something is challenging for you, chances are, it's challenging for others as well. Personally, I've never encountered a graphics engineer that wasn't welcoming towards questions and showing a fellow engineer the ropes. On that note, any interested would-be graphics engineers are welcome to comment with questions! I often field questions sent as DMs on twitter as well, so you're more than welcome to contact me there.
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.