Tech Talk: 'Unit Test the Docs: Why You Should Test Your Code Examples' (Write the Docs 2022)

✨ What is this post about: As a part of my professional growth, I make time to watch conference talks on Ruby, Rails, JS, React, tech writing, and tech trivia. Previously, I'd just watch them but now I will take and publish notes for future reference.

✨ Talk: 'Unit Test the Docs: Why You Should Test Your Code Examples' by Ben Perlmutter

✨ One-paragraph summary: What’s the fastest way to make developers complain about your docs? Give them code examples that don’t work. As technical writers, we face the constant challenge of keeping code examples accurate and up-to-date. Just as software developers write unit tests to validate that their code remains functional, technical writers can create code examples within unit tests to validate that their documentation is correct.

✨ Impression: I never thought about writing unit tests for my code examples or including the unit testing into CI/CD pipeline. By the way, StackBlitz has a wonderful SDK for hosting our code examples on GitHub, which helps with keeping examples up-to-date.

Notes

• Unit tests
  • bits of code that run other code to validate that your code is working as expected
  • any non-trivial software project would have unit tests
  • they are repeatable
• Code Examples
  • bits of code that live in the docs
  • they are often copy-pastable from the docs to code so they have to be clear and reliable
  • often stylized with syntax highlighting so they can stand out and resemble how they'd look in the dev's environment
• When documenting code examples it's difficult to keep the code in sync with the source code
• Why unit tests in code examples
  • code is actually used by a dev
  • validates that it works so devs can work with the examples
  • can be a part of the CI/CD pipeline
  • validates that code example works
  • ensures that the code is understood by the documentarian
• When to test code examples
  • SDKs, middleware
  • tutorials
  • 3rd party services integrations (ensuring that nothing changed and your code still works)
  • complex examples
• Why not manually
  • time consuming
  • difficult to maintain single source of truth if the same code is used in a few places
  • formatic gets weird
  • no programmatic modifications
• Possible tool
  • BLuehawk - CLI-based tool to extract code examples from source files
• Transclusion
  • including contents of one doc inside of another
  • like imports or partials
• Workflow with Bluehawk
  • write code example in unit test and make test pass
  • annotate + extract the code with Bluehawk
  • transclude the code example in the docs
  • pros/cons:
  • pro: unit tested code
  • pro: better record of where code examples are
  • pro: single source of truth
  • pro: more repeatable
  • con: could be more automated
  • con: rerunning the CLI operations (tiresome)
  • con: working with version control