That Time When the Only Test and Development Environment Was Production

If You Can Habitualize One Thing as a Developer It Is Create Quick Feedback Loops

In the early spring of 2009 I was working for a web development company. The company had written a PHP application which was responsible for generating a PDF version of report cards. Parents would log in, click a link and download a PDF of their children’s report card(s).

It was the end of the third 9-weeks grading period; and the school district called us the morning that parents were starting to download their children’s report cards.

“The report cards are broken, they don’t look right,” said our point of contact, “Fix it! Fix it! Fix it… Fix it!”
I’m taking some poetic license paying homage to a line delivered by Fry in Futurama; here’s the Youtube clip.
The entire system and setup was new to my small team (myself and another person); but my team was assigned to the client, so we were tasked with fixing it.

Another person, somewhat familiar with the likely setup, showed us how to access the application and how to update the code.

The application was written in PHP, a language not overly familiar to me nor my co-worker. There was source code in SVN, but a long-standing habit of past maintainers was to upload files directly to the server; or maybe even edit them directly.

So for intents and purposes, the only canonical code was that which ran in production. To get to production, we had to VNC to a local Windows server to then VNC to the district’s Window’s server.

Once on the machine within a machine within a machine, I could use vim or joe; at the time, I had only minimal familiarity with either one. I chose vim because it was modal. I had less of a chance of “making a change and accidentally saving it” with vim. For those not familiar, in this instance of PHP, whatever change you made to the file immediately took effect. There was no need to restart the web-server. It just took effect. Which proved to be a useful “constraint.”
My coworker and I started exploring the code, and it was not immediately obvious how to fix the problem. Methods were confusing, poorly named, and doing too many things. Variables were terse or opaque. As the lead, I laid out our game plan.

One of our computers would make the changes, the other would be dedicated to vetting our changes. We’d use my computer (by way of the nesting dolls of VNC clients) for editing and his for vetting.

On the remote server, we would install git. Once installed, we’d initialize a local git repository for the code base and make our first commit. I would then refactor a bit of code, my coworker would check the new PDF to see if it was the same or better. If it was the same or better, I would commit the code. If it was worse, I would revert.
This is a critical thing to understand about refactoring; not every refactor need be better, it just shouldn’t make things worse.
We spent the next several hours holed up in a room puzzling through the PDF generation code.

Together, we’d make a small change; I’d announce I was preparing to save the file. My coworker would give me the all clear, I’d save the file, and they would refresh.

When they gave me the all clear, I’d commit the change. Sometimes, the change would break the PDF, so I’d revert that change.

Sometimes we’d bump a variable’s integer value up a bit to see what it did. Based on the results of that change, we would revert value but then rename the variable to something more expressive (e.g. from $h to $height_of_header for example).

Eventually, after several dozen commits, my coworker and I arrived at a solution that worked.

I can hardly express the elation of that day. Forever seared in my memory is the intense sense of accomplishment; we had both hopped into a very adverse situation, got our bearings, and carved a pathway to success.

I have carried that lesson forward in much that I do. To the best of my ability, when coding, I try to establish quick feedback loops. From there, I can then iterate towards an acceptable solution.

Also, during a “crisis” strive to follow processes that won’t leave you in a worse situation. Checklists are your friend. Saying out loud what you’re doing so others can verify your plan. Slow down so as to avoid compounding mistakes due to your haste.

I wish I could tell you what we did around fixing the ecosystem problems, but that wasn’t burned into my memory. I was also in the process of interviewing for what would be my next job. And you best believe I used that story as part of my interview.