DevLog, Feb 15 2020

Hi, I'm Josh and I'm on sabbatical.

Sabbatical in this case means that I'm intentionally unemployed and am taking time to work on myself and my passion projects instead of working. Like most people, I'm not actually lucky enough to work for an org that has a proper sabbatical program so in that sense I'm just funemployed - to be fair.

But I'm definitely burned out and need the time off.

I started my career in earnest around Summer 2011 and I've been hammering doggedly on it since. My most recent job had a particular set of challenges that, while not entirely abnormal in the world of software development, made me realize that I didn't actually know what I wanted out of a job - much less a career! - and that this has made me susceptible to working jobs I find frustrating or dissatisfying. Given the environment I put myself in I was set up to crash and burn hard. I'd say watching me crater would have been fun, but the truth is that people were worried about me. It was bad news bears.

I'm lucky enough to have built up enough runway over the last few years that I have the freedom to work on things for myself, and for the first time since college I'm truly giving myself the full space to investigate these things, complete some passion projects, learn a few lessons and set myself up for success at whatever comes next.

One of my goals for this time off is to "write more". Actually sitting down to write consistently is tough for me, and for that reason I'm skeptical about becoming a writer, but I've also been told I'm (comparatively) good at it and I'm stoked about building up skills and exploring hobbies other than software.

As part of that, I decided that I want to write a semi-regular DevLog. I figure this will get me "writing more", be relatively low pressure, and give my adoring fans some updates on all the Cool Stuff I'm working on. I intend to write larger pieces about most of these things - or at least some talks - but in the meantime this puts metaphorical pen to paper.

I started this sabbatical about a month ago, so this update is going to be pretty beefy. I'm likely going to make this a bi-weekly thing, so hopefully later updates will be less verbose.

Meetups

I've been making an effort to go to more meetups recently. I used to go to them all the time in my early career, but about the time I moved to New York (2014) I stopped regular attendance. Now that I don't have a job to occupy my time, I want to try going to them again, become more connected with the communities I want to engage with, gain new skills and knowledge and perhaps even make new friends.

One of my favorites so far is the New York Emacs Meetup. It's a small meetup, with about a dozen people in attendance at any given time, and all of them are the kind of people that could get into lisp. The folks that run it are really chill and, if nothing else, the sheer enthusiasm for emacs that everyone has is refreshing. If the emacs bug has bitten you, I highly recommend it.

I've also been attending Python Project Night at the Microsoft offices. The format is a little different from talks-based meetups: when you attend, Microsoft gives you a temporary badge to get you to a conference room, where there's a big crowd of at least a dozen other Python programmers. While it's been a good way to ensure that I spend at least a few hours here and there on my many Python projects, it's also a great place to go to talk shop and pick peoples' brains on problems. So far it's very mentorship-focused.

It's definitely interesting coming to these meetups as a senior-level developer. I've been mildly surprised to find that, in these conversations, I have a unexpected amount of knowledge and experience to share. Part of me yearns for experiences that truly challenge me - which may be hard to find at generalist meetups - but I also enjoy teaching others.

Something that's been a bummer is that so far I've seen very few women/enbies at these meetups. I don't think either of the aforementioned meetups are being intentionally non-inclusive - after all, both of these aforementioned meetups have strong codes of conduct. It's possible that women are turned off by Python and emacs 😉 but I suspect that this is a more pervasive problem than that. I also worry that women that want to program in Python are missing out on the otherwise great resources that Python Project Night's mentors can be.

(Do you know of more inclusive meetups? Perhaps you have Opinions on why these meetups are so male-heavy? Please drop me a line.)

IHydra

Shortly after quitting my job, a friend referred me to a workshop at a video art hackspace in Bed-Stuy. I attended, and became inspired.

At this workshop, an artist named Olivia taught us how to use a project she wrote named Hydra. Hydra is a video synthesizer with a JavaScript-based embedded DSL that uses WebGL to render to a canvas. With this DSL, you can create neat and interesting procedural video, and in a REPL you can use it to create real-time performed experiences with a technique called livecoding.

You can try this yourself right now with the Glitch-based editor. After you click this link, it will ask for permission to access your microphone and/or webcam (say yes, trust me), and then load a random sketch. You can edit the sketch - try changing some of the numbers! - and press ctrl-shift-enter. This will update the sketch in real time.

Cool, right??

Of course, I immediately made the connection to Jupyter. Notebooks are imo, despite the naysayers, a pretty solid tool for interacting with code in real time. The experience of mashing ctrl-shift-enter to evaluate a buffer is a lot like pressing shift-enter in Jupyter to evaluate a cell. It's definitely more industrial grade than the Glitch editor, but the vibe is the same and in fact maybe industrial grade is desirable here.

I thought it would be fun and cool to build a Jupyter kernel for interacting with Hydra. It took way longer than I though it would - a long weekend vs nearly a month of semi-regular hacking - but it's been very rewarding. After being alienated from my labor for so long, it feels great to build something for myself from beginning to end. I'm also learning about how to write Electron apps using React.

Electron and React might seem surprising given the goals of this project, but I promise that given how Jupyter and my project - named IHydra - are architected, it makes sense.

Jupyter follows a client-server architecture, which is how it's able to run arbitrary non-Python backends, called kernels. Jupyter's server communicates with these backends via ZeroMQ. When a notebook gets loaded, the web UI for Jupyter tells the server to start the appropriate kernel. The server then starts the kernel as a child process with the parameters necessary to connect over ZeroMQ, the kernel boots up and connects to the necessary sockets, and everything starts working.

Hydra, meanwhile, is an embedded JavaScript DSL intended to be ran in a browser's global scope. When the Hydra object is instantiated it attaches the variables accessed in the editor onto the window object, and code ran in the Glitch editor is literally eval'd. Moreover, this DSL depends on WebGL and canvas. This means that, wherever our code runs, it needs to run in a browser-like environment.

I could have created a system that would have sent JavaScript over the wire to a proper browser window, with my kernel launching an Express server. While such an architecture would allow for some really interesting collaborative experiences, sending raw code over the wire like that makes me nervous.

So, I decided to use Electron to build my kernel. That means when Jupyter launches my kernel, it runs an Electron app, which then opens a window with the necessary browser-like environment. This means that the kernel must be used locally and that an Electron app is now involved, but it also sidesteps the serialization issue that the browser approach would have involved.

In order to implement this, I took the code for IJavascript, a kernel that runs in Node.js, forked it, made it run in Electron, and started hacking in all the other stuff I needed.

It turned out what I needed was way more than I originally thought. For instance, in order to use my kernel, there needs to be a way to install it, right? And because this is an Electron app, that would mean it would need a GUI. Moreover, repurposing IJavascript meant that I had to set up the Electron dev environment largely from scratch. This has led to much more dev time than I expected but has also led to me becoming pretty decent at React and having an almost annoying amount of knowledge about Electron. Right now I'm learning about flux, so that I can nail down the layouts of the UI I'm building.

The project is still in-progress with probably another week or two of off-and-on development, but the end is in sight. I have issues completing projects, so I'm hopeful I can actually ship this one. Keep an eye out - I'll have more to share soon!

Emacs and org-mode

Roughly six months ago, an SRE at my job saw an elaborate markdown-based To-Do list on my desktop and asked, "have you heard of org-mode?"

I told him I'd tried emacs in college and that I quickly moved to vim. He told me about an emacs "distribution" named spacemacs which superficially pretends it's vim. I installed it, gave it a shot, and fell in love. I've been using it ever since, and I'm getting ready to take it to the next level.

I have fairly severe adult ADHD. This has a big pile of effects on my life and I could write about ADHD all day. For instance, I have very little working memory, I constantly forget what I'm doing, and I have a difficult time planning ahead.

The basic strategy for mitigating this particular cluster of issues is to build a system for tracking and managing work. For many people with ADHD, myself included, that means taking a notebook everywhere, writing down everything, and following some sort of productivity system or strategy.

org-mode is a lot of things: it's TODO list software, note-taking software, a spreadsheets system, a literate programming framework, a static CMS and much more. Ultimately, however, org-mode seems intentionally tailored for implementing GTD.

Getting Things Done is a book by David Allen and a popular methodology for staying productive. David Allen developed the system outlined in this book while acting as a productivity consultant for high-powered executives. It's a relatively complicated system as compared to, say, bullet journaling, but also contains a lot of features.

While there's no generally accepted and used software stack for implementing GTD, users of org-mode have probably gotten the closest to living the dream. Many of the features of org-mode, such as capturing and the agenda, seem particularly tailored to this system.

One of the really cool things about emacs (and by extension org-mode) is that emacs is less a configurable text editor application and more of a framework for writing your own editor. I like to compare emacs to Node.js - Node.js isn't a webserver itself, but ships with the building blocks to let people construct their own servers using JavaScript. Emacs has a similar vibe.

When emacs boots up, it looks for a folder called ~/.emacs.d/ and a file called init.el within. This file, and all other .el files, are written in a fully featured lisp dialect called "emacs lisp" or "elisp". These files can be structured into complex systems - for instance, the spacemacs repo has a file layout that looks more like a full software project than a mere config.

This project, then, is two-fold. On one hand, I'm learning elisp and org-mode and writing my own productivity management software on top of emacs. On the other, I'm getting myself more organized and using this system to stay productive and make sure I'm working on the right things at the right time.

This project is still in its early stages and it will be a while before I have anything I can share publicly, but it's already paying dividends and I'm excited for where it's going.

db_hooks

In my last role, we used DataGrip for accessing our PostgresQL databases. DataGrip is OK. It manages connection data, has a sense of projects, and allows for evaluating snippets and expressions and getting results back in an interactive way. Unfortunately, I hate Jetbrains apps. It's nothing personal, they just don't fit my brain somehow. If you like DataGrip (or PyCharm, IntelliJ, etc) then rock it! But they're not for me.

I started a project during my time there to set up my own systems for managing common database connections, either in Python with SQLAlchemy (great for Jupyter notebooks!) or via the appropriate CLI for the database in question. I named it db_hooks, an homage to a small internal project at a prior employer that did something similar.

I've been slowly polishing it and adding features over the last six months or so. My current goal is to integrate it with sql-mode in emacs. sql-mode ultimately simply wraps the same CLI clients that db_hooks can already manage, and (as is the emacs way) allows for the kind of under-cursor eval that made DataGrip so useful.

I do have to ship a few changes to make that happen. I already got JSON config output working (since emacs doesn't have a modern TOML parser, the format I decided to use for config) and I just added keyring support (for looking up passwords in a secure manner). Now I need to build a system for managing my ~/.pgpass file, since sql-mode as far as I can tell doesn't know how to send environment variables.

This project is currently available on my GitHub, though it's a bit unstable. If you have this sort of problem, definitely kick the tires and let me know what you think!

Macbook Onboarding

I bought a macbook air a couple of weeks ago.

I already have two other machines - a Windows 10 PC that tends to be my daily driver (surprisingly good for dev!) and a fun but unstable archbook. However, I'm considering doing freelance work over the next few months (Need someone? Hit me up, my DMs are open), and I wanted to make sure that I had a machine that was casually capable of doing Real Work.

I livetweeted the initial setup when I picked it up, and when I very ironically had hardware issues and Apple wiped the disk while fixing it I also wrote down a checklist and was able to speedrun it in about 2 hours. If you happen to have similar development needs as myself, perhaps you'll find this useful!

Twitch

I've been getting into Twitch recently. I've been watching gamer streams of course but I've been looking into other content as well, especially programming streams.

Programming as a tag on Twitch is fairly quiet, with a few dozen people streaming at any given time, but I really like the format. In these streams, people share their code editor, work on a project, and talk to their viewers about what they're working on, answering questions and more. The highest quality of these programming streamers are on a team called Live Coders, if you want to see what it's like yourself.

Inspired, I set up OBS and it's actually really easy to get started! My setup kinda sucks - I don't have a desk, my only external monitor is a TV and I work off my couch, my webcam has intermittent hardware issues, and my headset is a $20 knock-off from Amazon. Even so, I can already stream with decent sound quality and I'm going to try to do more of this in the neat future. Subscribe for updates!

OBS itself turns out to be an interesting platform to work with. The thing that really enables interesting stuff is the browser source. OBS will spawn a headless chromium-based browser, load an arbitrary web page or file, and display it in a window, with transparency support. What this means is that you can use HTML, CSS and JavaScript to program widgets for your stream. This is how all the interesting interactive ones work. CalLowCreation has a lot of great examples of the kind of widgets and bot integrations that are possible on this platform. For instance, when he says the word "bugs" a bot draws a bunch of animated insects crawling over the stream. It's dope!

I haven't dug too deep into OBS browser sources yet, but I did hack up a POC for streaming peer-to-peer WebRTC into OBS and Twitch. The demo looks like butts but it shows that it's possible to take WebRTC and peer-to-peer streams and bridge them onto RMTP-based streaming infrastructure.

What's Next?

I'm still trying to get my bearings and project long-term. The emacs and org-mode work will help with that a lot. In the meantime, I'm hoping I can finish and ship IHydra, keep experimenting with streaming, try to write more DevLogs on a bi-weekly cadence, and figure out what comes next.