Guaranteed unique; Or, why dogfooding can be taxing.

As I looked towards the future of PliantDb, I thought my next step was to begin working on the permissions system. I've been setting a goal to try to have Cosmic Verge running on PliantDb by Saturday so that when I give an update on the game, it will have had some meaningful progress. In reviewing my action plan, I wanted the native clients to talk to the PliantDb server directly over PubSub. To do that without fear of people doing something that could break the game, I wanted to restrict unauthenticated database connections to specific actions. For the demo, there wouldn't be any user accounts.

I spent some time working on a permissions system design inspired by AWS's IAM policy system. I'm delighted with how that API is coming along, and I'm excited by the vision that we have on how we're going to try to build the permissions system in a way that makes applying it automatic and straightforward while still allowing flexibility to do complicated logic as-needed. But, this post isn't about that -- I'll write up a summary once I've finished implementing the system. The reason for this post is to talk about a seemingly unrelated feature: unique views. As odd as it sounds, I couldn't bring myself to finish the permissions system until after I solved this problem.

The Problem: Guaranteeing Uniqueness

As I finished writing the lower-level part of the permissions system, I began looking at how the permissions would be managed -- through roles and groups. My approach to Cosmic Verge's development is the same as PliantDb's development: If I can design some chunk of code that can be reused over and over to build my project, I'm going to want to use that tool for the job. PliantDb's job is to store collections of data. These permission groups and roles should be implemented using the same schema objects that PliantDb users will be using: PliantDb needs to eat its own dogfood.

For these structures, I was going to want to have a human-readable name be their unique identifier. When reading these permission structures, seeing "Administrators" as the group name instead of "21674831" is infinitely more useful. In a traditional database, the first tool I would use for this would be to use this a varchar as a primary key. In CouchDB, if you don't specify an id when you create a document, it will automatically generate a UUID-style ID. However, you can also specify an ID at the time of inserting, and it will use that ID -- and for CouchDB, that can be any JSON data type. In PliantDb, to keep things simple and efficient, I decided to restrict the document IDs to u64s.

Another approach that traditional databases can use is a "unique constraint" -- the ability to have the database check that before updating/inserting any data, it checks that certain constraints hold true. In PliantDb, I had the idea to support "unique views," which would allow any View to restrict the view entries to one per key optionally. For example, in the PermissionGroup collection, I could define this view:

impl View for PermissionGroupsByName {
    type Collection = PermissionGroup;
    type Key = String;
    type Value = ();
    const UNIQUE: bool = true;

    fn map(&self, document: &Document<'_>) -> schema::MapResult<Self::Key, Self::Value> {
        let group = document.contents::<PermissionGroup>()?;
        Ok(Some(document.emit_key(
            group.name.to_ascii_lowercase(),
        )))
    }
}

Whenever a new PermissionGroup is inserted or updated with a key that already exists, a UniqueKeyViolation error will be returned.

The first approach solves a core desire of mine, but the second approach is much more versatile. Ideally, both would be supported by PliantDb.

Supporting Arbitrary Primary Key Types

I felt inspired to dive into the first approach: supporting arbitrary primary keys. I started at Document, changing the id: u64 to id: Vec<u8> to support an arbitrary number of bytes. I then added an id() method, which attempts to decode the value using the Key trait that Views already use. Unfortunately, this can error, so the effect of changing all of the doc.header.id references to doc.header.id()? started to take a toll on the readability of the code. Eventually, I decided it was too many question marks for a user to endure and backed out of this approach.

Despite having rolled back my changes, I may still reattempt to support this feature using a different approach -- but it will need to involve a new Document type, one that already deserializes the ID into a generic type.

The challenges of dogfooding

This moment is where the inspiration for this blog post came from: dogfooding a large project can be hard. As I stared at the back-to-square-zero code base, I started to be tempted to ignore this problem. After all, the only real problem arises under a heavily concurrent situation, and will PliantDb users really be adjusting their permissions in contentious situations? Probably not.

Each decision to push functionality down the line yields some technical debt. PliantDb is the core of the architecture we're trying to build in Cosmic Verge. To me, the area you want the least technical debt in are the parts that are at the "core" of your codebase.

This high amount of dogfooding will hopefully allow us to acheive these grandious goals, but it does come at the expense of needing to spend extra time on the core components to ensure the entire machine works. My break from the computer helped me remember a few important lessons:

First lesson: don't set arbitrary deadlines when "passion" is in the project description. This post started off discussing how I was trying to progress on the game itself before the next game dev meetup. Yesterday was one week from the next meetup. The motivation for the goal seemed innocuous: it's a game dev meetup; I kind of want to show progress of the game itself. But, the reality is that there's no real pressure to do so. No one is seriosuly expecting an entire database engine to be done in less than two months. I knew I could hit that goal. Heck, I even think I might still hit that goal. But that's beside the point: Setting goals is different than setting deadlines. My goal is to get Cosmic Verge on PliantDb. But, if I force myself to meet that goal on a deadline, I might end up with technical debt, but worse, it might come at the expense of mental stress.

Second lesson: dream big, but take each day one step at a time. With just PliantDb, my list of things I could tackle each day is immense. Add a game to it, and there's no way for a single person to cross the finish line for my lofty visions for both projects. The reality is that I can't make PliantDb or Cosmic Verge reach their fullest vision on my own. But, I can try my best to ensure I'm a step closer to that vision after each day I work. This works right now because I generally try to plan a few steps ahead of where I'm going. For example, right now the high-level PliantDb list is:

• Permissions
• Platform Trait
• Multi-user support
• Replication

I tend to take this approach to planning because I like to reflect on the remaining items each time I finish one. I want to be sure they still seem like the best next steps, otherwise I might want to spend some time adjusting my plan. When I was in the moment and getting frustrated, I had lost sight of this process and was focusing on delivering an arbitrary feature set by an arbitrary date. While I could take on the stress of ensuring I have something by that date, the much healthier approach is to take each day in stride and evaluate where I'm at closer to the meetup.

Third lesson: Me-time is needed too. If you've seen my Discord statuses over the last month, you'll know that my progress on PliantDb has been made despite an increasing amount of time I've been playing Factorio. I've been having some regular gaming sessions with friends, spending time with my wife, and making progress on a re-listen of The Licanius Trilogy. But, nearly every waking moment that wasn't a chore or hanging out with someone else was spent working on PliantDb.

This realization hit me as I sat to really enjoy the piano for the first time in several weeks. I play regularly, but lately, it had been only for 20-30 minutes every few days. I still would have fun playing, but I was usually playing to feel like I'm practicing. I started the same way this weekend, and after a couple of songs, I came back and sat at the computer. I thought I was ready to tackle unique views. Yet as I stared at the monitor more and more, I just realized that playing the piano more sounded better to me at that moment in time. I went back to the piano and played until my back was sore -- in a good way.

Those moments enjoying the music for the escape it was providing made me realize I needed to relax for the rest of the day and take the artificial pressure off myself.

Unique Views

Last night while enjoying my time away from the computer, I still found myself pondering the challenges of implementing unique views. It sounds simple at first glance, but it flips the responsibility of view updating on its head. Before this implementation, when you save a document, no views are updated immediately. Instead, when a view is queried, at that time, the view will be indexed, and results are returned. This means if you update a document 5 times but only access the view one time, the view's code is only being evaluated once. However, for a unique view to work, document saving must take the responsibility of the view indexer.

A little while ago, I noticed that you could define an associated constant in a trait. Implementors of that trait can be required to provide their own value. I haven't seen this used much in practice, but I immediately thought of using it for this unique flag for the view. The inline example earlier in this post shows how this works for the View trait in PliantDb now. I'm not sure if I'm going to keep this approach or change it to how version() is a function. For today, the constant makes more sense, but I also envision dynamic views that will need to be created at runtime in the long term.

Despite my fear of revisiting some of the first code I wrote for PliantDb, overall it was a pretty painless process. The view indexer already had the individual-document update logic in its own function, so it was easy to call it from the transaction executor.

Final Lesson: Worrying isn't worth it

I'm happy I got this feature done. Its journey to completion started weeks ago: when I was thinking of porting Cosmic Verge to PlaintDb, I had identified this as something I would want. But, each time I thought of it, I lamented it. I worried about how annoying changing that logic was going to be. I felt like it was going to make the already complex code much harder to understand.

In the end, it was much easier than anticipated. And, now that it's done, I'm excited at how much less "blocked" I feel on the project. All of the worrying amounted to nothing except stress.

So, instead of promising when the next update will happen, I'll just say I'm looking forward to giving an overview of the permissions system whenever I'm done. Until next time!