So right now Dark is made up of 100,000 lines of code, a hosted service with 6000 users, a 1600 member Slack community, 39 people who have contributed or intend to contribute to our repo, and 500ish "real" projects.
And—as of recently—just me to build it.
This devlog is a weekly post where I try to organize my thoughts around how to make Dark successful from here on. It's going to be barely edited and I can't promise it'll always be coherent.
Short term
My short-term priorities right now are:
- help the team get new jobs
- reduce costs and complete the transition to a smaller team
- improve stability of the infrastructure
- engage contributors
- improve onboarding
Jobs
If you're hiring, please take a look at the Dark team looking for new roles. They're great engineers/designers/product folk and they would make great additions to your company!
Stability
Improving stability is a interesting one. I'm not really an infra engineer, but I guess I'm about to become one. I just bought the Google SRE book and Charity Majors' Database Reliability Engineering.
I'm the only one on the pager, so I need to figure out how to make this thing stop going off. We've had a few recent incidents that I need to go through to shore things up, and there's one active ongoing incident that needs to be handled.
Right now, our queues are going a little bit nuts due to some intense activity that repeats every day, causing our DBs to slowly increase to 100% CPU over the course of about 12 hours, and then when it hits 100% it stops and everything goes back to normal.
So far I don't know what it is, I don't seem to have enough information to figure it out just yet. I've been rooting through every graph we have in Honeycomb to try to figure this out. So far I've disabled a process we used to keep the databases from blowing up in size, and I that might be the cause, so I'll see what happens today.
Isolation / Rust
Once I've got that done, I need to work on isolation and abuse prevention. Isolation is tough because we use OCaml which is single threaded, and because we didn't use the fake-multithreading libraries (Async and LWT) in enough places. So one option is to do that, and another option is to put some Rust as a control plane in front of the OCaml processes to kill them effectively. Or maybe both.
We have a bunch of operational stuff that comes from having our code in OCaml, so the move to Rust has been long sought. Maybe now I have the time to do some of this.
I've been vaguely thinking "oh I have some time, I could finally do the Rust rewrite I've been talking about" but that seems a little bit like madness. At the same time, I've been having lots of "second-system syndrome" thoughts like "oh, if I rewrote it in Rust, I could make the HTTP stack much more usable" and similar. I probably need to tamp that down a bit.
Contributors
We have a ton of contributors who have written a PR and are interested to do more. We don't have enough issues for them, and don't have enough contributor docs explaining how to fix more complex issues, so I'll be working on those in the near future too. And of course reaching out to folks to see what I can do to make contribution easier.
More importantly, though, is trying to help make a long term community, where the product and how we build it is centered on the community. It's a bit unusual to go from a team of 10 building a product, to this community model, so I don't expect we'll figure this out overnight. Definitely something I'm looking for input on!
Onboarding
It's still quite challenging to get started with Dark, and while we have some docs and some in-product tutorials, there's a ton to do still. I'll be focusing on this once I've got a handle on operational work.
Long term
The most important thing that I need to focus on is Product Market Fit. It's pretty clear that we haven't yet reached Product Market Fit with Dark, and that we're probably not all that close. This is a place where I'm spending a lot of thought.
It's clear that extremely Dark is good at a few things: it's probably the easiest way to put an API on the internet. It's extremely easy to have datastores and queues and HTTP handling.
The most obvious thing is to find something that we're close to, and try to get PMF with that. Should we try to pare down to be "Zapier but with a programming language"? Or really focus on the niche of React SPAs?
Part of the problem with Dark so far has been that programming is pretty general, and so it's been hard to make Dark really great for a tiny niche. It's very hard to predict what integrations and features people need for any app. After all, what do people build with React? Well, just about everything.
Similarly, it's very hard to support simple verticals. You can't be "the programming tool for the travel industry", it doesn't make sense.
A related problem is about how we position Dark. "Dark is a cool new way to program" doesn't tell anyone about Dark or how to use it. We tried to position very small at one point--just Slackbots--but there's not that many people trying to build Slackbots on any given day.
No solutions here yet, but this is what I'm thinking.
I'm new to dev.to, but I'm planning to write here every Monday, mostly stream of consciousness about what's going on with Dark. Would love thoughts and comments here or via twitter or in our community slack.
Top comments (6)
I've been following some of the content floating around about dark. I've not really had a usecase to go into anything more than what you and Ellen have put up on blogs, but at the same time I liked the message.
I think I liked the message most because dark solved a friction I could see between langauge-editor-deployment which is solved in a parallel way for my core work, data science in R.
RStudio have in many ways built for data science what you are doing now.
I'm not saying that this works for dark at all. I'm sure there are nuances, applications and use case differences in the vision that mix these up, but as a model, maybe it might be worth understanding?
I could say a lot about why I like Dark and what would appeal to me about it if I hadn't tried it, but the main draw is that I constantly have ideas that are not logically complicated, and the reason I don't act on them is because I know by the time I spin up a server, blah blah blah, I will have lost the excitement. This is especially problematic for perfectionists, who are likely going to try to solve things like scaling issues because they've ever even written a line of application logic. There is so much room for simple and effective utility out there that doesn't get done because there is a wall of devops in the way. I love the idea of blasting that out of existence. I think Dark has its own challenges based on where it is in maturation as a product, but it could dramatically change the equation of cost/benefit analysis of back-end implementation. When I watched your video way back and you talked about eliminating accidental complexity I was hooked.
I don't know how you package or market that correctly, but I hope the input is helpful in trying to figure out how to get more people in the net. Maybe you've already done exactly this, but the question that comes to mind for me is "What are people not doing because it just isn't worth the hassle of AWS?"
I like the idea of turning the idea of "no code" solutions into "just code" solutions
Love this! I really like this concept of "what aren't you building", might be a nice positioning to try.
Definitely does sound like you've got a tonne on your plate Paul. So far, things are looking promising and I did enjoy experimenting with Dark quite a bit.
As a person looking forward to contributing (learning OCaml at the moment), I definitely agree with your plan to move to a more community focused approach. The question on this that I think most would ask would be, what kind of license would this be under? From what I understand, it is still "closed"-source (ish).
Was about to suggest the from python and from javascript but seems to be on the roadmap. The links do seem to be broken though. My two cents would be to do something like From NodeJS/Express , which focuses more on what people are using to build APIs currently than it does on the general technology (Javascript). I'd love to hear your thoughts on this.
Unreal engine's blueprints seem to be the recurring thought when reading this. By far the most intuitive visual programming tool I've used.
On a similar note: perhaps the solution lies in collecting data on various use-cases and finding meaningful abstractions that would cover what most people want. I realize that it is easier said than done and personally believe that Dark has actually done a pretty darn good job on this.
What do you think of "Dark is the fastest way to build an API"?
I think I've typed a tonne and this comment will turn into a blog post so I'll hold it off for now. I'd be open to a zoom call later on (will message on slack) if you'd be down.
I'll be thinking more about the license in the future for sure, and I'll discuss it when I announce the open repo.
Would love help with docs like that - we threw up some barebone docs - but having someone dig in and design a great learning experience for people who know Python/JS/Ruby/etc would be awesome. The docs repo is open and would love ideas.
I like it! We've been toying with variations of that for a while. Developer tools are notorious for claiming "fastest" about everything and not really being specific about what it means. The other thing is we've been trying to be super specific that Dark is code (as opposed to something like Zapier).
In terms of positioning, we(Reify Academy) have found that positioning it “ Dark is an easy to discover backend language which allows you to focus on your business logic rather than configuration.” with specific target for Frontend developers and mobile developer who need a backend but don’t know where to start.
Frontend devs inherently want something visual they can see - that’s where dark fits perfectly and on top of that it eliminates headache of running server infrastructure.