The Problem
Well, it is no news for us that developing software is expensive, and takes A LOT of time and effort, after all, we're talki...
For further actions, you may consider blocking this person and/or reporting abuse
Please don't do this without typescript and strong types.
At first config objects seems like a great idea to simplify and write less repetitive code. However, leaning too heavily on them can lead to rigid janky code to handle new cases and edge cases.
The dev experience becomes terrible when your coworkers have to read through hundreds of lines of config object parsing methods.
The time you initially thought you'd save is lost again to debugging and trying to find some undocumented setting or side effect that's not working as you expect.
IMO config objects are best when they are simple, self documenting, and no more than one or two layers deep
Edit:typo
This is absolutely a great call! Given the amount of data that can be condensed on such objects, skipping one or 2 props will eventually happen. Having typescript (or any other type system) is a MUST.
FYI, Meta-System is made with typescript under the hood, and I'm glad I used it.
Nice! If your config objects are created and consumed strictly by software (which I assume is likely the case if you're doing language translation) then these problems probably won't put as much burden on developers.
My feelings mostly come from projects I've implemented where developers were intended to create the configs. Not having strong enough types as a POC lead to needing a big refactor to improve discoverability of features for developers.
Common case of a POC mutating from "hey this could save us time" into "all my time is spent maintaining and documenting" 🤣
You're 100% correct, the idea is nice and will definitely save some time at the beginning, but with time and team growth, it becomes quite a mess specially if there's no proper documentation on how to use such files, and no one documents their work... Right now I'm working for a company that uses this approach, with old javascript split in different repositories, it is really impossible to debug and sometimes I take days just to find the right piece of code to update.
Using typescript for this seems a very good idea indeed ... JSON-schema maybe?
And I can imagine that at some point you'd also want some sort of a graphical editor, so that you can edit a "model" in a meaningful way (MDD, model-driven development) ...
I must say that when I saw the big blob of JSON that's needed for even a pretty simple app it gave me a bit of a headache, I can't imagine that it will really be a joy to "program" like that. Some sort of higher-level editor support would alleviate that ... and the ability to split the JSON declaration into separate files or "components".
The way I could really see this working is that you'd use a JSON model to generate the repetitive, boiler plate kind of stuff within your app, then the more "interesting" logic would still be coded in a conventional programming language.
Well this does look like MDD (Model Driven Development) all over again.
I can definitely say that we're on the same page :)
All of this is on the Meta-System's Roadmap, which you can check it here.
I see it! "Meta-System configuration can be split in multiple files"
Yes, you can make a Turing-complete programming language whose syntax is a subset of JSON. Your if statement example demonstrates this nicely.
But what problem does that solve?
You can write any program as a JSON data structure, but you could write that same program in C# or JavaScript and it will be far more readable and succinct.
JSON can be used to represent just about any kind of data, including a computer program, but it's not optimized for this. Like JSON, C# and JavaScript can be viewed as formats for storing data — the difference being that C# and JavaScript were designed specifically to represent computer algorithms while JSON was not.
Hey Sam, thanks for coming up with this!
Well, I do agree with you. For simple tasks, such as the
if statementin the article, it surely can take up more space than we would usually need with our own and beloved usual programming, and this is a good sign of the development of good languages for writing such rules in an easy manner.I feel like data cannot simply replace our traditional programming, this is impossible given that the data needs to be eventually resolved in code. However, the great usability of this comes when we can abstract good chunks of repetitive logic in a simple API. If we combine that with a predictable API across multiple modules/libraries, we have come up with a way to greatly reduce programming effort.
Now, whether we want to represent our logic with data or not is up to the developer, and it also requires a good understanding of the problem (and its complexity) we are trying to solve. If we have enough flexibility within the data model, or the code is organized enough, this is likely influence our decision.
Couldn't agree more
This is called an Abstract Syntax Tree (AST) but why would you want to write out raw syntax trees by hand?
If for some reason you needed an AST, you could probably create a real syntax for your language and have a compiler mode that emits the syntax tree as JSON.
You're kind of starting with the middle state of a language/compiler and working your way backwards. This idea is very much missing a "why".
The idea is to represent the code as data not to write the data by hand. Nor to read it in JSON format. This aproach enables projectional editing (term popularized by Intellij in their MPS) that has much more potential then traditional text editors.
Why? Because we can write the code once and generate multiple versions of binaries choosing implementation language that suites the best the target evironments with their constraints. For example runs in the browser and also super fast on the desktop.
I actually faced this when migrating a gui component from SWT (Java) to Javascript). It felt such a waste of time to rewrite everything.
But it's still a language.
What you put in the JSON is effectively just a serialized AST for a specific language. Yes, it's JSON, but it's not just JSON - the structures and values have syntax and form that goes beyond JSON, much the same as how source languages go beyond ASCII or Unicode.
Plenty of compilers transform the AST in the middle, applying optimizations and so forth - applying transformations to JSON isn't fundamentally different from applying transformations to an AST. It's a model, with a specific schema, and you transform it. The back-end of the compiler ultimately transforms the final AST into some other source language or binary form.
That just explains why we have high-level languages and compilers.
From what you've explained, all you're proposing is a high-level language with a syntax that happens to be a superset of JSON.
I mean, yes, you'd be able to parse it using a standard JSON parser - it would work sort of like a high-level lexer, but you would still need to build all of the remaining parts of a compiler. (And probably a highly specialized editor as well, since working with these bulky JSON structures in a text editor would be... less than appealing.)
I was going to say the same thing
I worked at a multibillion dollar company that had a single Google sheet control software operations. It was done this way so finance, sales, and C-level emoyees could configure our platform. Even if it was only dev controlling it, the adapter code for it grew exponentially with each update request. It ended up being a nightmare for everyone and eventually the company restructured communication to avoid that mess in the future.
Yeah, I think this emphasizes the problem of "good code" accessibility, It's hard even for a multi billion dollar company.
For this specific kind of issue though, I'd say the problem was with the lack of a predictable interface. Can't say for sure since I don't know the code...
I love this concept, but it would benefit greatly from being more extensible and allow for higher level features. Client needs such as "Render a paginated, filterable, sortable list of objects" is the kind of need which absolutely should be programmable by config.
I think that when you start implementing loops and branching with your JSON, you've gone too far in a weird direction.
You would love to see Meta-System in depth then! We made the core to be extensible as you wish, so you can implement such high-level features and use them in your JSON file. Come say hi at the discord, I love to discuss about this sort of things :)
I’ve been moving one of my react projects to a similar predetermined or templated structure using firebase and a “Core” collection. Basically like a schema for your schema. It’s a lot of work upfront but definitely pays dividends in the long run not having to chase down irregularities with how your data interacts as your project grows or changes. Also very helpful with data relationships as changing how those relationships function or are defined is as easy as changing the Core schema.
Word of caution though, abstraction always comes with the temptation to further abstract and can become a time sink for no actual improvement. I know because I’ve ended up there at times. Don’t let that dessuade you though as I’ve learned an invaluable amount about Js and React data handling in the process.
Don’t play too hard!!
I'm having a similar approach. I have a few predefined/builder components and I define layout and hierarchy in JSON. With a single instace of this codebase I serve 100+ completely different websites, with a 150k React bundle size, including styles!
This is an amazing work! Keep up <3
Nice article, and definitely you are on a point too little considered by the community that generate it.
I have touched code as data on a recent project. It may be interesting for you since it show another perspectives on that.
You may look at: github.com/HRI-EU/JSEN
You may also have a look at an article on that: ronpub.com/OJWT_2021v8i1n01_Ceravo...
For a complex and large code it will be never helpful. For smaller jobs or for product managers it can be helpful. When you already know the stuff why do you need so much simplification. Write good code with well documentation. That's is it.
So... I'm still failing to see why this isn't better?
function (functionInput1) {
const highestAllowedNumber = 3;
if (functionInput1 > highestAllowedNumber) {
console.log('too high!');
return;
}
console.log('you're fine.')
}
Unless you are parsing common language to create a program (Function that takes an input number X; If X greater than 3 print ... otherwise print ... ) I don't think JSON is a better way to describe code to be created, They are much harder to read and get invalidatet if you forget a comma somewhere, or has an extra comma at the end of you properties.
Like you Fabio, I'm self-taught, and I make a great effort to constantly improve myself to write GOOD CODE, but this json can become very messy as written code can... so why the extra step? Or have you thought about another data structure to use as a base? YML would be much clear, but a wrong indentation and kaput ..
Raí, you're absolutely right!
I'll start answering your last question: "Or have you thought about another data structure to use as a base?"
Yes, Meta-System uses another data structure, which I would love to show it to you, if you're willing so. Come say Hi at the discord link provided above :)
Now for the "why the extra step?". Well, one thing about data, is that we can more easily interact with it. Writing the JSON directly is hard? Well, we can then just have a drag and drop GUI for writing it for us. In the end, we're [sort of] writing code, but with a giant leap on accessibility.
I started tinkering with a similar concept for blazor. Write a bunch of json files to create components. Handles state fairly well. Much more form oriented than whole website type of thing. Allows some json to be injected so that users can create forms on the fly, validated and possibly passed to some api for processing.
I cringed all the way to the end. Whatever you are trying to achieve exists as "magic" in many opinionated frameworks.
JSON is not expressive enough for whatever you mentioned. It will look ugly and unreadable while making anything meaningful.
I would have cringed less if it was say YAML.
There exists a lot of configuration driven plug and play frameworks, all of it starts to bite you in the ass while straying away from what the author wants you to do.
I use java and rely on opinionated frameworks along with open source libraries like apache commons lang to do even string comparisons.
This basically accomplishes the same thing with a window for pulling out big guns if you want to.
You are on the road to making a Lisp-like system ;) . This article from 2006 goes on a similar journey starting from XML of all things to show the power of data as code which is the fundamental principle of lisp.
I find the proposition a little confusing I admit. Data is code, and code is data, always was always will be, which is more or less the fundamental proposition of IT from the outset. What new observation is being tabled here? I'm not so sure. There is also nothing new in abstracting code so it can be translated into another code easily. Which is all I see in the JSON example is that, a new code defined in such a way that other codes can easily be generated from it.
I mean by all means, such a thing is an interesting idea and if you can justify developing one, go for it and see if you can win uptake and adoption. But make no mistake code is data and data is code, and all you are doing is defining a new language: Meta-System.
If 10 years hence it proved to have been a success (the way Python has for example), it will just be seen as the new language on the block "Meta-System", perhaps with a new name by then or not. As it is just a new language on the face of it.
Or, by all means, let me know if I've misunderstood.
I would stop programming and go find another career if I had to deal with contrived indirection like this which I believe was ultimately conceived not to democratize the process of algorithm creation (which becomes an enjoyable process once one achieves flow) but to exclude “the unwashed masses” from “mucking up our pristine code wonderland” the original intention of abstraction-in-the-cloud seldom serves the noble goal of abstraction as labor saving device and is increasingly observed as a mechanism for implementation of INaccessibility (while perhaps not the direct intention of this author) is a theme often manifested in the web programming world where life is easier than all other manifestations of software engineering.
Meanwhile in the engineering world where software devs meet electrical engineers and summarily get dehumanized merely for being software specialists, things like the downsides of IF statements do not happen because the comparator is an elegant construct.
We’re often battling to justify our work to engineers of physical systems who feel as though we make nothing but fluff.
Then I get recommended articles like this…
No, procedural languages are beautiful, C,C++,JavaScript,Python,Rust and so on are beautiful and the art of programming is also beautiful.
JSON is also beautiful but I’ll not be making a programming language out of it.
Reminds me a lot of the 4GL's of the 80s, or of "Model Driven Development" (and it sounds a lot like the "no code" or "light code" movement too) ...
But, I do see the value of the idea of 'automating away' the repetitive, boiler-plate like parts of coding/development, and of replacing "imperative programming" by "declarative programming".
The point is - this has been tried in so many different ways, shapes and forms (see my opening sentence), but reality is we're still writing code "in the old way".
But, never say never, who knows how we'll construct software in 10 or 15 years from now - I can't really imagine that we'll be doing it exactly as we are doing it now, by endlessly coding our React hooks and components and whatnot - I do expect a large part of it to be done somehow in a more "high level", abstracted, declarative, model-driven way.
Oh I'm working on something very similar myself but a slightly different spec than you have here. I almost think of it more like js arrays written as a LISP. Very similar concept just written with less keys etc and using functional paradigms.
Awesome, I'll have a better look and join the Discord!
so a compiler? I don't see why we need json to write a compiler that operates on a high level easy to understand language.
Code is data. That's why arguing about layout is so silly. Ultimately we're just writing a code dom using a user friendlier syntax. Metadata as baseline code is what we need more of.
I think the idea itself can be good, I don't think I personally like this approach though. There are real languages out there that aims to provide features like this, for instance functional programming languages like Haskell, OCaml etc. The code is very much structured like data and provides much of the same benefit, while still being actual code. I can see a few problems with this implementation, at first glance.
Any programming language today, does not involve programming "against the computer", meaning, we don't write actual binary machine code. We program against so called abstract machines (the C++ standard for instance is a good resource to check out, if one wants to learn more). As such, generalizing this "code as data" is hard. Is it impossible? No. But suddenly we will be left with config files for config files, and that's just one level of indirection, that I could think of, off the cuff. With "normal" code, compilers do this, they are essentially code generators, with the difference being, when we need to, we can delve into the actual code and handle edge cases. With "code as data" this becomes pretty involved. Reading package.json for a NodeJS project today, is more cumbersome than following actual code for instance. Even though to the computer - everything is data - to the person reading it, it isn't.
Reading code is what we all do and what we all are good at. It's "easy" to step through, it's "easy" to build mental models (when it boils down to functions, or some part of some system). With code as data, like this, suddenly "following the thread" becomes harder. How would we debug a system like this? What code would be generated? The problem with many of today's "code generators" is that debugging them becomes incredibly hard. One of the many questions become, "what goes where?". Suddenly we've introduced (or at least made a problem harder than it was before) problems we did not have before.
We have no up front way of evaluating the product. We have no "debug builds" or "release builds" or "release with debug info" builds. We just have the spit out code from the system and that's that. This I think is a major problem. This will lead to bloat (like any framework today does, I don't think I've ever seen a framework that doesn't involve pretty substantial amounts of bloat), unforeseen inefficiencies and so forth. And in the end, I think we'd all be able to blow our own foot off with a system like this, just like we do with "normal programming" on a regular basis. Part of our jobs are fixing our own mistakes and I think code as data, doesn't prohibit us from making those mistakes.
It's an interesting approach none the less, even though it's not in my taste so to speak.
I’m exploring something similar to this in github.com/samhinton88/code-fold
Quite interesting project, and a related concept too. Keep up :)
The idea of code as data is an idea lisp had had for a long time. Clojure is a good example.
I'll tell you what. Our company had a similar idea to create what today would be called a "low code" solution: a drag and drop interface for building business web-applications. Page definitions are nothing more than JSON, you can think of it as a React application, but the JSX is expressed as only JSON. Several of these apps are deployed in production and do fairly well. The main benefit is that it is fast to configure something new and it is great for standardisation (since only a limited set of building blocks can be used).
Now what you might expect: as business requirements become more complex, things get nasty real quick. You will need to define all sorts of conditional logic inside the JSON tree to fulfill all those crazy business requirements. The JSON structure becomes a programming language of its own, which is impossible to debug, let alone do proper version management (merge conflicts...). When things get too complex you fall back to just writing JS again, so why bother in the first place?
Oh and by the way, since this is all proprietary stuff, you need to maintain the application builder as well. New developers need to learn this highly non-standard tooling.
This sort of stuff is great for letting the end-user create dashboards with simple configurable widgets, but building whole apps this way is a Sisyphean task and can be a huge risk. We're moving away from this approach and writing apps the 'old fashioned way' now that the whole React ecosystem has become more mature.
Um, just parse code into an AST instead. Terser 🐈⬛
Cheers!
I had the privilege of you convincing me of this concept on Discord. I'd love to see this pick up and widely used to build reproducable services. Keep it up!
Love this concept. I've been working on a "Code as JSON" project also which you might want to check out! github.com/satelite-digital/engineer
The example you give is not "Data as code", is just code. A more declarative aproach shouldn't have instructios
Hell no!
You are just reinventing what Json schema and hyperschema are already offering. This design has been utilized in the industry for almost two decade and commonly referred as config driven development.
This is just Lisp but without parens.