On modern web applications stability

I don't like how modern web applications are built. Many of the web applications
are too unstable, That you can't imagine having the system running without a
team supporting it. The fact that we try to automate manual processes then the
automation needs manual intervention defies the purpose. Some companies has an
army of developers if they were to do the business by hand they would make a
better job than the programmed system. There are many reasons for this
situation. One of the reasons is the excessive use of third party dependencies.

Lets take a look on a basic modern web based system, There are several layers
on software running on the machine, starting from firmware to your business
logic.

I don't think this is very helpful to understand the gravity of the situation.
There are many actors that are not considered in this picture. Layers are also
missing because they are implicit in other layers. Lets expand these hidden
layers and actors. It will help us understand better why that small Nodejs or
RubyOnRails application we wrote isn't just one layer in this picture.

Here are the layers we added this time:

• System core utilities
• Other processes your application depends on like "memcached, Redis, MySQL, Postgres...etc"
• Third party code your application depends on an ORM, Template engine, pagination library, a library that pads your string with spaces just because.
• Server applications that sits in front of your code handling HTTP and response compression...etc.

For each of these layers there is at least one team responsible for
maintaining it.

Again, We missed other layers and people in this picture. Most of the
applications are using external SAAS providers for logs or monitoring or bug
reporting or provide parts of the system functionality that can take more time
to build by the company team. lets add them to the picture along with their
teams.

This picture is for one application, I won't expand it to a whole system with
different services and programs that is the reality of all companies.
Lets stick to one application for the sake of simplicity.

So here is the first point I want to make: With every service you use you're not
just a user, This service is now part of your application, You are held
responsible for it's behavior and misbehaving. You will inherit bugs in their
system. When this service team is affected by COVID-19 and get reduced to the
point where they can't fix issues you will be affected too. When They get slower
your application will get slower too. When their service is down your
application will experience malfunction too, Your system and theirs is now
connected. So add external services integration cautiously. By adding an
external system you're putting your trust in this service team and their ability
in delivering what the service is promising now and in the future. This is not
an easy decision and it should be treated as such.

Now lets move to the direct dependencies of your application. If you're using
any modern programming language it'll have a way to package code into reusable
format that could be reused by other applications. one package can use code from
other packages, these packages can use other packages and so on like a tree.

With every package in this tree we depend on the code inside this package and
the team that maintains it. A freshly generated rails project depends on 74
packages for ruby and Yarn list that lists JavaScript dependencies output 3102
lines, that's 3176 packages with teams maintaining them and bugs and new
versions all the time.

This is wrong for many reasons. I will list some of them here for the sake of
clarity.

• You have put your trust in at least 3176 other developers. You have never met them, never talked to them, there are no guarantee they will continue to maintain this package. There are no guarantee they won't put code in their package to show ads in your terminal or code that steals your bitcoin wallets .
• You are not really using all of this code. When someone is writing an open source package it will suffer sooner or later from feature creeping You are probably using couple features of this package and don't need the rest, but you wanted the banana and got the whole forest now.
• With every package update you're inventing unnecessary work for yourself. New versions of packages are released all the time. Updating your project to get the latest bug fixes and features is usually what people do. Most of the time because of feature creeping these versions changes are not relevant to you at all, but you won't know until you read the change log. If it's relevant to your project you'll need to do an update. if something is deprecated or changed you'll need to change your code. So suddenly someone somewhere is telling you to change your code. That's part of the control you have over your code handed over to someone you never talked to or knew.
• When your programming language has a new release you can't update unless all of your dependencies are up to date. For ruby 2.7.0 for example some language syntax is now deprecated and shows warning when you run your project. So to fix that you either fix it in the package and open a PR with the change or wait for the maintainer to update it.
• When you encounter a bug in a dependency you will have to understand this package code, fork, branch, fix rinse and repeat. That requires a some cooperation from the library maintainer which is most of the time isn't possible because most of the open source projects are voluntarily maintained.
• Developing new features or modifying existing features are ordre of magnitude harder. You'll need to dig into the documentation of the dependencies looking for support for this little feature you want to add. That is if there is any documentation at all for that part of the code. Otherwise you'll have to dig in to the library code.

This is the second point I want to make: Using external library implies that you
trust the maintainer and you also inherit his decisions about using other
libraries and so forth. This decision should be weighed based on the benefit of
the library and how many of it's features you're going to use and other factors
like the maturity and how responsive is the maintainer, please don't use GitHub
stars as a factor in your decision it's misleading. And if the part you use from
the library isn't too big I recommend using the library to save some time and
effort upfront but make sure you get rid of it and implement the part you need.
An example of that is a pagination library like rails "Kaminari" if you're using
it to save you some time then sure. But keep on your todo list a task to remove
it and implement the feature yourself. An example of libraries that's hard to
get rid of it "OpenCV" This is something that reimplementing the part you need
probably will be a huge task so it can stay. You'll need to use your best
judgment to decide between these 2 sides of the spectrum.

I like to think of what I do as building an automated system, I would like this
system to run by itself, keep itself clean and healthy, doesn't need manual
intervention. If the whole team disappeared out of existence I would like that
system to work for a very long time without any supervision.

More code means more bugs for me to fix, by extension more code that I didn't write
means more bugs that I probably can't solve. This is dangerous and shouldn't be
taken lightly. Extending your code with external libraries or systems can cut
down effort hence the cost of development. But when this is taken lightly it
backfires badly.