Warning!!! I expect a lot of negative comments and hatred for being pragmatic and honest, and will be happy to receive them!
Kubernetes. There is so much noise about this system in the industry, from leaders to devops and developers. Everyone seems to be excited about it and there is a promise that Kubernetes will solve all your problems. But what is K8? The official website says:
Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
What the heck does it mean? Literally what it says here - K8 is designed to help with container deployments, upgrades, and scaling.
All about containers
K8 only works with containers, and only with Docker containers. It helps you to host them. K8 was not the first one to do that, there are many other players on the market. There is Docker's own Docker Swarm, Marathon, Amazon ECS, Azure Container Instances, HashiCorp Nomad and so on. And those are just for running Docker containers only. But why everyone shouts about K8? It's just because it became the most popular, the most reachable and has the biggest community. Does it mean it's the best on the market?
It's about cloud portability
K8 is the most popular and the most accessible container orchestrator, and is offered by all the major cloud providers and more, for instance:
- Azure Kubernetes Service from Microsoft
- Google Kubernetes Engine
- Managed K8 from Amazon - Amazon EKS
What this means is that at least in theory you are not tied to a particular cloud vendor to develop your applications, provided that they are containerised. In theory.
It's not portable between clouds
Cloud portability is a big myth and it just doesn't work. What K8 promises is that you can just take your containers and run anywhere, on premise or in any cloud. And this is absolutely true. It works very well in practice too, provided that:
- Your application doesn't store any data
- It doesn't do anything with a network, at least it doesn't accept any incoming traffic
- It doesn't require any monitoring
- It doesn't use any other PaaS services
- Fully stateless, has no interaction with anything outside of itself
And those are just the most obvious points. I haven't seen many applications that don't need any of those, maybe only the ones in starter tutorials or tech talks.
This is not a development framework
Against popular impressions that K8 is needed by developers, it's actually not true. K8 has to do nothing with code or application, it's a pure infrastructure automation solution. There is nothing to code specifically for K8 or anything to take into account. K8 runs containers, that's about it. It's a DevOps solution.
You don't need any special tools to develop for K8, locally or remotely. As long as your application can be packages as a container it doesn't really matter. There is nothing you can use in code to take advantage of K8 services. There are no services in K8 for applications to use. Are you a K8 developer? Bullsh*it, there is no such thing.
Setting it up is a pain in a bum
Setting up K8 is a real pain. Especially if you are installing it on premise. Of course the set up procedure is scripted and easy, but what you will get out of the box is just a starter template you need to work on for a long time to add services that can actually support your application(s) in real world production scenarios. This is why K8 is usually used in the cloud, when a large company provides a managed instance for you. Just because setting it up yourself is a massive overcomplicated pain. And you can easily screw things up bringing down your whole cluster.
It's designed for small workloads
Surprised? Yes, K8 is not a distributed system. Funnily enough, it's a centralised, slow and not designed to scale. It's a heavily centralised system. It has API server in the middle, and agents called Kubelets installed on all worker nodes. All Kubeletes talk to API server all the time for various administration reasons. Apparently, the more worker nodes you have, the more traffic and pressure is required for a cluster to operate normally. Of course you can increase the amount of API server nodes, but that has certain limits and they are pretty easy to reach.
Deployment is hard
In order to deploy anything to K8, you need to create a deployment resource. This is somewhat large and clunky yaml file. In practice, no one is using native K8 features for deploying services, because of it's rather ugly limitations:
- There is no templating (i.e. using simple environment dependent variables)
- No way to describe dependencies on other services
- No way to describe an application, you can only describe a single service, which is not that useful.
To overcome those limitations, people have started creates systems on top of K8 just to deal with simple deployments, and there are so many of them:
- Helm package manager, which is one of the most popular ones. It's actually beautiful on the surface, however is a mix of yaml and go templating languages and quickly grows into an unreadable nightmare.
- GitKube
- Kapitan
- Kustomize
- Kontemplate
- and so on...
The fact the amount of these "tools" are growing and that companies are creating their own custom tools to do similar job, because built-in just don't deliver, doesn't make me trust this system very much.
But it's an industry standard
This is probably the major argument towards K8. It's open, understood (somewhat) by many, and isn't owned by a single company. I'm not sure this is a good argument though when it comes to success.
Data Storage is non existent
You can't store data in K8, there is simply no way to do it. It's not supported at all. Support for volumes is laughable, it's just a way to emulate a disk by mounting it to a remote storage over a network. Yes you can mount a folder to something like Azure File Share or S3 bucket, but performance will be extremely bad. If you run on premise, you can probably use some sort of physical disk, but it's a huge pain to set it all up and make it reliable.
The only way to store data is to use external databases, but those have to run somewhere as well. And you can't run them in Kubernetes, because there is no storage support, so you have to rely on external providers running on bare metal or PaaS services (SQL Server, DynamoDB etc.) which again doesn't make it self-sufficient and cloud portable.
You need an army to support it
It doesn't come cheap and can't be just used by developers.
What is it good for?
To be honest, I have no idea. Can you think of a real use case?
If I need to write a stateless service, that is scalable, and I do care about productivity, price and delivering business goals, I'd go with serveless functions (AWS Lambda, Azure Functions etc).
If I need to process large amounts of data in a long running background process, I'd go for batch solutions like Azure Batch or AWS Batch - no infrastructure management and much lower cost.
If I need to develop a service that needs to work with a lot of data and performance is critical, go for Service Fabric, which by the way can run Docker containers too, amongst many other things.
Hosting a web site? I'd be crazy to host this in K8, because PaaS solutions like Azure App Service are next century tech in terms of features comparing to K8 where you need to build everything yourself for another year.
Do I need to host containers? If so, Azure Container Instance or alternative from other providers is a way forward.
This post was originally published on isolineltd.com on April 25, 2019
Top comments (17)
This is good to see a practical view about Kubernetes. Yes I agree to all your points. In my opinion Kubernetes is not for all applications.
If we want our applications to be running all the time, so that we don't lose revenue to our competitors then Kubernetes will help solve that problem. This purely works for e-commerce ordering systems.
If we want our applications to have the behavior of self healing, I think Kubernetes solves that problem.
With Azure Functions we can develop as many services and have them scale as required, but it comes with a cost. The consumption plan does not have control on the region where the function would be hosted. This would add some concerns to teams where there is strict requirement to maintain services with in their geographic region. But recently Azure Functions came up with Premium plan which is in preview mode and it might solve some geographic region requirements through VNET access. I am not sure about AWS Lambda.
You're 100% correct and pragmatic my friend. However, K8 still feels like a pain to me. Why not looking at "more mature" solutions, like Service Fabric?
Service Fabric also does not suit well for small applications. Whether it is K8 or Service Fabric it is not for all. For example if we create an instance of Service Fabric in Azure, you can see the long list of resources which gets created and if not configured properly it adds to more hidden costs which will come as surprise while seeing the bills.
But most of these resources don't cost anything. You are paying for compute and disks on the VMs, those can cost as little as a web app. The resources are created explicitly so you can see what it consists of and customise to your needs if required, otherwise leave it alone ;)
We develop for Service Fabric with an 'on-premise' first mindset, since most of our customers still prefer that. Then when moving to the cloud, service configurations can drive substituting in cloud services (i.e analytics or data persistence) that make more sense for the cloud environment.
I respect your opinion and most of it is true and relevant but few things you have told sound like grapes are sour because you can't reach them.
Thanks so much for this.
In-house we are utilizing Azure Service Fabric. Although the learning curve is still quite high for Service Fabric, we're finding it makes it easy to develop an on-premise solution that has all of the niceties of a cloud-based solution, just without auto-scaling (i.e. to scale we would have to install more server nodes).
When researching how to build our next generation of software, from our current monolithic, service-oriented architecture, I was turned off from K8 as the amount of new skills required for our development team was just too high. With Service Fabric we can start learning how to decompose our old application into micro-services and let the platform do most of the work for us. It's not perfect by any means, but I'm loving it!
What you said is definitely true. Kubernetes is not for smaller teams and fewer application deployments. Its not made for application that are not mission critical. Kubernetes was never made for Developers, unless you are developing something for Kubernetes itself.
But you got the title wrong, its a great friend when you know its true value and when you have opportunity to harness its power.
As the official website says, its an Enterprise Grade Container Orchestrator. That means its not made for small businesses, even if its free and readily available. You have few applications of your own with a few hundred users a day, a linux server with apache/nginx, php mysql will be sufficient. Kubernetes simplifies, complexity of managing and maintaing a large number of heterogeneous server nodes, by creation an abstration of a unified infrastructure which can be managed and maintained using an API, so the operations team don't have to worry about managing every application and its dependency for each node, Also it reduces the networking complexity by creating a Software Defined Cluster Network, which controls how your traffic flows inside, ouside and into the Cluster. It leverages the container runtime technologies to abstract each component of an Infrastructure. Consider a scenario where 100 different applications and services are deployed on 10 different nodes, 10 on each, and now you have one application that now requires more than available resources on server. you have other node which has more than available resources for that application to run, then you'll have to migrate the application, create apache/nginx configurations, configure ssl, configure DNS, etc. Its time taking and complex, unless you have some Automation is place using scripts, also the difference is os versions and types, and human errors (no matter how exper you are you can still make mistakes) will add to the headace. Now consider those applications running on Kubernetes cluster with 10 nodes. Each havin its deployment, service and ingress resource, One of those applications request more than available resources, new container will be automatically created on other node with available resources without any downtime and human intervention.
Obviously its not easy and Kubernetes is not for noobs, but when learned and implemented properly it can reduce Operations Engineer's server maintenance workload by a Significant level so they can focus on building efficient app containers, create error free,efficient, high availability and self healing deployments.
And I'd suggest anyone new to devops and system administraton, must not even think of Kubernetes, its a luxury that one gets to value after a lots of learning and getting their hands dirty on Linux Fundamentals and Containers, or it will disappoint/discourage you.
You write that there is no K8s cloud portability if:
"Your application doesn't store any data" - This is not correct. It is no problem moving a service which stores data between cloud vendors (= uses physical volumes).
It doesn't do anything with a network, at least it doesn't accept any incoming traffic - This is not correct. All K8s clould vendors allow you to use K8s ingresses (there are also other options).
It doesn't require any monitoring - This is not correct. All K8s cloud providers support liveness/readiness probes (it's a part of the standard). All cloud providers allow you to install Promethes + AlertManager for monitoring. And I'd guess that all cloud provider give you integration with their monitoring system "out of the box" (at least the provider we use, but I'd be extremely surprised if that isn't the case also with other providers)
It doesn't use any other PaaS services - That's actually a point. Using Paas services sometimes means vendor lock in.
Fully stateless, has no interaction with anything outside of itself - This is not correct. As stated above those service can store state in physical volumes and/or they can store state in external data stores - like an external database. And I see no problems in a K8s service using an external database as a service. That's simply a good design and it's portable between cloud vendors.
I've probably misunderstood your arguments completely, but one of the things I like with K8s is that you can move (close to) exactly the same deployment between on-prem and different cloud providers.
I do agree that K8s isn't even close to solving all problems, but it's a really nice platform for it's use cases - and they are many. I don't doubt that Service Fabric also is a good platform for it's use cases - and possibly a more mature platform. But currently it is extremely limited in that it is:
And yes.. I think one should utilize functions as a service, database as a service, containers as a service, etc. But this does not rule out also running cloud portable applications in a K8s cluster.
Btw: On of our use cases is running small Redis clusters in K8s. For our needs this is a lot cheaper than buying Redis as a service.
"(for all practical purposes) completely in the hands of Microsoft.
an uninteresting platform for the most of the opensource community"
These are legitimate concerns for many people, however for many paying customers they are irrelevant. We must be sure to separate our personal views from what is in the best interests of our clients. There is absolutely nothing wrong with being a 100% "Microsoft shop" in 2019, just as there is nothing wrong with being an AWS or IBM shop. In the real enterprise world moving between major clouds is extremely rare.
This was not meant as an argument against being a 100% Microsoft, AWS or whatever shop. The problem when Azure Service Fabric is an "uninteresting platform for most of the opensource community" is lack of access to competent resources, lack of solutions built to run on Azure Service Fabric and lack of a vibrant community around the platform. And this is relevant for paying customers, even though your personal views might favor a particular technology.
I'm pretty sure Azure Service Fabric is a good platform which Microsoft will develop further. I'm also pretty sure Azure Service Fabric might be the way to go for many solutions, but......
Microsoft just announced Microsoft Arc (ref forbes.com/sites/janakirammsv/2019...), and here even Microsoft uses Kubernetes in ways that would have been impossible, or at least extremely unwise, if the picture this article tries to paint of Kubernetes is correct.
Or to cite from the article: "Microsoft is also one of the first to bring managed data services to the hybrid cloud. Since these database services are packaged as containers and run on top of Kubernetes, managing them from the centralized Azure control plane becomes efficient."
So Microsoft has decided to deploy these databases on top of Kubernetes and not on top of Azure Service Fabric (directly).
What's hard about deploying a container storage interface plugin for your cloud (assuming that it wasn't already deployed for you) and having a backup system like velero to make stateful workloads portable? Look if you're a developer and just want to stay on that level that's cool. But FaaS has to run on something. VMs only do so much. K8s is a platform for platforms and I dare you to do what you can now a days with something like Ansible easier. There's a lot more to software than just FaaS too so don't think that's the be all end all unless your goal is to yield control over to a specific could, their management and price tag.
Those of us who have been in this industry a while have seen this sort of fad many times. Thanks for the "Emporer Wears No Clothes" takedown of this regressive technology that does little but generate lucrative but pointless consulting activity.
Actually -- Kubernetes does a bit more than generating lucrative but pointless consulting activity. One can even build a bank on this technology, ref. youtube.com/watch?v=azDS51am7W4 - or maybe a music streaming service backend, ref. youtube.com/watch?v=eFE-X8FlyLQ.
But I most certainly agree that Kubernetes is not "the endgame" and it will never be the right solution for everything.
I feel that you don't quite understand what Kubernetes is. You aren't criticizing actual problems, almost everything you list is a feature and not a bug. Kubernetes only makes sense in the context of Docker. Docker doesn't allow you to define port mappings and volume mounts inside Dockerfile (by design), so if you are using Docker - you need to find a way of doing that. Kubernetes is just the most popular solution among many others. If you are not using Docker - starting with Kubernetes will be absolute waste of time. But if you are - then you can talk about Kubernetes VS [your favourite Docker orchestrator].
Aha! You are biased. Same as I am. And it's not about technology.
First, I agree. Within 17 years of my career, I never had a problem where K8s would help. Even with developing cloud apps today, I still don't need it.
Then, I worked at Microsoft with Service Fabric before it became available to users. And it had much more powerful features and a framework. Surprisingly, AWS didn't have an analogue. Nor big corporations who are afraid of the cloud. That's why I'm biased - Azure spoiled me :)
Some corporations have to host things on premise, and K8s is the salvation, no doubts, rare cases. Other corporations are/were scared of the cloud and didn't want to lock-in to a specific vendor - they want a flexibility to save money if a different vendor can offer a better term and a contract.
What would you do as a vendor? You say that you use a standard, which allows to switch vendor any time. This is your weapon against the fear of executives, which make the decision, not developers. Executives don't know that this is not exactly true, but marketing is about perception, not capabilities.
K8s is good for cloud providers to acquire enterprise customers and grow revenue. Thus the hype.
Great article, thanks for sharing, at my company we are using normal JVM applications on Linux directly and for some parts Apache Mesos that from our point of view it’s a lot better than Kubernetes