Jeremy

Posted on Sep 29, 2023 • Originally published at noted.lol on Sep 25, 2023

Introducing Cosmos: The All-in-One Secure Platform for Self-Hosting

#selfhosting

As the developer of Cosmos, when I write about it I very often get to talk about the what. On the other hand, I rarely get the opportunity to talk about the how or even the _ why _. I am grateful to receive such an opportunity here, not only because Noted is an awesome blog to read, but also because it is a great platform to share the story behind the inception and development of Cosmos.

Most people know me by my Github and Discord handle: azukaar, and I have now been solo developing Cosmos for about 6 months (0.1.0 was released in April). In a nutshell, Cosmos is an all-in-one solution for self-hosting on home server , which includes:

A full reverse proxy (like Nginx or Traefik) with Let’s Encrypt HTTPS negotiation to manage domains and secure access to applications
A docker container manager (like Portainer) to edit, start/stop but also automatically update your containers
A community maintained app-store to easily install many applications in a few clicks
A suite of security tools like anti-bot and anti-DDOS mechanisms, as well as a strong authentication gate supporting 2FA for any container
A few other things such as an Identity Provider with an OpenID server to make your journey a breeze

The Cosmos home page

With Cosmos you can easily self-host any Docker based application (Jellyfin, Nextcloud, Navidrome, etc...) and secure them behind strong authentication, server hardening, etc... All of this without having to spend hours debugging a reverse proxy setup that does not work each time you install something! And while it does take care of the few annoying bits for you, it does not hide or lock away anything : you can still tinker with your containers in Portainer / the CLI and you can still manually look into the innards of each application freely to customize them.

If you are interested to hear more about the details of the functionalities, the details can be found here https://github.com/azukaar/Cosmos-Server a demo can also be found here: https://cosmos-cloud.io/cosmos-ui

While all those tools are already available separately, the first thing that makes Cosmos special is proposing those tools in an integrated, all-in-one and seamless experience. The reverse-proxy naturally react to your security settings, and can directly connect to containers rather than IPs/Hostname, the OpenID Provider uses the same accounts than the admin dashboard, the Anti-DDOS can prioritize your traffic based on what account you logged in with. This synergy is the foundation of the Cosmos experience.

Here's an example, when creating a URL in the reverse proxy you can directly pick a container in a list and it will even automatically select the right port for you

The second most important core value for Cosmos is the home-first approach. NGINX, Portainer, Traefik are all amazing tools but they have one big defect in common: they are not designed for you. They are designed for people whose jobs are to use those tools for 8h a day. Not only that make them un-necessary tedious and verbose to setup, but it also mean that many vital features might be paywalled behind very expensive price tags (affordable to companies, but not to individuals). Cosmos on the other hand, has been designed with one and only one target in mind: you. It does not intend to scale to a multi-billion company’s large-scale infrastructure. Instead, it has been designed for you and your family, to own and use a home server.

While highly customizable, The feature set of Cosmos is kept to a level of what is useful for you and your setup

Why did I start

The first feature I wrote for Cosmos (before I even found the name) was the file manager. Ironically, 6 months later that feature has still not been released! Initially the direction the product was taking was to propose an alternative to Nextcloud. I was mostly frustrated with the bugs and performance issues it had. Over time, something odd got me interested. It was not the “office tool” Nextcloud that allows you to open edit share office files, and collaborate on them. What I found most interesting was the app store, and the integration to the API it proposed. I thought it was very satisfying to setup the tool once, and then to allow other software to re-use the authentication, the file system, and so on, simply behind an API integration.

example of a budget app integrated in Nextcloud

Of course the proposition behind Cosmos evolved a lot since. Mostly for two reasons. First, Self-Hosting is still currently at a “baby” stage. There are a lot of challenges for a lambda person to be able to self-host, especially around security and reliability. Most self-hosted tools have weak setup, questionable security, and do not offer the right amount of scaffolding for a user who is not a professional to use the tool in a healthy and safe manner. Even when they do, it does not make sense for example for a user to setup dozens of HTTPS certificates (in the case of tools like Jellyfin of Nextcloud who natively support HTTPS).

The second reason is in fact the opposite: Self-Hosting is meant to be somehow a journey. Users enjoy tinkering with their servers, finding new things to self-host and setting them up. They usually like to go into a terminal and learn things, and they want to know what exactly is running on their server (Fair! Since, after all, it is theirs!). While Nextcloud and their very abstracted system solved the first problem well, it definitely clashed heavily with the second one.

Designing a system that allow both beginners and power users to enjoy themselves and find the level of abstraction they seek is a huge challenge from product and UX perspective that I had to tackle all throughout the development of Cosmos. I also believe that while Nextcloud was (mostly) a smooth experience to install and use apps, it being as gate-closed as it is has adverse implications especially around vendor-locking, that in my sense are against the basics of the self-hosting philosophy. All that thinking brought me to the dream I had for the future of the internet: an Internet rooted in its origin — a decentralized network collectively owned by its users, prioritizing the interests of the many in both consumption and production aspects.

The vision of the Universe

That dream, that vision, is what gave Cosmos its name: a place beyond the clouds, only accessible with the apotheosis of technological progress, that brings all nations together working tirelessly to achieve a single common goal. In this case, that goal is sharing information. In an age where the internet seems to drift away bits by bits from its users to become a tool of alienation, I saw the chance for a different future.

While there’s nothing that can be done to prevent corporation from organizing monopolies, there is a chance to help the internet become more than a high-way for the same few videos, the same few articles, and the same few musics to be consumed by everyone. There is a way to let users own the content of the internet, and shape it the way we want, the way we need, to express everyone’s individuality. Currently, the way the main platforms' algorithms work, the way Netflix cancels shows or remove them because they are not popular enough even when they have some large viewership (just to give an example that I think everyone can relate to), and the way decisions are made by either bots or business teams preemptively and mostly arbitrarily on a scale that impacts literally everyone’s life, is putting that individuality to risk. People are getting mind-numbed into consuming what is trendy, what the algorithms push to them, what makes them docile to repetition.

Furthermore, aside from the content’s diversity issue, there is an additional problem about its safety. Examples are plenty. The Last Pass hack of all their users’ credential data, Gitlab accidentally deleting all their data from their database, and having no backups almost costing hundreds of thousands of people all their hard work. This year alone, millions of users have seen their data breached by dozens of companies, such as T-Mobile, Google, Norton, MailChimp, etc...

If we were to somehow crack the code, and allow self-hosting to be a legitimate way to consume and share data for the many, then suddenly both those problems would be solved. Additionally, it would have the benefit of being both consumer and business friendly. The people consuming the data could own data again , smaller data producer (aka. Content creator) would have more personal platform to express themselves where they would feel less helpless against the cogs of the machine , and everyone would get the chance to make the internet their own. Larger companies would also find that not investing a large chunk of their earnings into infrastructure is beneficial. Finally, decentralizing the data would make it near impossible to orchestrate large scale hacking, lowering the risk personal data is currently at, both for the users and for the companies developing softwares.

Between the moment I had this vision, and the moment where I created the Cosmos repository, over 2 months had elapsed. I had to figure out how to somehow conjugate power, security, stability and ease of use in a single solution. How to make things accessible and seamless , while still keeping it exhaustive , fun to use for advanced users, and transparent for all to learn how things work. It was clear to me that I needed to re-think fundamentals, such as how Docker and HTTP are used, to design what would eventually become Cosmos.

The current state of Cosmos

A few weeks of work later, I had released the first version of Cosmos. Back then it was only the reverse proxy with the authentication gate. Having a fully fledged reverse proxy behind a nice UI was already kind of ground breaking on its own. NPM being the only alternative, adding the authentication on top of it would already make Cosmos a somehow attractive option for a lot of people to consider.

The first few users had it rough: not going to lie, it was not a disaster but there were a few hiccups for sure. I quickly realised that I had to speed up the release of very crucial features, especially the Docker container management system, for Cosmos to truly shine as a promising piece of tech. And so I started working on it. One week-end, I went a little crazy and pretty much rewrote docker-compose entirely to offer the powerful cosmos-compose alternative. Finally, Docker management got fully released, and so did OpenID support eventually.

The community of users has been growing steadily and I am still very excited to meet all the users who join the Discord server on a daily basis. At this point, Cosmos is full packaged experience from install to usage for users. First, you can find the apps you need in the app store:

Audiobook Shelf in the Cosmos App Store

Installing this app will take 2 clicks, and it is fully customizable if you want to change anything:

The Cosmos app store relies on Cosmos-Compose to offer a unique cocktail between a solid and a flexible setup. In the landscape you currently have two alternatives. Either you get a simple/bare setup but very flexible (CasaOS, Umbrel, ...) or you get a solid setup with HTTPS, and improved security, but in a very inscrutable black box (Cloudron, YUNoHost, ...). Cosmos sits in the middle, although most apps will fully work and will be secure from the get go , on rare occasions you might have to do a tiny bit of manual tinkering. The upside on the other hand, is you have full visibility and full control over your apps as if you had set them up from CLI with hundreds of lines of config files.

Of course in the spirit of control, you can simply import an existing custom docker-compose file directly:

Importing a custom Jellyfin compose

And thanks to its design, doing this will still allow you to use the full range of functionalities on your newly created container, just as if as you used the app store.

Talking about functionalities, you can of course fully manage your container, such as editing its labels and network settings:

And obviously, it wouldn't be complete without the reverse proxy, and its ability to quickly connect a container to the world with a full set of protection, security layer, HTTPS encryption. All this, in 2 clicks.

First click

Second click (Cosmos auto discover port and propose a URL based on the container name, which you can of course customize in the form)

This 2 click install, 2 clicks connect approach, while being completely customizable from A to Z, is the essence of what makes Cosmos the solution to bridge the disconnect between self-hosting as a whole, the power user and the beginner. It is why Cosmos is a suitable solution to the previously mentioned challenges. Why Cosmos seduced both advanced and inexperienced users alike. It makes things easy without dumbing them down. It let you appreciate every step for full transparency and let you learn things at your own pace.

On top of that, the strong accent on security makes Cosmos a safe platform for everyone to experiment, discover and learn. I have seen way too many people with extremely insecure setup lacking authentication, using HTTP only, having no backups, etc... And I do not blame the users, as they were never educated about those issues and solutions like CasaOS and Umbrel who target newer users conveniently do a very bad job at warning users about those things. Users also tend to settle for half solutions such as Cloudflare Tunnel that actually leave your setup very vulnerable.

One thing that I keep repeating is that keeping your server safe is difficult and requires you to use the full amount of tools available to you. People often think that because their server is not exposed to the internet it will be safe, and that may be they don't even need HTTPS, but they couldn't be further from the truth. Your local network is not safe. There are way too many devices in your local network that can be easily taken advantage of (Smartphones, Smart TV, PCs, IOT devices, ...). In fact, even your server is not safe from itself as you install multiple Docker containers that might not be completely safe. That is why Cosmos puts so much effort into making security accessible in a unique way. And no one should not take those point lightly.

With one single click you can enable authentication (think Authentik/Authelia), use the smart shield protection and/or restrict the container to your VPN

Ironically, the more advanced of a user you are, the more you will be crippled by your own experience and knowledge. I'm sure some of you have experienced the "setup fatigue" before. Knowing the amount of work you have to do to setup properly a new software makes you less likely to even try it. Or sometime, you like a new software but just not quite enough to go through the effort of "productionising" it. Let's be honest: while setting up Backups, HTTPS, Encryption, etc... on individual applications the first few times might be fun, I am sure a lot of you will find that actually the real fun part comes after; when the software is up and running and you can dive into its admin panel and discover its functionalities. As you become more familiar with it, Cosmos is designed to make setting up and managing multiple apps easier, without taking away the joy of experiencing new software as it gets up and running for the first time.

The future of Cosmos

Cosmos has come a long way, but the end of the road is still far. Aside from some of the missing, more immediate features such as monitoring and backups, 3 big milestones are left to be reached.

First of all, Constellation. Setting up a VPN to access your server securely without opening your ports is not only a crucial step for a safe self-hosting environment, but it is more often than not even a required one (for example because of CGNAT, but also because of mobile apps, since they can't support web authentication). The problem is, it is also one of the most complex things to setup. Not only it is very demanding (many different things to setup) but also very difficult to get right, and very manual too (ex. having to manually rewire DNS in your local setup).

Constellation aims to solve all those issues. It is a fully automated VPN that requires no maintenance , is setup in one click and (as for everything in Cosmos) is fully integrated to the rest of the stack: it automatically rewrite your domains to use encrypted tunnels , it allows you to restrict a reverse proxy URL to the VPN in (literally) one single click, and allow your users to manage their devices themselves in an easy to use interface where they just have to scan a single QR code. It is also naturally split tunneling all the routes defined in Cosmos for you, to not affect the rest of your traffic. Constellation is close to completion, and an open beta phase should be available soon.

Next milestone, going back a bit on the initial goals of Cosmos, is the API. In the same fashion as Nextcloud and its apps, I would like Cosmos to provide functionalities that make developping self-hosted apps easier. While I already started designing Cosmos in a way that achieve such result (from day one Cosmos has been sending authentication headers to apps for example) I want this to go much further, to make application development easier but also to improve user experience.

The reason why I am leaning that way is because s elf hosted tools are realistically still a poorly developed ecosystem. A large chunk of the community either started to self-host to replace Dropbox/Office, in which case they started with Nextcloud (or similar) OR they started self-hosting to do some other questionable things ( arr , arr!). That is why those two sides are the most developed by far, and offer the most advanced solutions. I think it is increasingly though, that people start seeing self-hosting for the wonderful opportunity that it is, and try to push it further. Unfortunately it is no small task , and that is why I believe that making self-hosted apps as easy to develop as mobile app , rather than being of the complexity of a SaaS app (with manually maintaining Databases, having to re-write a whole authentication system every time, and having a huge burden of having to maintain performance and security to a high standard) will boost up the possibilities for developers to propose myriad of new experiences and tools for users to enjoy.

Last but not least, I would like some day to leap into the infrastructure side of things. Having some more integrated web architecture that would be hosted to allow users to further deepen their reliance on their own server without hindering their ability to enjoy their everyday usage of services. I am purposely being vague on this as I don't like to over-promise, but I do hope that one day people will be able to get their domain-names directly from Cosmos (if they want to do so of course! As usual Cosmos will continue to foster customization and individuality).

Sign up for Noted

Maximize Your Homelab Potential with Self-Hosting and Open-Source Solutions.

                <span>Subscribe</span>
                <span>







            .nc-loop-dots-4-24-icon-o{--animation-duration:0.8s}
            .nc-loop-dots-4-24-icon-o *{opacity:.4;transform:scale(.75);animation:nc-loop-dots-4-anim var(--animation-duration) infinite}
            .nc-loop-dots-4-24-icon-o :nth-child(1){transform-origin:4px 12px;animation-delay:-.3s;animation-delay:calc(var(--animation-duration)/-2.666)}
            .nc-loop-dots-4-24-icon-o :nth-child(2){transform-origin:12px 12px;animation-delay:-.15s;animation-delay:calc(var(--animation-duration)/-5.333)}
            .nc-loop-dots-4-24-icon-o :nth-child(3){transform-origin:20px 12px}
            @keyframes nc-loop-dots-4-anim{0%,100%{opacity:.4;transform:scale(.75)}50%{opacity:1;transform:scale(1)}}


</span>



            Email sent! Check your inbox to complete your signup.

No spam. Unsubscribe anytime.

Closing words

If you've read this article in its entirety, THANK YOU!

I cannot express enough gratitude to the growing community around Cosmos. You all have been amazing , and I am so pleased to have worked on this project to what it has come to be today. This makes me want to work even harder toward a world where self-hosting becomes a viable solution for both users and business alike as an alternative to the cloud and SaaS solutions, while offering the same level of service quality.

You can find me on the Noted and Cosmos Discord, or on github. If you have any questions or feedback please let me know I would love to hear it! And if you share the same dream I have, and I think you probably do since you are here, every little bit of contribution matters, you matter! Let's make it happen together.

Happy hosting!