A cluster I inherited came back from a Proxmox point upgrade with mon_target_pg_per_osd changed, the autoscaler in a different mode, and nobody noticed until a rebalance ate a weekend of client latency. That is the Proxmox Ceph experience in a sentence: it works until it doesn't, and when it doesn't you are debugging a cluster you didn't build, against a config you didn't write, through a GUI that hides the exact field you need to see.
The Proxmox integration is training wheels. It wraps cephadm opinions, pins versions, and owns your config lifecycle. You inherit every decision it made for you, including the ones that only surface during an incident. This guide is for engineers who want to own the storage layer: mixed hypervisors, bare-metal Kubernetes, S3 endpoints via RGW, shared CephFS for applications, or simply understanding what sits under your VMs.
By the end you will have bootstrapped a cluster with cephadm, designed pools and CRUSH rules deliberately, created and mapped RBD images, deployed redundant MDS for CephFS, and built an alerting posture that separates "ignore" from "wake someone up." We assume Linux nodes, dedicated disks, and cephadm as the deployment tool. ceph-ansible is deprecated — stop using it. ceph-deploy is dead. If you are still running imperative ceph osd create by hand, this guide is partly aimed at you.
:::note[TL;DR]
- Get the network split and OSD media layout right before
cephadm bootstrap; re-architecting live is a project, doing it right is an afternoon. - Use
cephadmwith declarative service specs stored in git. That is the whole point of the orchestrator. - Design pools, PG counts, and CRUSH device classes before writing data — resharding is online but not free.
- Enable the right RBD features and understand
exclusive-lockbefore you multi-attach anything. - Snapshots are not backups. Replicate off-cluster or you don't have backups. :::
Prerequisites
- Three or more Linux nodes (Ubuntu LTS or Rocky/Alma — see below), each with a dedicated OS disk and one or more dedicated OSD devices.
- A recent kernel;
bluestorebehavior and kernel CephFS clients track kernel version, so run something current for your distro. - 10GbE or better between nodes for the cluster network. 1GbE is malpractice for anything beyond a throwaway lab.
- Working time sync (chrony recommended), consistent hostnames resolvable in
/etc/hostsor DNS. - SSH root access between nodes (cephadm manages its own key, but you need a path to bootstrap).
-
podmanordockerwill be installed by cephadm's bootstrap script; you do not need to pre-install it.
Why You're Here (And Why "Just Use Proxmox" Is Not an Answer)
Standalone makes sense when you need the storage layer to outlive any single hypervisor platform: mixed KVM/VMware/K8s estates, RGW S3 endpoints, CephFS for shared application storage, or a team that wants to actually understand what is happening. The cost is real — you now own upgrades, config drift, and the 3 AM page. That is the trade. You get control and you own the consequences.
The one rule that matters: you should be able to explain every line of your ceph.conf. If you can't, you don't own your cluster — you're renting it from a GUI.
Prerequisites and the Hardware Decisions That Bite You Later
Everything downstream depends on two choices you make in the first hour: network topology and OSD media layout. Get these wrong and you will be rebuilding under load.
⚠️ TRUNCATED VERSION
This is an abbreviated cross-post. Full article (all config files, architecture diagrams, images): valtersit.com
🛠️ Partner Tools for Developers
ValtersIT curates partner deals for developers and sysadmins — VPS hosting, security tools, monitoring platforms and dev productivity gear. No filler, no affiliate spam.
Top comments (0)