Proxmox VE Cluster Setup: Corosync, Quorum and Storage

Proxmox cluster setup: corosync network separation, creating the cluster with pvecm, QDevice for two nodes, choosing Ceph, ZFS or iSCSI, HA testing and common quorum failures.
Technical summary
Technology
Proxmox VE, Corosync, KVM, Ceph, ZFS, iSCSI, NFS
Topic
Proxmox VE cluster setup
Level
Intermediate–advanced
Prerequisites
  • At least two Proxmox VE nodes on the same major version
  • Basic Linux command line and networking knowledge
  • A dedicated cluster network plan
Related documentation
In this article8 sections

Turning a single node into a cluster in Proxmox VE takes a few commands. Building a production cluster that doesn’t reboot itself at midnight, however, depends on getting network, quorum and storage decisions right from the start. This guide covers corosync network design, creating the cluster with pvecm, the QDevice in a two-node cluster, storage choice, HA configuration and common cluster failures, for system administrators moving to Proxmox VE or building a new cluster.

Components of a Proxmox VE cluster

Three components work together in a Proxmox VE cluster:

  • Corosync: The membership and messaging layer between nodes. It decides which node is in the cluster and whether quorum is met.
  • pmxcfs (Proxmox Cluster File System): The configuration file system mounted at /etc/pve and replicated to all nodes over corosync. VM configurations, storage definitions and users live here. When quorum is lost, this file system becomes read-only; you can’t start VMs or change configuration.
  • HA manager (pve-ha-crm, pve-ha-lrm): Watches the resources placed under HA, restarts them and moves them to another node. It uses watchdog-based self-fencing so the VMs of an unreachable node can be started elsewhere safely.

The practical result: if the corosync network is unstable, that isn’t “a warning.” It brings quorum loss, read-only configuration and, if HA is enabled, nodes rebooting themselves (fencing). The cluster goes down because of the network.

Network design: corosync, storage and VM traffic

Corosync wants low, stable latency. Its bandwidth needs are small, but it is very sensitive to latency fluctuation. The Proxmox documentation explicitly recommends that corosync traffic not share the same physical link as storage traffic. Recommended separation:

Network Purpose Recommendation
Corosync link0 Cluster communication Separate physical NIC, 1 GbE is enough
Corosync link1 Backup cluster path Over a different NIC and switch
Storage Ceph, iSCSI, NFS 10/25 GbE, separate VLAN or physical network
Management and migration Web interface, live migration Separate VLAN; a separate network can be defined for migration
VM traffic Guest networks VLAN-aware Linux bridge, bond

Corosync (with the Kronosnet transport) supports multiple links; if one link drops, it switches to the other. Building this redundancy with corosync’s own link mechanism rather than a bond gives more predictable results, because a bond’s LACP and failover timings don’t always match corosync’s tolerance. If there is a firewall, verify that UDP ports 5405–5412 are open between nodes for corosync.

Each node’s name must resolve to its IP on the cluster network (/etc/hosts or DNS), and time synchronization (chrony) must run on all nodes. Clock drift looks small but breaks joining.

Creating the cluster and adding nodes

Create the cluster on the first node. The --link0 and --link1 parameters set which addresses corosync will use:

# On the first node (pve1)
pvecm create prod-cluster --link0 10.10.50.11 --link1 10.10.51.11

# On the other node (pve2), giving the address of a node in the cluster
pvecm add 10.10.50.11 --link0 10.10.50.12 --link1 10.10.51.12

# Status check
pvecm status
pvecm nodes

The node to be added to the cluster must have no VMs or containers. During joining, the contents of /etc/pve are replaced with the cluster’s, and guests with the same ID conflict. Back up existing guests first and restore them after joining. All nodes must be on the same Proxmox VE major version.

The same operation can be done in the web interface from the Datacenter → Cluster menu. The command line is clearer for specifying link addresses explicitly; in the interface it’s common for the management IP to be selected by default and corosync to end up on the management network.

Quorum and the QDevice for two nodes

In corosync, each node has one vote by default; the cluster needs more than half of the votes to operate. A three-node cluster tolerates the loss of one node (2 of 3 votes). In a two-node cluster, when one node goes down, the remaining node has only 1 of 2 votes. Quorum is lost.

The solution is a QDevice running outside the cluster. The QDevice is the corosync-qnetd service running on a small Linux machine (it can be a virtual machine, but not inside the cluster) and gives the cluster an extra vote:

# On the QDevice host outside the cluster
apt install corosync-qnetd

# On all nodes in the cluster
apt install corosync-qdevice

# On one node in the cluster
pvecm qdevice setup 10.10.50.20
pvecm status   # check the "Qdevice" line and the total vote count

The pvecm qdevice setup command connects to the QDevice host as root over SSH and distributes certificates; this access must be open during setup, and afterwards you can restrict SSH password login again. The Proxmox documentation recommends a QDevice for clusters with an even number of nodes. In an odd-numbered cluster, the extra vote can make cluster behavior unpredictable in some failure scenarios.

Comparison of storage options

Proxmox VE supports many storage types. The right choice for a cluster depends on the number of nodes, the existing server and storage infrastructure (SAN) and the acceptable data loss (RPO):

Option Shared? Strength Limitation
Ceph RBD Yes (distributed) HCI, snapshots, self-healing At least 3 nodes, fast network and disks
ZFS + replication No (asynchronous copy) Simple, local performance Data after the last replication is lost on failover
LVM over iSCSI/FC Yes Uses the existing SAN Snapshot support depends on version, test it
NFS (qcow2) Yes Easy setup, snapshots The NAS can be a single point of failure

In Ceph pools the default is size=3, min_size=2: three copies of each piece of data are kept, and I/O continues as long as two copies are accessible. If one node goes down in a three-node Ceph cluster, data stays accessible, but there is no separate node on which to rebuild the third copy; the cluster runs “degraded” during that time. Prefer four or more nodes for lasting resilience. Always separate the Ceph network from corosync.

ZFS replication (pvesr) is practical for small clusters of two or three nodes; the interval can be set at the minute level. When used with HA, changes after the last replication are lost at failover. Have the business owner accept this RPO in writing. For an architecture-level comparison of HCI and SAN, see the HCI or classic SAN decision guide.

HA configuration

To put a VM under HA, its storage must be shared (or replicated). The resource is added from the command line like this:

ha-manager add vm:101 --state started --max_restart 1 --max_relocate 1
ha-manager status

In older versions, which node a VM prefers was defined with HA groups; with Proxmox VE 9 this structure moved to the “HA rules” model (node and resource affinity rules). Configure according to the documentation for your version. When draining a node for maintenance, use HA maintenance mode: the command ha-manager crm-command node-maintenance enable pve2 moves HA resources to the other nodes.

Always test fencing behavior. Pull a node’s network cable or shut the node down, then measure and record how many seconds it takes for the VMs to start on the other node. The recovery time you tell the business owner is this measurement, not the brochure value. Untested HA isn’t HA.

Common cluster problems

  • Random node reboots: A common cause is the corosync link sharing the same NIC as storage or backup traffic. During heavy backup hours latency rises, the node loses quorum and the HA watchdog fences it. Look for link losses in the journalctl -u corosync output; if the timestamps match the backup window, the diagnosis is largely made.
  • Joining the cluster fails: Name resolution, clock skew, a different major version or existing guests on the node.
  • /etc/pve is read-only: There is no quorum. Check the vote count with pvecm status; don’t use pvecm expected without fixing the network problem.
  • Removing a node from the cluster: After the node is shut down, pvecm delnode pve3 is run on a remaining node. The removed node must not reconnect to the network with the same identity; reinstalling it is recommended.
  • Update order: Update nodes one at a time, draining VMs first; for major version upgrades, run Proxmox’s upgrade check script (e.g. pve8to9) and don’t proceed until the warnings are cleared.

For a comparison of Proxmox VE with other platforms, see the platform choice after VMware article; for setup and operations support, see the Linux and open-source platform services page. The detailed reference is the Proxmox VE administration guide.

Frequently asked questions

How many nodes does a Proxmox VE cluster need at minimum?

A cluster can be built with two nodes, but a QDevice outside the cluster is needed so quorum isn’t lost when one node goes down. If Ceph will be used, at least three nodes are recommended, and four or more for lasting resilience.

Can corosync traffic share the same NIC as the management network?

It works in a small, low-traffic environment, but if it shares the link with storage, backup or live migration traffic, latency fluctuates and quorum can be lost. Put at least one corosync link on a separate physical NIC and define a second link.

Is there data loss if HA is used with ZFS replication?

There can be. ZFS replication is asynchronous; when a node goes down unexpectedly, the VM starts on the other node with the disk as of the last successful replication, and later changes are lost. If this RPO is unacceptable, use Ceph or shared storage.

Can a node that already has VMs be added to the cluster?

No. On joining, the node’s configuration is replaced with the cluster’s, and ID conflicts can occur. Back up the guests first, then restore them with new IDs after the node joins the cluster.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek System Support

Server installation, configuration, incident response and scheduled maintenance. Active Directory, virtualization and backup are managed with the same discipline.

Knowledge Center

Other articles