HCI or Classic SAN? S2D, vSAN and Three-Tier Architecture

HCI or SAN? Compare S2D/Azure Local, vSAN and Ceph with three-tier architecture by scaling, capacity efficiency, failure risk during maintenance and lifecycle.
Technical summary
Technology
Storage Spaces Direct, Azure Local, VMware vSAN, Ceph, SAN, iSCSI, Fibre Channel
Topic
Choosing a virtualization storage architecture
Level
Intermediate–advanced
Prerequisites
  • Current capacity and performance data
  • A virtualization platform decision or short list
In this article8 sections

The storage array’s support is about to end, and the hosts have aged too: will you renew the three-tier architecture, or move to hyperconverged infrastructure (HCI) that embeds storage in the servers? Vendor presentations usually answer this question in a single sentence. In the field, the answer depends on the growth profile, the number of nodes, the network and what the team can operate. This guide compares the two approaches on scaling, capacity efficiency, failure domains, workload fit and lifecycle, and ends with a decision tree.

The basic difference between the two architectures

In three-tier architecture, compute (hypervisor hosts), network (iSCSI/FC/SMB) and storage (SAN or NAS array) are separate tiers. The array is an independent system with its own controllers, RAID structure, cache and software; hosts access it over the network.

In HCI, each server carries its own local disks. A software-defined storage layer presents these disks as a single pool across the cluster and protects data by copying it between nodes. There is no separate array; software running on the hypervisor hosts does the “controller” work. The main examples:

  • Storage Spaces Direct (S2D) — the storage layer of Windows Server Datacenter and Azure Local
  • VMware vSAN — integrated into the ESXi kernel
  • Ceph — distributed storage integrated into Proxmox VE or used standalone
  • Nutanix AOS — the distributed storage layer of the Nutanix platform

The practical result of this difference: the network between nodes takes the place of the storage network. In HCI, every write is copied to at least one other node; that’s why 25 GbE and RDMA requirements come up more often. If the network is weak, storage is weak.

Scaling and capacity efficiency

In three-tier architecture, compute and storage scale independently: when capacity is needed, a disk shelf is added to the array; when processing power is needed, a host is added. In HCI the scaling unit is the node; each new node brings processor, memory and disk. In an environment growing mainly in disk (such as archive or file server data), this means unnecessary processor and per-core licensing cost. Some HCI solutions offer storage-only nodes or adding disks; ask about this in your evaluation.

Capacity efficiency depends on the resilience method. Values for S2D from the Microsoft documentation:

S2D resilience type Failures tolerated Capacity efficiency Minimum nodes required
Two-way mirror 1 %50 2
Three-way mirror 2 %33,3 3
Dual parity 2 50%–80% (depending on node count) 4
Mirror-accelerated parity 2 Depends on the mirror/parity ratio 4

In a classic array, RAID 6 or the vendor’s distributed RAID usually gives a higher net/raw ratio; data reduction (deduplication, compression) is built into the controller software in most arrays. Example calculation: for an environment needing 100 TB net capacity, HCI with three-way mirroring needs about 300 TB raw capacity excluding cache disks; the same need is met with about 120 TB raw capacity, excluding spares, on an array using 10+2 RAID 6 groups (100 × 12/10). Using parity in HCI increases efficiency, but lowers write performance and needs more nodes. Compare quotes not by raw TB but with the same net capacity and the same resilience target.

Failure domains and resilience by node count

In HCI, the failure domain is the node by default. Rebooting a server for maintenance means all the disks on that node are temporarily out of service. A cluster with two-way mirroring is unprotected against a second disk failure while one node is in maintenance. Planned maintenance is a risk too.

Number of nodes S2D vSAN (FTT=1 / FTT=2) Ceph (size=3)
2 Two-way or nested mirroring + witness 2 nodes + external witness Not recommended
3 Three-way mirror FTT=1 RAID-1 Works, no spare node for self-healing
4–5 Three-way mirror, dual parity FTT=1 RAID-5; 5 hosts for FTT=2 RAID-1 Recommended starting point
6+ All options FTT=2 RAID-6 Erasure coding options

In vSAN, tolerating n failures with mirroring requires 2n+1 hosts. In the vSAN ESA architecture, the number of hosts required for RAID-5 differs from OSA (ESA uses a 2+1 or 4+1 scheme depending on cluster size); check version-specific details in the Broadcom documentation. Two-node HCI clusters need a file share, cloud or USB witness for S2D, and a separate witness host for vSAN. Placing the witness inside the same cluster is a design mistake still seen in the field.

In three-tier system infrastructure, failure domains are separate: rebooting a host doesn’t affect storage resilience. On the other hand, the array is a single system; even with dual controllers, a controller software (firmware) bug or a faulty upgrade can affect the whole cluster. Two controllers, one piece of software.

S2D/Azure Local, vSAN and Ceph comparison

Criterion S2D / Azure Local vSAN Ceph
Hypervisor Hyper-V ESXi Proxmox VE, KVM, others
Hardware Approved solution/catalog vSAN ReadyNode / HCL Flexible, enterprise disks required
Network RDMA recommended 10/25 GbE Separate public and cluster networks recommended
Licensing WS Datacenter or Azure subscription Tied to the VMware package Open source, support separate
Operations PowerShell, WAC, Azure portal vCenter Proxmox interface or ceph CLI

S2D and vSAN are products put together by the vendor and integrated into the management tool. Ceph is very flexible, but operating it takes more knowledge. If no one on the team understands placement groups, the CRUSH map and rebalancing behavior, recovery after a disk or node failure will be long and stressful. Using consumer-grade SSDs in Ceph is also a common mistake; on disks without power loss protection (PLP), synchronous write performance collapses. For Ceph setup and network separation on Proxmox, see the Proxmox VE cluster setup article.

Workload fit

  • General server virtualization (AD, applications, file, mid-sized SQL): Both architectures fit; the decision is made on operating model and cost.
  • Branches and remote sites: Two- or three-node HCI is a strong candidate because it removes the need for a separate array and SAN switches.
  • VDI: Compute and storage needs that grow linearly with the number of users fit HCI’s node-based scaling well.
  • Capacity-heavy data (archive, large file servers, video): An independently scaled array or separate file/object storage is usually more economical.
  • Storage that physical servers also need to share: If non-virtual database clusters or other systems will use the same storage, an array is the natural choice.
  • Databases that need very low latency: Possible in both architectures. Between NVMe-based HCI and an all-flash array, the deciding factor is measured latency, not the architecture label.

Lifecycle and upgrades

In HCI, the operating system, hypervisor, storage software, disk and NIC firmware and drivers must be validated together. An incompatible NIC driver causes not only a network problem but also a storage resynchronization problem. That’s why HCI products come with approved hardware lists and orchestrated update tools (Cluster-Aware Updating, vSphere Lifecycle Manager, the Azure Local update process). After each node update you must wait for data to resynchronize; skipping this and moving to the next node means leaving two nodes degraded at the same time. In large clusters the update window gets longer.

In three-tier architecture, server and array lifecycles are separate: the array may be used for five to seven years while servers are renewed in between, or the other way around. Still, you need to track the compatibility matrix between array firmware, HBA/NIC drivers and the hypervisor version separately. Replacing the array is usually a large project that requires storage migration of all VMs.

Decision tree

  1. Does the existing array have more than 3 years of support left? If yes, keeping the three-tier architecture and renewing the hosts is usually the lowest-risk path.
  2. Do physical servers or other platforms also use the storage? If yes, you need an array; HCI covers only virtual workloads.
  3. Is the need for capacity growing much faster than the need for processing power? If yes, an array or a separate storage tier is more economical.
  4. Does the team not want to operate a separate storage system, and is a single console preferred? If yes, HCI comes forward.
  5. Do the number of nodes and the network meet HCI’s resilience requirements? If at least three nodes (a witness for two nodes), 25 GbE and, if needed, RDMA can’t be budgeted, HCI’s advantage shrinks.
  6. Has the hypervisor decision been made? The HCI product determines the hypervisor (S2D → Hyper-V, vSAN → ESXi). For the platform decision, see the platform choice after VMware article.

If you’ve decided to stay with three-tier architecture, continue with the Hyper-V shared storage: iSCSI, SMB3 and FC article to choose the protocol. We can work together on analyzing current capacity and performance data and on hardware selection within our hardware consulting and virtualization and data management services.

Frequently asked questions

Is HCI always cheaper than SAN?

No. HCI removes the cost of a separate array and storage network, but mirroring-based resilience needs more raw disk; node-based scaling can also cause unnecessary processor and licensing cost when growth is capacity-heavy. Make the comparison with the same net capacity and resilience target.

Is a two-node HCI cluster safe in production?

It is supported with the right witness configuration and is common in small environments such as branches. But while one node is in maintenance, data drops to a single copy; options such as nested resilience in S2D reduce this risk at the cost of lower capacity efficiency.

Can I connect my existing SAN to an HCI cluster?

Many hypervisors can use HCI and external storage at the same time, but solutions such as S2D and Azure Local support external storage only to a limited degree. During the transition, using the SAN in a separate cluster or only for migration is a cleaner approach.

Is a 10 GbE network enough for HCI?

It can be enough for small clusters with a low write load. Because every write is copied to other nodes and resynchronization after a disk failure runs over the network, 25 GbE and, where suitable, RDMA are recommended. Follow the vendor’s minimum network requirement for the product in question.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek System Support

Server installation, configuration, incident response and scheduled maintenance. Active Directory, virtualization and backup are managed with the same discipline.

Knowledge Center

Other articles