Implementing HA Storage on Dedicated Servers for Controlled Failover

Downtime can interrupt revenue, data access, and recovery operations, but its impact varies by workload and business. High-availability storage should therefore be designed around explicit recovery point objectives (RPOs), recovery time objectives (RTOs), and tested failure scenarios—not a single availability percentage or a universal cost estimate.

This article outlines a blueprint for HA dedicated storage on dedicated servers: RAID 10 for local disk resilience, synchronous replication across nodes, quorum and fencing to prevent split-brain behavior, and automated failover to reduce recovery time. It also compares DRBD, GlusterFS, and Ceph for different access models and rolling maintenance.

Choose Melbicom — 1,100+ ready-to-go servers — 20 Tier III & Tier IV data centers — 39 CDN PoPs worldwide Order a server

HA Dedicated Storage Blueprint

• Local protection on each server (RAID 10).
• Synchronous replication across peers to protect acknowledged writes from a single-node loss.
• Quorum plus a witness or qdevice to preserve a single authoritative partition.
• Automated failover after quorum and fencing checks.

This “defense-in-depth” design helps prevent a disk, server, or network failure from becoming a prolonged outage and keeps recovery measurable.

RAID 10: Local Redundancy and Faster Rebuilds

RAID 10 (striped mirror pairs) provides low write overhead and typically faster rebuilds than parity RAID because recovery copies data from the surviving mirror instead of recalculating parity. This shortens—but does not eliminate—the degraded period after a disk failure. RAID 10 can also preserve more predictable write behavior under load. On modern dedicated servers, enterprise SSDs or NVMe drives should be sized for application I/O, replication, rebuild activity, and endurance rather than assumed to provide sufficient headroom automatically.

Synchronous Replication for RPO Zero

Synchronous replication can provide an RPO of zero for writes acknowledged by both replicas, but only within the failure modes the design covers. The surviving node must have current data, the cluster must preserve a single writer through quorum and fencing, and application or filesystem state must be recovered consistently before service resumes.

Size the replication path from measured peak write throughput, resynchronization demand, latency, and safety headroom; there is no universal 10 GbE minimum. A dedicated or VLAN-segregated path can isolate replication and cluster traffic from client traffic. Melbicom dedicated servers can support private networking and VLANs, and global configurations can provide up to 200 Gbps per server. Keep NICs, cabling, and switch paths symmetric, and test latency under the load and failure conditions expected in production.

Two established approaches:

• Block-level replication (DRBD): DRBD replicates block-device writes between nodes. With synchronous Protocol C, a write completes only after local and remote disk writes are confirmed. It is commonly used in active-passive designs in which one node owns and mounts the filesystem while the peer tracks writes and can be promoted after cluster safety checks.
• Replicated file storage (GlusterFS): GlusterFS provides a shared file namespace with replicated volumes and active-active client access. Availability and consistency behavior depend on volume type, quorum, client protocol, and heal state. Before a new deployment, verify current upstream maintenance, distribution support, and the documented upgrade path.

Two-node cluster with synchronous replication and a quorum device

Quorum and Split-Brain Prevention

Split brain occurs when two partitions accept writes independently. Quorum identifies which partition has a majority and may continue under the configured policy, but quorum alone does not fence hardware or guarantee storage integrity. In a two-node Pacemaker/Corosync cluster, add a qdevice or another third vote where supported, and define fencing so only one side can promote the storage resource.

Place the quorum device in an independent failure domain with reliable connectivity to both nodes. It should not store application data, but it remains part of the control plane and must be monitored and secured. For three or more data nodes, design vote distribution and failure behavior from the cluster software’s documented quorum model rather than relying only on an odd-node rule.

Fencing (STONITH) is the required safety mechanism that prevents an untrusted node from continuing to serve or write before a peer is promoted. Use IPMI or another out-of-band method independent of the data network, and test fencing under power, management-plane, and network faults.

Safe Automated Failover and Recovery

Automated failover should execute a defined sequence only after failure detection, quorum, and fencing checks. Pacemaker with Corosync is a common Linux stack: Corosync provides membership and messaging, while Pacemaker manages resources, ordering, constraints, and recovery actions. Actual RTO depends on detection timeouts, fencing, promotion, filesystem recovery, service startup, endpoint movement, and client retries.

A common DRBD + filesystem + export failover workflow:

• Confirm quorum and fence or isolate the failed primary.
• Promote DRBD on the secondary only when its data state is current.
• Mount the filesystem and start the export service in the defined order.
• Move the service IP or update the load-balancer target; use DNS only when TTL and cache behavior fit the RTO.

Returning nodes must not serve stale data. Use storage-layer resynchronization or healing together with cluster health gates:

• DRBD performs an incremental resynchronization of changed blocks.
• GlusterFS heals file replicas under its configured volume and quorum rules.

Keep orchestration conservative: re-admit a node only after replication, filesystem, service, and cluster checks pass. Codify resource ordering and timeouts to prevent timing races. Measure RTO and RPO during power-loss, NIC, network-partition, and process-failure tests.

DRBD vs. Distributed Storage Maintenance

Both designs can support rolling maintenance, but neither guarantees that every client sees uninterrupted I/O. Choose based on access model, consistency needs, failure domains, client retry behavior, and operational expertise.

Rolling maintenance considerations

• With DRBD, perform a controlled failover to the secondary, patch and reboot the former primary, resynchronize it, and optionally fail back. Client I/O can pause during the switch, so application retries must be tested.
• With GlusterFS, follow the documented upgrade order and update one node at a time only when replica and quorum state permit it. Verify heal completion and volume health before proceeding.
• With Ceph, use the supported rolling-upgrade path and allow cluster health checks to gate daemon restarts.

Single-controller SANs concentrate maintenance and failure risk in one appliance, but distributed storage is not a universal winner; compare failure domains, upgrade behavior, operational complexity, and application requirements.

Operational Checklist

• Validated builds: use identical or compatibility-tested drives, NICs, controllers, firmware, and settings across nodes.
• Replication fabric: size for peak writes, resynchronization, and headroom; isolate where practical; use symmetric paths and redundant links when the failure model requires them.
• Quorum discipline: document vote behavior; place the qdevice or witness in an independent failure domain; test STONITH fencing.
• Health gates: verify replication, filesystem, service, and cluster health.
• Failover drills: test power, NIC, process, and partition faults on a defined schedule; record measured RTO and RPO.
• Security and compliance: isolate cluster traffic; encrypt where required; maintain separate, tested backups, with immutability where the chosen target supports it; consider S3-compatible offsite copies and SFTP backup for suitable workflows.
• Geographic strategy: keep synchronous replicas within a tested latency budget; use asynchronous replication for regional disaster recovery; place the quorum device in an independent failure domain.

Conclusion: Bringing the Parts Together

HA storage is not a single technology or a zero-downtime guarantee. A practical design combines local disk resilience, replication selected for the required RPO, quorum and fencing to prevent divergent writers, and orchestrated recovery tested against an explicit RTO. DRBD, GlusterFS, and Ceph address different access and scale models; each requires supported versions, documented failure behavior, and capacity for normal I/O plus recovery work.

If you are planning a refresh, begin with two compatibility-tested servers, a replication path sized from measured writes and resynchronization demand, an independent quorum device where the cluster design requires one, and repeatable failover tests. Expand only after the failure domains, client behavior, and operational load are understood; adding nodes without revisiting quorum and placement rules does not automatically improve resilience.