# grommunio 2026 Cluster and High Availability: from multi-server lab to tested active/standby homeserver

Source: https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step
Language: en
Published: 2026-09-13
Updated: 2026-11-17

Part 10 shows grommunio HA with DRBD, Pacemaker, Corosync, STONITH, witness, VIP and an active/standby homeserver.

## Quick Overview

Part 10 of the grommunio 2026 series turns the tested active/standby homeserver into a practical step-by-step guide. The goal is one logical homeserver, hs1.cluster.test, that can run on two data nodes and is protected by witness quorum, STONITH, DRBD, /grodata, Pacemaker, Corosync and a service VIP.

The final lab result remains PASS WITH CONDITIONS: the stack was built and tested against planned moves, hard node failures, partitions and witness loss. The limits stay visible as well: production fencing must be real, DRBD is not backup, existing Web sessions were not proven to continue transparently, and load-balancer integration to the hs1 pair remains a known limitation because of separated QEMU L2 networks.

| Area | Result |
| --- | --- |
| Homeserver HA gate | PASS WITH CONDITIONS |
| MariaDB-only HA | FAIL for full Mail/Web/SMTP continuity |
| Web | PASS WITH CONDITIONS: new login after failover proven |
| IMAPS | PASS |
| SMTP | PASS |
| Load balancer to hs1 | KNOWN LIMITATION |
| Backup/restore marker | PASS WITH CONDITIONS |

## Tested Environment

The environment was an isolated QEMU/KVM lab on grommunio 2026.06.1. No production ForgeOne systems were changed. These are lab values, not vendor sizing recommendations.

| Role | Hostname | IP | Purpose |
| --- | --- | --- | --- |
| Data node A | hs1a.cluster.test | 10.210.10.41 | DRBD peer and possible active service node |
| Data node B | hs1b.cluster.test | 10.210.10.42 | DRBD peer and possible active service node |
| Witness | hs1w.cluster.test | 10.210.10.43 | Quorum-only, no mailbox data |
| Logical homeserver | hs1.cluster.test | 10.210.10.40 | Service VIP and client target |

```bash
grommunio: 2026.06.1
Storage: /dev/vdb -> /dev/drbd0 -> /grodata
DRBD resource: hs1-grodata
Logical endpoint: hs1.cluster.test / 10.210.10.40
```

## Target Architecture

The important distinction is physical node versus logical homeserver. hs1a and hs1b only carry the service stack. Clients use hs1.cluster.test with VIP 10.210.10.40.

```bash
quorum only
              hs1w.cluster.test
                10.210.10.43
                       |
hs1a.cluster.test -----+----- hs1b.cluster.test
10.210.10.41                 10.210.10.42
DRBD Secondary/Primary        DRBD Primary/Secondary
                       |
              hs1.cluster.test
                VIP 10.210.10.40
```

Resource order is the core of the guide: quorum and fencing first, then DRBD promotion, then /grodata, bind mounts, VIP and the grommunio service stack.

```bash
STONITH/quorum
  -> DRBD promote
  -> mount /grodata
  -> bind persistent service paths
  -> service VIP
  -> mariadb, redis, php-fpm, gromox, antispam, admin-api, nginx
```

## Prerequisites

Start only with two clean grommunio data nodes on the same software level, a separate witness, static IPs, working DNS or consistent /etc/hosts entries, NTP, root access, a dedicated DRBD data disk and working fencing.

Create a snapshot or backup before changing the system. DRBD mirrors state and mistakes; it is not backup and does not protect against deletion, corruption or incorrect administration.

## Step 1: Check the Baseline

Before installing cluster software, both data nodes must look the same. This check decides whether you should continue at all.

```bash
hostname -f
cat /etc/os-release
rpm -qa | sort | grep -E "grommunio|gromox|mariadb|redis|nginx"
systemctl --failed
ip -br addr
lsblk
chronyc tracking || timedatectl
```

Expected result: both data nodes report grommunio 2026.06.1, no unexplained failed units, the same package baseline and the additional /dev/vdb disk for the DRBD resource.

## Step 2: Prepare the Cluster Network

All cluster names must resolve consistently. The tested hs1 lab uses 10.210.10.41 for hs1a, 10.210.10.42 for hs1b, 10.210.10.43 for hs1w and 10.210.10.40 as service VIP. \`/etc/hosts.d/hs1-cluster.example\` was not evidenced as the tested default resolver path, so it is no longer shown as an executable standard instruction.

Classification: ILLUSTRATIVE. Use DNS or the resolver mechanism that is actually active on your distribution; afterwards \`getent hosts\` and cluster communication must resolve the names consistently.

```bash
getent hosts hs1a.cluster.test hs1b.cluster.test hs1w.cluster.test hs1.cluster.test
ping -c3 hs1b.cluster.test
ping -c3 hs1w.cluster.test
```

Verification: \`20260913-hs1-corosync-pacemaker-baseline.txt\` and the final Corosync configuration with node IPs and node IDs.

## Step 3: Install and Configure Corosync and Pacemaker

In the final lab, hs1w is not qdevice/qnetd. It is a regular third Corosync/Pacemaker voting node. It must not run productive resources; its role is quorum and control-plane participation.

Classification: tested for package installation and service start; tested for the Corosync configuration. The evidence proves the final configuration, not a \`crm cluster init/join\` history, so this article does not claim such a history.

```bash
zypper in corosync pacemaker crmsh resource-agents
install -m 0400 -o root -g root /path/to/lab-authkey /etc/corosync/authkey
install -m 0644 -o root -g root corosync.conf /etc/corosync/corosync.conf
systemctl enable --now corosync pacemaker
crm status
corosync-quorumtool -s
```

```ini
totem {
    version: 2
    cluster_name: grommunio-hs1-ha
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
}

nodelist {
    node {
        ring0_addr: 10.210.10.41
        name: hs1a
        nodeid: 1
    }

    node {
        ring0_addr: 10.210.10.42
        name: hs1b
        nodeid: 2
    }

    node {
        ring0_addr: 10.210.10.43
        name: hs1w
        nodeid: 3
    }
}

quorum {
    provider: corosync_votequorum
    two_node: 0
}

logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
}
```

Verification: \`cluster-control/hs1-corosync.conf\` and \`20260913-hs1-corosync-pacemaker-baseline.txt\` with expected votes 3, total votes 3 and quorum 2.

## Step 4: Classify the Witness Correctly

In this article, Witness describes the role of hs1w in the lab. Technically, hs1w is a normal third Corosync/Pacemaker voting node without DRBD, without \`/grodata\`, without VIP and without the grommunio service group. No qnetd/qdevice design was tested.

Classification: tested. The final CIB explicitly bans DRBD, filesystem, bind mounts, VIP and service group from hs1w.

## Step 5: Configure STONITH

This STONITH setup is lab-only. The tested agent controls QEMU through a host-side controller. In production, replace this with real hardware, hypervisor or cloud fencing.

Classification: tested, but LAB ONLY.

```bash
crm configure <<'EOF'
primitive fence-hs1a stonith:fence_qemu_part10_hs1 \
    params plug=hs1a base_url="http://10.0.2.2:18092" pcmk_host_list=hs1a pcmk_reboot_action=off \
    op monitor interval=60s timeout=20s
primitive fence-hs1b stonith:fence_qemu_part10_hs1 \
    params plug=hs1b base_url="http://10.0.2.2:18092" pcmk_host_list=hs1b pcmk_reboot_action=off \
    op monitor interval=60s timeout=20s
EOF
```

Verification: \`20260913-final-quant-pacemaker-config.txt\`, \`20260913-hs1-stonith-resources.txt\` and \`cluster-control/fence\_qemu\_part10\_hs1\`.

## Step 6: Configure DRBD

DRBD uses \`/dev/vdb\` on hs1a and hs1b and exposes \`/dev/drbd0\` or \`/dev/drbd/by-res/hs1-grodata/0\`. The published secret is intentionally redacted; the real value must never appear in articles, screenshots or the repository.

Classification: tested because the secret is redacted; the remaining values are taken from the tested file.

```bash
zypper in drbd-utils kernel-default-extra drbd-kmp-default
modprobe drbd
install -m 0600 -o root -g root hs1-grodata.res /etc/drbd.d/hs1-grodata.res
drbdadm create-md hs1-grodata
drbdadm up hs1-grodata
drbdadm status hs1-grodata
```

```ini
resource hs1-grodata {
  protocol C;

  net {
    cram-hmac-alg sha256;
    shared-secret "<redacted-lab-secret>";
  }

  disk {
    on-io-error detach;
  }

  on hs1a {
    device /dev/drbd0;
    disk /dev/vdb;
    address 10.210.10.41:7790;
    meta-disk internal;
  }

  on hs1b {
    device /dev/drbd0;
    disk /dev/vdb;
    address 10.210.10.42:7790;
    meta-disk internal;
  }
}
```

Verification: \`cluster-control/hs1-grodata.res\`, \`20260913-hs1-drbd-module-install.txt\`, \`20260913-hs1-drbd-connected.txt\` and \`20260913-hs1-drbd-sync-complete.txt\`.

## Step 7: Build /grodata and Pacemaker Resources

The final chain starts with DRBD as a promotable clone and starts \`/grodata\` only on the promoted data node. \`p-drbd-hs1\` and \`p-fs-grodata\` must be created explicitly before later constraints reference them.

Classification: tested from the final CIB.

```bash
crm configure <<'EOF'
primitive fence-hs1a stonith:fence_qemu_part10_hs1 \
    params plug=hs1a base_url="http://10.0.2.2:18092" pcmk_host_list=hs1a pcmk_reboot_action=off \
    op monitor interval=60s timeout=20s
primitive fence-hs1b stonith:fence_qemu_part10_hs1 \
    params plug=hs1b base_url="http://10.0.2.2:18092" pcmk_host_list=hs1b pcmk_reboot_action=off \
    op monitor interval=60s timeout=20s
primitive p-drbd-hs1 ocf:linbit:drbd \
    params drbd_resource=hs1-grodata \
    op monitor interval=20s role=Unpromoted timeout=20s \
    op monitor interval=10s role=Promoted timeout=20s
clone ms-drbd-hs1 p-drbd-hs1 \
    meta promotable=true promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true interleave=true target-role=Started
primitive p-fs-grodata Filesystem \
    params device="/dev/drbd/by-res/hs1-grodata/0" directory="/grodata" fstype=xfs run_fsck=no \
    op start timeout=60s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
EOF
```

## Step 8: Migrate Data

The migration was verified for seven paths. During copying, uncontrolled writes to the old paths must be stopped. The lab evidence proves the resulting target state; a complete production-safe stop sequence including Postfix was not evidenced as exact shell history.

Classification: tested for the verified target state; the stop sequence is marked RECOMMENDED and does not replace a maintenance plan.

```bash
systemctl stop nginx grommunio-admin-api grommunio-antispam php-fpm 'redis@grommunio' mariadb gromox-http gromox-midb gromox-zcore gromox-event gromox-timer gromox-imap gromox-pop3 gromox-delivery-queue gromox-delivery
systemctl stop postfix
rsync -aHAX /var/lib/mysql/ /grodata/mysql/
rsync -aHAX /var/lib/redis/ /grodata/redis/
rsync -aHAX /var/lib/gromox/ /grodata/gromox/
rsync -aHAX /var/lib/grommunio-web/ /grodata/grommunio-web/
rsync -aHAX /var/lib/grommunio-antispam/ /grodata/grommunio-antispam/
rsync -aHAX /var/lib/grommunio-dav/ /grodata/grommunio-dav/
rsync -aHAX /var/lib/grommunio-admin-api/ /grodata/grommunio-admin-api/
```

| Path | Owner | Mode |
| --- | --- | --- |
| /grodata/mysql | mysql:mysql | 700 |
| /grodata/redis | redis:redis | 750 |
| /grodata/gromox | gromox:gromox | 770 |
| /grodata/grommunio-web | groweb:groweb | 770 |
| /grodata/grommunio-antispam | groas:grommunio | 750 |
| /grodata/grommunio-dav | root:grodav | 770 |
| /grodata/grommunio-admin-api | grommunio:grommunio | 750 |

Verification: \`20260913-hs1-grodata-copy-metrics.txt\`, \`20260913-hs1-local-dir-prep-port-36122.txt\` and \`20260913-hs1-local-dir-prep-port-36222.txt\`. No broad \`chown -R\` correction is retained as article guidance.

## Step 9: Configure Bind Mounts

All seven bind mounts are part of the final resource group. Showing only mysql, redis and gromox would be misleading here.

Classification: tested from the final CIB.

```bash
crm configure <<'EOF'
primitive p-bind-mysql Filesystem \
    params device="/grodata/mysql" directory="/var/lib/mysql" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
primitive p-bind-redis Filesystem \
    params device="/grodata/redis" directory="/var/lib/redis" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
primitive p-bind-gromox Filesystem \
    params device="/grodata/gromox" directory="/var/lib/gromox" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
primitive p-bind-grommunio-web Filesystem \
    params device="/grodata/grommunio-web" directory="/var/lib/grommunio-web" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
primitive p-bind-grommunio-antispam Filesystem \
    params device="/grodata/grommunio-antispam" directory="/var/lib/grommunio-antispam" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
primitive p-bind-grommunio-dav Filesystem \
    params device="/grodata/grommunio-dav" directory="/var/lib/grommunio-dav" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
primitive p-bind-grommunio-admin-api Filesystem \
    params device="/grodata/grommunio-admin-api" directory="/var/lib/grommunio-admin-api" fstype=none options=bind run_fsck=no \
    op start timeout=40s interval=0s \
    op stop timeout=60s interval=0s \
    op monitor interval=20s timeout=40s \
    op_params OCF_CHECK_LEVEL=20
group g-grodata-binds p-bind-mysql p-bind-redis p-bind-gromox p-bind-grommunio-web p-bind-grommunio-antispam p-bind-grommunio-dav p-bind-grommunio-admin-api
EOF
```

## Step 10: Configure the Service VIP and grommunio Service Group

The service group starts on the active data node after \`/grodata\` and the bind mounts. In the tested lab, Postfix is explicitly not part of \`g-grommunio-svc\`; it remains systemd-managed. This also explains the later queue observation: Postfix can accept submission while local delivery to Gromox on \`::1:24\` is not ready yet.

Classification: tested from the final CIB.

```bash
crm configure <<'EOF'
primitive p-vip-hs1 IPaddr2 \
    params ip=10.210.10.40 cidr_netmask=24 nic=enp0s3 \
    op monitor interval=10s timeout=20s
primitive p-mariadb systemd:mariadb op start timeout=120s interval=0s op stop timeout=120s interval=0s op monitor interval=20s timeout=30s
primitive p-redis-grommunio systemd:redis@grommunio op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-php-fpm systemd:php-fpm op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-http systemd:gromox-http op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s meta target-role=Started
primitive p-gromox-midb systemd:gromox-midb op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s meta target-role=Started
primitive p-gromox-zcore systemd:gromox-zcore op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-event systemd:gromox-event op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-timer systemd:gromox-timer op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-imap systemd:gromox-imap op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-pop3 systemd:gromox-pop3 op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-delivery-queue systemd:gromox-delivery-queue op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
primitive p-gromox-delivery systemd:gromox-delivery op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s meta target-role=Started
primitive p-grommunio-antispam systemd:grommunio-antispam op start timeout=90s interval=0s op stop timeout=90s interval=0s op monitor interval=30s timeout=30s
primitive p-grommunio-admin-api systemd:grommunio-admin-api op start timeout=90s interval=0s op stop timeout=90s interval=0s op monitor interval=30s timeout=30s
primitive p-nginx systemd:nginx op start timeout=60s interval=0s op stop timeout=60s interval=0s op monitor interval=20s timeout=30s
group g-grommunio-svc p-mariadb p-redis-grommunio p-php-fpm p-gromox-http p-gromox-midb p-gromox-zcore p-gromox-event p-gromox-timer p-gromox-imap p-gromox-pop3 p-gromox-delivery-queue p-gromox-delivery p-grommunio-antispam p-grommunio-admin-api p-nginx \
    meta target-role=Started
EOF
```

## Step 11: Set Ordering, Colocation and Witness Bans

The final ordering is group-based: DRBD promotion before \`/grodata\`, \`/grodata\` before all seven bind mounts, bind mounts before VIP and VIP before the service group. hs1w is banned from data and service resources.

Classification: tested from the final CIB.

```bash
crm configure <<'EOF'
colocation col-fs-with-drbd inf: p-fs-grodata ms-drbd-hs1:Promoted
colocation col-binds-with-fs inf: g-grodata-binds p-fs-grodata
colocation col-vip-with-fs inf: p-vip-hs1 p-fs-grodata
colocation col-svc-with-binds inf: g-grommunio-svc g-grodata-binds
colocation col-svc-with-vip inf: g-grommunio-svc p-vip-hs1
location loc-fence-hs1a-not-on-hs1a fence-hs1a -inf: hs1a
location loc-fence-hs1b-not-on-hs1b fence-hs1b -inf: hs1b
location loc-no-drbd-witness ms-drbd-hs1 -inf: hs1w
location loc-no-fs-witness p-fs-grodata -inf: hs1w
location loc-no-binds-witness g-grodata-binds -inf: hs1w
location loc-no-vip-witness p-vip-hs1 -inf: hs1w
location loc-no-svc-witness g-grommunio-svc -inf: hs1w
order ord-drbd-before-fs Mandatory: ms-drbd-hs1:promote p-fs-grodata:start
order ord-fs-before-binds Mandatory: p-fs-grodata:start g-grodata-binds:start
order ord-binds-before-vip Mandatory: g-grodata-binds:start p-vip-hs1:start
order ord-vip-before-svc Mandatory: p-vip-hs1:start g-grommunio-svc:start
EOF
```

## Step 12: Configure the Logical Homeserver Identity

The active stack presents the logical homeserver \`hs1.cluster.test\`. Both data nodes use the same \`host\_id\`, while only the Pacemaker-active node runs the services. EXMDB allows the VIP and both physical data-node addresses.

Classification: tested for values and validation, tested for the redacted execution of configuration changes.

```ini
# /etc/gromox/gromox.cfg
host_id=hs1.cluster.test

# /etc/gromox/http.cfg
host_id=hs1.cluster.test

# /etc/gromox/exmdb_provider.cfg
listen_ip=::
exmdb_hosts_allow=::1 127.0.0.1 10.210.10.40 10.210.10.41 10.210.10.42 ::ffff:10.210.10.40 ::ffff:10.210.10.41 ::ffff:10.210.10.42
```

```bash
grommunio-admin server show hs1.cluster.test
grommunio-admin domain show ha1.example.test
grommunio-admin user show ha-user@ha1.example.test
crm status
findmnt /grodata /var/lib/mysql /var/lib/gromox
```

Verification: \`20260913-hs1-gromox-logical-endpoint-config.txt\`, \`20260913-hs1-exmdb-acl-expanded.txt\`, \`20260913-hs1-logical-server-domain-user.txt\` and \`20260913-hs1-full-stack-baseline-active.txt\`.

## Step 13: First Start

After configuration, Pacemaker starts the complete chain. Expected state: STONITH on a suitable peer or hs1w, one promoted DRBD clone, \`/grodata\`, seven bind mounts, VIP and \`g-grommunio-svc\` on the same data node.

Classification: tested for validation commands.

```bash
crm status
crm configure show
findmnt /grodata /var/lib/mysql /var/lib/redis /var/lib/gromox /var/lib/grommunio-web /var/lib/grommunio-antispam /var/lib/grommunio-dav /var/lib/grommunio-admin-api
ip addr show enp0s3 | grep 10.210.10.40
```

## Step 14: Baseline Mailbox Test

The baseline counts only when Web, IMAPS and SMTP work through the service VIP and the queue is empty after delivery. Test credentials and passwords remain internal.

Classification: tested for the tested logic, credentials redacted.

```bash
curl -kI https://10.210.10.40/web/
python3 imaps_probe_redacted.py --host 10.210.10.40 --user ha-user@ha1.example.test
python3 smtp_submission_probe_redacted.py --host 10.210.10.40 --port 587 --user ha-user@ha1.example.test
postqueue -p
```

Verification: \`20260913-hs1-vip-mail-final-baseline-redacted.txt\`, \`20260913-hs1-full-stack-baseline-active.txt\`, \`final-quant-baseline-protocol-probe.txt\` and \`final-quant-baseline-web-results.txt\`.

## Test HA

Every failover test used the same structure: starting state, failure injection, expected behavior, observed behavior, service test, data integrity and result.

### Planned Move

```bash
crm resource move g-grommunio-svc hs1b
crm status
drbdadm status hs1-grodata
findmnt /grodata
ip addr show | grep 10.210.10.40
crm resource clear g-grommunio-svc
```

Result: PASS. The planned move worked. In the quantitative run, five planned moves showed mail-delivery recovery between 6.188 and 7.410 seconds, median 7.076 seconds.

### Hard Node Failure

In the lab, the active node was powered off through QEMU control. In production, the equivalent test is a controlled hypervisor, hardware or cloud fencing test.

```bash
# Lab example, do not copy as a production command:
# power off the active QEMU node hard
crm status
drbdadm status hs1-grodata
findmnt /grodata
ip addr show | grep 10.210.10.40
mailq
```

Result: PASS WITH CONDITIONS. The stack came back on the remaining node. One valid hard-failure service sample was 25.798 seconds, mail delivery was 33.206 seconds. This is one lab sample, not a statistical RTO.

[![grommunio Web is reachable again through the hs1 VIP after a hard failure of the active node.](https://forgeone.solutions/media/hs1-vip-web-after-hard-active-hs1a-failure-retry.png)](https://forgeone.solutions/media/hs1-vip-web-after-hard-active-hs1a-failure-retry.png)

### Network Partition and Split-Brain Protection

The active-partition tests checked whether the cluster prevents dual ownership. The goal is not availability at any price, but no unsafe promotion.

| Test | Expectation | Result |
| --- | --- | --- |
| Active partition | one side takes over in a controlled way, no dual-primary state | PASS |
| Fencing unavailable | cluster blocks safely instead of promoting unsafely | PASS WITH CONDITIONS / SAFE BLOCKED |
| Old primary returns | no automatic overwrite of the active side | PASS |

### Witness Loss plus Active Node Loss

If the witness is missing and the active data node then fails, the remaining node must not take over blindly. In the lab this correctly ended in SAFE\_UNAVAILABLE.

```bash
crm status
corosync-quorumtool -s
# Expected without quorum: no unsafe promotion, manual recovery by runbook
```

## Postfix and Delivery Queue during Hard Failover

The most important finding from the quantitative hard-failure tests was a readiness window: Postfix was not part of the Pacemaker service group and could accept SMTP while gromox-delivery-queue on ::1:24 was not ready yet.

```bash
Postfix submission accepted mail
-> local delivery to ::1:24 was not ready yet
-> message stayed deferred in Postfix
-> message was later delivered after delivery-queue readiness / queue retry
```

This is not observed message loss; it is delayed local delivery. In the verified benchmark data, 0/10 SMTP-accepted messages were lost. This is explicitly not a guaranteed RPO=0 claim.

## Assess Web, IMAP and SMTP

Web was functionally proven as a new login through the hs1 VIP before and after failover. Existing Web sessions were not classified as transparently continued; a historical test showed the login form again after reload.

[![Historical web session test: after reload, the login screen appears again. This does not prove transparent session continuation.](https://forgeone.solutions/media/existing-web-session-after-failover-reload.png)](https://forgeone.solutions/media/existing-web-session-after-failover-reload.png)

| Protocol | Acceptance |
| --- | --- |
| Web new login | PASS WITH CONDITIONS |
| Existing Web session | NOT PROVEN |
| IMAPS | PASS |
| SMTP AUTH/submission | PASS |
| Mail delivery during hard failover | DELAYED possible, no loss observed in verified samples |

## Classify the Load Balancer

The earlier HAProxy/VRRP load-balancer layer was built separately. For the final hs1 pair, end-to-end integration remained a lab limitation because the LB VMs and the hs1 pair were on separate QEMU L2 segments.

```bash
LB lab LAN:      10.210.10.20/21/22 on QEMU socket LAN 230.10.20.26:1210
hs1 HA LAN:      10.210.10.40/41/42/43 on QEMU socket LAN 230.10.20.27:1211

Result: same written IP range, but separate lab L2 segments.
```

## Backup and Restore

DRBD is not backup. The lab checked a narrow restore marker, but not a complete backup strategy. Production needs independent backups, restore tests, retention, offsite copies and documented recovery objectives.

## Why MariaDB-only HA Is Not Enough

The historical MariaDB-only approach remains important, but it belongs in lessons learned, not in the setup path. The lab showed that database failover alone is not enough for end-user continuity of Web, IMAP and SMTP. grommunio needs the consistent homeserver stack with data, services, VIP, quorum and fencing.

```bash
DB moved: partially successful
Admin/Postfix maps: partially usable
Web/IMAP/SMTP continuity: not sufficient

MARIADB_HA_GATE = FAIL
```

## Measured Failover Times

| Scenario | Measurement | Classification |
| --- | --- | --- |
| Planned homeserver move | 5 samples: min 6.188 s, median 7.076 s, mean 6.929 s, max 7.410 s | publishable lab observation for mail-delivery recovery |
| Hard-failure service RTO | 1 valid sample: 25.798 s | single sample only, not a statistic |
| Hard-failure mail delivery | 1 valid sample: 33.206 s | delayed queue delivery, no SLA statement |
| SMTP-accepted messages | 0/10 verified accepted messages lost | observation, no guaranteed RPO=0 claim |
| Web RTO | not measured reliably | do not publish a number |

## Technical Acceptance

| Area | Check | Result |
| --- | --- | --- |
| Cluster bootstrap | Manual Corosync configuration evidenced; init/join history not evidenced | PASS WITH CONDITIONS |
| Witness | hs1w is a regular third voting node, not qdevice/qnetd | PASS |
| STONITH | Lab fencing fully documented, production fencing required | PASS\_WITH\_CONDITIONS |
| DRBD | Resource, clone and sync evidenced; secret redacted | PASS |
| /grodata | Filesystem resource and mounts present | PASS |
| Bind mounts | all seven bind mounts in group, ordering and colocation | PASS |
| Service group | final units and order taken from CIB | PASS |
| Postfix | systemd-managed outside Pacemaker documented; port 24 explained | PASS |
| Logical homeserver | \`hs1.cluster.test\`, \`host\_id\` and EXMDB evidenced | PASS |
| Data migration | target owners and modes verified; no broad chown commands | PASS |
| Service stop sequence | production stop order is recommendation, not lab history | FAIL |
| Clean-room reproduction | no fresh ha2a/ha2b/ha2w built solely from the article | FAIL |

Reproducibility audit: NOT TESTED = 0, TECHNICAL\_EXPANSION\_REQUIRED = 2, ILLUSTRATIVE\_COMMAND\_BLOCKS = 1. Every executable snippet without complete evidence was removed, marked as redacted/normalized, or visibly classified as illustrative.

## Troubleshooting and Lessons Learned

DRBD module missing: drbd-utils alone was not enough in the lab. kernel-default-extra and drbd-kmp-default were required.

Wrong homeserver identity: if host\_id and the logical homeserver are inconsistent, EXMDB or access errors can appear.

Fencing unavailable: the desired behavior is blocking, not risky continuation. Recovery then needs manual cleanup by runbook.

Postfix reachable too early: SMTP reachability is not automatically mail-delivery readiness. Monitoring should treat queue state, port 24 and gromox-delivery-queue separately.

## Production Notes

The lab deliberately keeps critical warnings visible: real fencing, independent backups, monitoring, documented recovery procedures, repeated hard-failure measurements, load-balancer end-to-end tests and a session/reconnect concept belong on the pre-go-live checklist.

## Conclusion

The tested path is not MariaDB-only and not simply multi-server. The lab became viable as a complete logical homeserver: DRBD, /grodata, bind mounts, VIP, service stack, quorum and STONITH in a controlled order. That order is what turns the lab report into a reproducible HA guide.

## Sources and Further Documentation

Primary sources and vendor documentation used to classify this article technically.

[**grommunio High Availability**](https://docs.grommunio.com/admin/high-availability/)

[**grommunio Architecture**](https://docs.grommunio.com/admin/architecture/)

[**grommunio-admin server**](https://docs.grommunio.com/cli/grommunio-admin/server/)

[**grommunio Administration**](https://docs.grommunio.com/admin/administration/)

[**Pacemaker explained**](https://clusterlabs.org/projects/pacemaker/doc/)

[**DRBD User Guide**](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/)

## Review the cluster design?

ForgeOne supports grommunio architecture, HA labs, migration planning, operations, monitoring and documented failover runbooks. [**Discuss a grommunio project**](https://forgeone.solutions/en/contact)

## Hero image

[![grommunio Web is reachable through the hs1 VIP before failover.](https://forgeone.solutions/media/hs1-vip-web-baseline-after-safety-gates-2.png)](https://forgeone.solutions/media/hs1-vip-web-baseline-after-safety-gates-2.png)

## Author

Mathias Rumbold

## Category

- [grommunio & Groupware](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?category=grommunio-groupware)

## Tags

- [grommunio](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?tags=grommunio)
- [Groupware](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?tags=groupware)
- [Mobile Device Management](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?tags=mobile-device-management)
- [Exchange ActiveSync](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?tags=exchange-activesync)
- [Digital Sovereignty](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?tags=digitale-souveraenitaet)
- [Native Exchange Replacement](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step?tags=native-exchange-replacement)

## Plan grommunio HA reliably

We support cluster design, high availability, failover tests and operational validation.

[Discuss HA concept](https://forgeone.solutions/en/products/grommunio)

## grommunio 2026 Schritt für Schritt

Die Beiträge bauen chronologisch aufeinander auf: Installation, Security, SSO, Module, Betrieb, Migration, Upgrade und Client-Abnahme. Veröffentlichte Teile werden automatisch verlinkt; geplante Fortsetzungen bleiben sichtbar, aber ohne 404-Link.

1. [Install grommunio 2026.06.1: From ISO to Working Groupware](https://forgeone.solutions/en/tech/install-grommunio-2026)
2. [grommunio-antispam with Rspamd: Spam Filtering, Learning and Operations Step by Step](https://forgeone.solutions/en/tech/grommunio-antispam-rspamd-2026-step-by-step)
3. [grommunio-auth with Keycloak: SSO and central login step by step](https://forgeone.solutions/en/tech/grommunio-auth-keycloak-2026-step-by-step)
4. [Set up grommunio Meet 2026 without Keycloak and optionally add SSO](https://forgeone.solutions/en/tech/grommunio-meet-2026-step-by-step)
5. [Set Up grommunio Chat 2026: Teams, Channels, Users and Multi-Tenancy Testing](https://forgeone.solutions/en/tech/grommunio-chat-2026-step-by-step)
6. [Set Up grommunio Files and Office: Validate Files, Web Entry and Online Office](https://forgeone.solutions/en/tech/grommunio-files-office-2026-step-by-step)
7. [Set Up grommunio Archive 2026: Archive, Search and Restore Email](https://forgeone.solutions/en/tech/grommunio-archive-2026-step-by-step)
8. [grommunio MDM 2026 step by step: manage ActiveSync devices safely](https://forgeone.solutions/en/tech/grommunio-mdm-2026-step-by-step)
9. [grommunio Multitenancy 2026: organizations, domains and multiple LDAP sources](https://forgeone.solutions/en/tech/grommunio-multitenancy-2026-step-by-step)
10. [grommunio 2026 Cluster and High Availability: from multi-server lab to tested active/standby homeserver (current part)](https://forgeone.solutions/en/tech/grommunio-cluster-ha-2026-step-by-step)
11. [grommunio 2026 disaster recovery and single mailbox restore: backup, full DR and targeted mailbox recovery in the lab](https://forgeone.solutions/en/tech/grommunio-disaster-recovery-single-mailbox-restore-2026-step-by-step)
12. [Migrate IMAP mailboxes to grommunio with imapsync](https://forgeone.solutions/en/tech/grommunio-imap-migration-imapsync-2026-step-by-step)
13. [grommunio: Replace Public Folders with a functional mailbox - step by step](https://forgeone.solutions/en/tech/grommunio-public-folder-functional-mailbox-multiple-domains)
14. [grommunio Web 5.0 in the lab: settings namespace, shared mailboxes and admin impact](https://forgeone.solutions/en/tech/grommunio-web-5-0-in-the-lab)
15. [grommunio 2026.06.2 Upgrade and Technical Acceptance: Keycloak SSO, MFA, AutoDiscover and Mail Flow](https://forgeone.solutions/en/tech/grommunio-2026-06-2-upgrade-keycloak-sso-practice)
16. [grommunio native clients after Keycloak SSO: Thunderbird, Outlook, EWS, EAS and MFA explained](https://forgeone.solutions/en/tech/grommunio-native-clients-after-keycloak-sso)
17. grommunio Monitoring und Alerting im Betrieb
18. grommunio Security Hardening in der Praxis
19. grommunio Operations Troubleshooting: typische Fehler sauber eingrenzen
