# Proxmox VE: Migrating CephX from AES to AES256K with Rook/OKD

Source: https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration
Language: en
Published: 2026-09-27
Updated: 2026-11-19

CephX key migration from AES to AES256K in Proxmox VE with external Rook/OKD, Ceph-CSI secrets and final cipher lockdown.

Ceph Squid and current Proxmox VE releases make it visible when CephX keys still use the legacy aes key type or when old ciphers are still allowed. That does not automatically mean a storage outage. A cluster can replicate data cleanly, have all OSDs up/in, and still report CephX warnings.

The migration becomes critical in two areas: running consumers and external clients. A pure Proxmox/Ceph cluster can be migrated in a comparatively controlled way with the Proxmox migration helper. If a Kubernetes or OKD cluster is attached through Rook and Ceph-CSI, however, its CephX identities and Kubernetes secrets must be handled separately.

This article documents a real migration on a Proxmox VE/Ceph cluster with Ceph Squid 19.2.6 and an externally attached OKD/Rook cluster. It is not a universal copy-and-paste guide. The output of the Proxmox migration helper and the actual consumer inventory of your own cluster remain the authoritative basis for decisions.

Hinweis: Do not copy this blindly into production. Before every step, verify Ceph health, backups, external consumers, Kubernetes secrets, version compatibility, and a recovery path.

## Example Setup

The environment in the documented project consisted of:

- Proxmox VE cluster with 4 nodes
- Ceph Squid 19.2.6
- 4 MONs
- 4 MGRs
- 4 MDS daemons
- 16 OSDs
- CephFS and RBD
- external OKD/Kubernetes cluster
- Rook CephCluster in external mode
- active Ceph-CSI RBD provisioning
- existing CephFS credentials in the Rook external setup

The external CephX identities included:

```text
client.csi-rbd-node
client.csi-rbd-provisioner
client.csi-cephfs-node
client.csi-cephfs-provisioner
client.healthchecker
```

The target state was a fully migrated setup:

```text
auth_service_cipher: aes256k
auth_allowed_ciphers: aes256k
auth_preferred_cipher: aes256k
```

At the end, Ceph reported:

```text
HEALTH_OK
16/16 OSD up/in
209 PGs active+clean
```

## 1. Check the Initial State

Before rotating anything, start with a sober inventory:

```bash
ceph -s
ceph versions
pveceph auth status
```

The important signals are not only CephX warnings, but the overall state:

- MON quorum available?
- OSDs up and in?
- PGs active+clean?
- MDS and CephFS healthy?
- Which CephX warnings are active?
- Which clients are still visible with the old key type or old sessions?

At the beginning, warnings like these may appear:

```text
AUTH_INSECURE_CLIENT_KEY_TYPE
AUTH_INSECURE_KEYS_ALLOWED
AUTH_INSECURE_KEYS_CREATABLE
AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE
AUTH_INSECURE_SERVICE_KEY_TYPE
AUTH_INSECURE_SERVICE_TICKETS
```

These messages do not automatically mean data is degraded. They first indicate that CephX keys, service tickets, or allowed ciphers are not yet in the desired state. Still, they must be taken seriously because disabling aes too early can lock out old consumers.

## 2. Upgrade All Ceph Daemons First

The migration should only start once MON, MGR, MDS, and OSD daemons run a version that supports aes256k. In the project, Ceph was upgraded from 19.2.3 to 19.2.6.

After upgrades, service restarts, or rolling reboots, check again:

```bash
ceph versions
pveceph auth status
```

The important part was:

```text
Monitor quorum
aes256k capable: yes
```

During rolling reboots, only one Proxmox node was rebooted at a time. After each node, the Ceph state was checked again. Only once quorum, OSDs, and PGs were clean again did the next node follow.

## 3. Use the Proxmox Migration Helper

The central entry point is the Proxmox helper:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --verbose
```

Without --apply, the helper performs a dry run. That is intentional. The output shows what Proxmox would change, which steps are blocked, and which consumers still need attention.

In the concrete run, the helper planned among other things:

- rotation of MGR keys
- rotation of MDS keys
- rotation of OSD keys
- migration of service tickets
- temporary noout
- monitor elections

A particularly important warning concerns OSD lockbox keys:

```text
Never rotate a 'client.osd-lockbox' key by hand
```

That warning is serious. OSD lockbox keys are not only stored in the Ceph auth database, but also tied to local metadata. Rotating them manually with ceph auth risks inconsistencies. The helper keeps the auth database and local metadata aligned.

## 4. Migrate Service Keys

If the dry run reports no blockers:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply
```

Then verify:

```bash
ceph -s
pveceph auth status
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
```

A desired intermediate result is:

```text
PASS: Every service key uses 'aes256k',
and so do the service tickets.
```

Only then does it make sense to look at cluster keys, bootstrap keys, and clients.

## 5. Cluster Keys, Bootstrap Keys, and OSD Lockbox Keys

In the real run, the helper provided the next step for cluster-internal keys:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply --rotate-cluster-keys
```

This covers more than simple client keys. Relevant identities include:

- bootstrap keys
- client.crash
- OSD lockbox keys
- other cluster-managed identities

OSD lockbox keys are especially critical. The helper updates not only CephX entries but also the related local metadata. Therefore, “just rotate all keys” is not a safe description of this step.

## 6. client.admin Is Special

client.admin deserves separate attention. In the project, the dry run showed:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --rotate-admin-key --verbose
```

more than 200 running client.admin sessions.

That does not mean that more than 200 VMs have to be running. QEMU/RBD clients can open multiple Ceph connections, and a running VM keeps existing Ceph sessions alive. The number of sessions is therefore not a one-to-one VM count.

The rotation is staged:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply --rotate-admin-key
```

The old key remains temporarily valid while the new key is distributed. Proxmox keyring copies are updated. Existing sessions can continue to live; new sessions must already be established with the new key. Only once all consumers have really been refreshed may the old key be removed finally.

## 7. CephFS Mounts Can Block the client.admin Refresh

In the project, the CephFS mount at:

```text
/mnt/pve/cephfs
```

was busy. Diagnosis:

```bash
fuser -vm /mnt/pve/cephfs
lsof +D /mnt/pve/cephfs
```

The cause was running VMs with ISO images inserted directly from CephFS:

```text
/mnt/pve/cephfs/template/iso/...
```

The fix was not to hard-stop VMs. Instead, the ISO images were removed from the virtual CD-ROM drives. Example:

```bash
qm set <VMID> --ide2 none,media=cdrom
```

The concrete slot must be checked first. Not every VM uses ide2. The key point is that no running VM should still keep an ISO open directly from the CephFS mount.

Then check again:

```bash
lsof +D /mnt/pve/cephfs
```

Once the mount is no longer busy, run the helper again.

## 8. Refresh Running QEMU/RBD Sessions

After the CephFS refresh, almost 200 old client.admin sessions remained in the example. These were refreshed through live migration of the VMs:

```bash
qm migrate <VMID> <TARGET_NODE> --online
```

A live migration is sufficient in this context because a new QEMU process is created on the target node and its Ceph connections are established with the new key.

For larger environments, a one-time script is useful that:

- records running VMs
- migrates each VM exactly once
- stores successful VMIDs in a state file
- logs errors separately
- selects target nodes based on current load or VM count
- checks the CephX state after each migration

A simple generic sketch:

```bash
#!/usr/bin/env bash
set -euo pipefail

STATE=/root/cephx-refreshed-vms.txt
touch "$STATE"
chmod 600 "$STATE"

for vmid in $(qm list | awk 'NR>1 && $3 == "running" {print $1}'); do
  if grep -qx "$vmid" "$STATE"; then
    continue
  fi

  current_node="$(hostname)"
  target_node="<choose-target-node>"

  echo "Migrating VM ${vmid} from ${current_node} to ${target_node}"
  qm migrate "$vmid" "$target_node" --online
  echo "$vmid" >> "$STATE"

  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
done
```

This is intentionally only a skeleton. In production, target node selection, error handling, and maintenance logic must fit the environment.

## 9. Confirm client.admin Finally

Only when the helper reports:

```text
Ready for confirmation: client.admin
```

run:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply --confirm-all-clients-refreshed
```

Before doing this, also account for external copies of client.admin. These can be manual keyring copies, old automation, backup jobs, or other integrations.

## 10. External Rook/OKD Consumers

The key difference to a pure Proxmox cluster is that an external Rook cluster uses its own CephX identities. In the project, these were:

```text
client.csi-rbd-node
client.csi-rbd-provisioner
client.csi-cephfs-node
client.csi-cephfs-provisioner
client.healthchecker
```

The Proxmox helper deliberately does not change these keys automatically. The helper output effectively said:

```text
Left to whoever manages the client that reads them
```

That is correct. Proxmox does not know the secrets stored in Kubernetes. A CephX rotation on the Ceph side is only complete once the corresponding Kubernetes secrets in the consumer cluster are synchronized as well.

## 11. Identify the Rook External Cluster

In the OKD/Kubernetes cluster:

```bash
oc -n rook-ceph get cephcluster
oc get storageclass
oc -n rook-ceph get secrets
```

In the project:

```text
CephCluster EXTERNAL=true
StorageClass rook-ceph-block
Provisioner rook-ceph.rbd.csi.ceph.com
```

The RBD secrets were:

```text
rook-csi-rbd-node
rook-csi-rbd-provisioner
```

with:

```text
userID
userKey
```

The important part is alignment: the CephX user on the Ceph side and userKey in the Kubernetes secret must match. Do not copy keys into tickets, chat, Git, or logs.

Before changing Kubernetes secrets, check whether the secret is managed by GitOps or a controller:

```bash
oc -n rook-ceph get secret <SECRET> -o json | jq '.metadata | {ownerReferences, annotations, labels}'
```

ownerReferences, ArgoCD tracking labels, and the annotation kubectl.kubernetes.io/last-applied-configuration are especially important. If that annotation contains old secret values, it can later be accidentally reapplied or become visible in backups or exports. In our run, those metadata fields were checked before changes, and stale last-applied annotations were removed deliberately.

## 12. Special Case: ceph-csi Version and AES256K

A real blocker appeared directly with an external RBD key. After rotating one RBD key, provisioning initially failed:

```text
connecting failed: rados: ret=-22, Invalid argument
```

The error could be narrowed down to AES256K compatibility of the CSI stack in use. After updating to a compatible Rook/ceph-csi state, provisioning worked again. The verified state in the OKD cluster was:

```text
Rook Operator: v1.20.7
Ceph-CSI: v3.17.1
```

Matching CSI sidecars were also used, including:

```text
csi-provisioner v5.2.0
csi-attacher v4.8.1
csi-resizer v1.13.2
csi-snapshotter v8.2.1
csi-node-driver-registrar v2.13.0
```

Before rotating external CSI keys, verify that the deployed Rook/ceph-csi version supports the new key type. Otherwise the key may be rotated correctly while the Kubernetes consumer is locked out.

## 13. Rotate the RBD Provisioner Key

On the Ceph side, back up first and then rotate:

```bash
umask 077

ceph auth get client.csi-rbd-provisioner \
  > /root/client.csi-rbd-provisioner.before-aes256k.keyring

ceph auth rotate --key-type=aes256k client.csi-rbd-provisioner \
  > /root/client.csi-rbd-provisioner.aes256k.keyring

chmod 600 /root/client.csi-rbd-provisioner*.keyring
```

The new key is then safely written to the Kubernetes secret:

```text
rook-csi-rbd-provisioner
```

Only update the secret value, not the userID. After ceph auth rotate, the old key is no longer the current key; therefore the secret update must follow immediately and in a controlled way before new provisioner connections are tested.

Important: after a rotation, do not copy back from an old local file, an old chat snippet, or a previous secret. The currently valid key is retrieved directly from Ceph:

```bash
ceph auth get-key client.csi-rbd-provisioner
```

If a key is accidentally rotated twice, the first new key is already obsolete again. In that case, synchronize exactly the currently valid key, not the key from the first rotation attempt.

After the update, the RBD provisioner was refreshed and a new PVC was provisioned. Only once a new PVC becomes Bound has the provisioner path been tested in practice.

## 14. Rotate the RBD Node Key

Analogously:

```bash
ceph auth rotate --key-type=aes256k client.csi-rbd-node
```

Kubernetes secret:

```text
rook-csi-rbd-node
```

Then test the node plugin path in practice:

- new PVC
- pod mounts PVC
- write a file
- read the file
- delete the pod
- recreate the pod
- read the file again

In the project, this end-to-end test succeeded. That matters: provisioning alone does not prove that node staging, mounting, and pod access work.

## 15. Check Active Sessions

The working path for session inspection was:

```bash
ceph tell mon.<MON-NAME> sessions --format json-pretty
```

Do not use:

```bash
ceph sessions
```

With multiple MONs, ceph tell mon.\* sessions can produce multiple JSON outputs. That is not necessarily friendly for direct jq processing. It is better to check per MON or split the output intentionally.

In the project, active RBD sessions could be observed with:

```text
auth_key_type: aes256k
```

Missing active sessions are not proof that an identity is no longer needed. Provisioner accounts may only appear for new PVCs, CephFS credentials may be prepared even when no CephFS PVCs currently exist, and healthchecker access depends on Rook operator behavior.

## 16. CephFS CSI Keys

These accounts were also rotated:

```text
client.csi-cephfs-provisioner
client.csi-cephfs-node
```

The corresponding secrets:

```text
rook-csi-cephfs-provisioner
rook-csi-cephfs-node
```

were updated.

At that point, the concrete cluster had:

- no CephFS StorageClass
- no CephFS PVCs
- no running CephFS CSI pods

Nevertheless, the credentials were synchronized because they were part of the external Rook setup. “No active session right now” does not automatically mean “credential is no longer needed.” Deleting it would be a separate architecture decision, not a side step of a key migration.

## 17. client.healthchecker

The final external key was:

```text
client.healthchecker
```

It was stored in the secret:

```text
rook-ceph-mon
```

with the fields:

```text
ceph-username
ceph-secret
```

The decoded username was:

```text
client.healthchecker
```

Rook uses this user for external cluster health and monitor access. The operator logs confirmed this:

```text
will use "client.healthchecker" to check health and monitor status
```

After rotation:

```bash
ceph auth rotate --key-type=aes256k client.healthchecker
```

only ceph-secret in the secret rook-ceph-mon was updated.

Immediately afterwards, the Rook CephCluster briefly reported:

```text
HEALTH_ERR
RADOS permission denied
```

The running operator still had the old healthchecker access active. A targeted operator refresh was then required:

```bash
oc -n rook-ceph rollout restart deployment/rook-ceph-operator
```

After the restart, rook-ceph-external was back to:

```text
Synced / Healthy
```

and the CephCluster warnings were reduced to the global cipher settings.

Before the patch, it was verified that the secret would not be reset automatically through OwnerReferences or ArgoCD. If GitOps manages this secret, the new value must be updated in the authoritative source, otherwise the cluster can later lose the correct key again.

## 18. Reconcile Externally Rotated Keys with the Proxmox Helper

Because the five external keys were rotated outside the Proxmox script, the helper reported that individual keys had been changed outside the script.

For each external consumer, a first measurement was triggered:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --confirm-clients-refreshed <CLIENT> --apply
```

On the first run, the expected message was effectively:

```text
not accepting ... yet
complete measurement is now recorded
```

This first FAIL is not automatically a cluster error in this context. The helper does not yet have a complete measurement for that external consumer and records it first. Only after all relevant external consumers have been measured can the dry run show that confirmation is possible.

If the helper reports full inventory unavailable, its statement is also limited: then the helper cannot safely evaluate the complete consumer inventory. In that case, the final lockdown must not be derived from partial information alone.

After the first measurement for all five clients, the dry run reported:

```text
Ready for confirmation:
client.csi-cephfs-node,
client.csi-cephfs-provisioner,
client.csi-rbd-node,
client.csi-rbd-provisioner,
client.healthchecker
```

Only then was the helper ready to confirm the consumers as refreshed.

## 19. Disable AES Finally

Hinweis: Important: only run the final cipher lockdown once the Proxmox helper suggests it after complete measurement and all external consumers demonstrably work with the new keys.

Only now did the helper provide the final command:

```bash
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply \
  --confirm-all-clients-refreshed \
  --restrict-ciphers
```

From this point on, old aes keys are actually rejected. Anyone who missed an external consumer can lock it out of Ceph with this step.

The expected completion:

```text
PASS: only the 'aes256k' cipher is allowed for authentication now
PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.
```

## 20. Final Status

Final verification:

```bash
pveceph auth status
ceph -s
ceph health detail
```

Final state:

```text
Cephx health checks
none active

auth_service_cipher: aes256k
auth_allowed_ciphers: aes256k
auth_preferred_cipher: aes256k
```

and:

```text
HEALTH_OK
```

In the example setup:

```text
mon: 4 daemons in quorum
osd: 16 up / 16 in
mds: healthy
209 PGs active+clean
```

## 21. Do Not Delete the Recovery Journal Too Early

The helper works with a journal:

```text
/etc/pve/priv/cephx-key-migration.json
```

This file can contain recovery information and previous keys. Therefore:

- protect file permissions
- do not publish it
- do not copy it into repositories
- do not place it uncontrolled into less protected backups
- clean it up only after completed access verification according to the Proxmox documentation

## 22. Lessons Learned

The most important points from the run:

- First bring all Ceph daemons to a compatible version.
- Take helper dry runs seriously.
- Never rotate OSD lockbox keys manually.
- client.admin requires a consumer refresh.
- Live migration works well to refresh QEMU/RBD sessions.
- CephFS can remain busy because of inserted ISO images.
- External Rook/CSI keys are not maintained automatically by Proxmox.
- Kubernetes secrets must be synchronized immediately with the CephX keys.
- Check ceph-csi compatibility before rotating external CSI keys to AES256K.
- Run practical provisioning and mount tests.
- Disable aes only at the very end.
- Run the final --restrict-ciphers step only when the helper suggests it after complete measurement.

## 23. Security Notes

Hinweis: Do not copy these commands blindly into production.

Before every step, check:

- Ceph health
- backups
- consumer list
- external clusters
- Kubernetes secrets
- version compatibility
- maintenance window
- recoverability
- monitoring and alerting

The most important failure class is not the rotation itself, but a missed consumer. If old external consumers are overlooked and aes is disabled too early, those consumers can be locked out of Ceph.

## Conclusion

The actual CephX migration inside the Proxmox cluster is only part of the work. External consumers such as Rook and Ceph-CSI are the decisive part. Only once their credentials have been updated, consumers refreshed, and storage functions practically tested may the old cipher be disabled.

In this project, the final state was clean: only aes256k allowed, Ceph HEALTH\_OK, all OSDs up/in, all PGs active+clean, and the external Rook/OKD consumer back to Synced / Healthy.

## Hero image

[![Anonymized Proxmox VE cluster overview with multiple nodes and healthy Ceph status](https://forgeone.solutions/media/proxmox-cephx-aes256k-rook-okd-hero-2.png)](https://forgeone.solutions/media/proxmox-cephx-aes256k-rook-okd-hero-2.png)

## Author

Mathias Rumbold

## Category

- [Proxmox & Virtualization](https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration?category=proxmox-virtualisierung)

## Tags

- [proxmox](https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration?tags=proxmox)
- [okd](https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration?tags=okd)
- [kubernetes](https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration?tags=kubernetes)
- [Open Source](https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration?tags=open-source)

## Need a clean Ceph, Proxmox or Kubernetes storage migration?

ForgeOne supports Proxmox VE, Ceph, Rook, OKD, Kubernetes storage, operational documentation and controlled migration workflows.

[Discuss a storage project](https://forgeone.solutions/en/contact)

## Proxmox / Ceph Schritt für Schritt

Geplante Betriebsreihe für Proxmox VE, Cluster, Ceph, Backup, HA, Monitoring, Upgrades, Hardening und Recovery. Noch nicht vorhandene Teile bleiben geplant und unverlinkt.

1. [How to Set Up Proxmox VE](https://forgeone.solutions/en/tech/how-to-set-up-proxmox-ve)
2. Proxmox Networking sauber aufbauen
3. Proxmox Cluster und Quorum
4. Ceph Design für Proxmox
5. CephFS und RBD im Proxmox-Betrieb
6. Proxmox Backup Server und Restore
7. Proxmox HA und Fencing
8. Proxmox und Ceph Monitoring
9. Proxmox Upgrade und Maintenance
10. Proxmox Security Hardening
11. [Proxmox VE: Migrating CephX from AES to AES256K with Rook/OKD (current part)](https://forgeone.solutions/en/tech/proxmox-ve-cephx-aes256k-rook-okd-migration)
12. Proxmox Disaster Recovery

## Further content

- [Proxmox VE](https://forgeone.solutions/en/products/proxmox-virtual-environment)
