Proxmox VE: Migrating CephX from AES to AES256K with Rook/OKD

Ceph Squid and current Proxmox VE releases make it visible when CephX keys still use the legacy aes key type or when old ciphers are still allowed. That does not automatically mean a storage outage. A cluster can replicate data cleanly, have all OSDs up/in, and still report CephX warnings.

The migration becomes critical in two areas: running consumers and external clients. A pure Proxmox/Ceph cluster can be migrated in a comparatively controlled way with the Proxmox migration helper. If a Kubernetes or OKD cluster is attached through Rook and Ceph-CSI, however, its CephX identities and Kubernetes secrets must be handled separately.

This article documents a real migration on a Proxmox VE/Ceph cluster with Ceph Squid 19.2.6 and an externally attached OKD/Rook cluster. It is not a universal copy-and-paste guide. The output of the Proxmox migration helper and the actual consumer inventory of your own cluster remain the authoritative basis for decisions.

Hinweis: Do not copy this blindly into production. Before every step, verify Ceph health, backups, external consumers, Kubernetes secrets, version compatibility, and a recovery path.

Example Setup

The environment in the documented project consisted of:

• Proxmox VE cluster with 4 nodes

• Ceph Squid 19.2.6

• 4 MONs

• 4 MGRs

• 4 MDS daemons

• 16 OSDs

• CephFS and RBD

• external OKD/Kubernetes cluster

• Rook CephCluster in external mode

• active Ceph-CSI RBD provisioning

• existing CephFS credentials in the Rook external setup

The external CephX identities included:

client.csi-rbd-node
client.csi-rbd-provisioner
client.csi-cephfs-node
client.csi-cephfs-provisioner
client.healthchecker

The target state was a fully migrated setup:

auth_service_cipher: aes256k
auth_allowed_ciphers: aes256k
auth_preferred_cipher: aes256k

At the end, Ceph reported:

HEALTH_OK
16/16 OSD up/in
209 PGs active+clean

1. Check the Initial State

Before rotating anything, start with a sober inventory:

ceph -s
ceph versions
pveceph auth status

The important signals are not only CephX warnings, but the overall state:

• MON quorum available?

• OSDs up and in?

• PGs active+clean?

• MDS and CephFS healthy?

• Which CephX warnings are active?

• Which clients are still visible with the old key type or old sessions?

At the beginning, warnings like these may appear:

AUTH_INSECURE_CLIENT_KEY_TYPE
AUTH_INSECURE_KEYS_ALLOWED
AUTH_INSECURE_KEYS_CREATABLE
AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE
AUTH_INSECURE_SERVICE_KEY_TYPE
AUTH_INSECURE_SERVICE_TICKETS

These messages do not automatically mean data is degraded. They first indicate that CephX keys, service tickets, or allowed ciphers are not yet in the desired state. Still, they must be taken seriously because disabling aes too early can lock out old consumers.

2. Upgrade All Ceph Daemons First

The migration should only start once MON, MGR, MDS, and OSD daemons run a version that supports aes256k. In the project, Ceph was upgraded from 19.2.3 to 19.2.6.

After upgrades, service restarts, or rolling reboots, check again:

ceph versions
pveceph auth status

The important part was:

Monitor quorum
aes256k capable: yes

During rolling reboots, only one Proxmox node was rebooted at a time. After each node, the Ceph state was checked again. Only once quorum, OSDs, and PGs were clean again did the next node follow.

3. Use the Proxmox Migration Helper

The central entry point is the Proxmox helper:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --verbose

Without --apply, the helper performs a dry run. That is intentional. The output shows what Proxmox would change, which steps are blocked, and which consumers still need attention.

In the concrete run, the helper planned among other things:

• rotation of MGR keys

• rotation of MDS keys

• rotation of OSD keys

• migration of service tickets

• temporary noout

• monitor elections

A particularly important warning concerns OSD lockbox keys:

Never rotate a 'client.osd-lockbox' key by hand

That warning is serious. OSD lockbox keys are not only stored in the Ceph auth database, but also tied to local metadata. Rotating them manually with ceph auth risks inconsistencies. The helper keeps the auth database and local metadata aligned.

4. Migrate Service Keys

If the dry run reports no blockers:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply

Then verify:

ceph -s
pveceph auth status
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys

A desired intermediate result is:

PASS: Every service key uses 'aes256k',
and so do the service tickets.

Only then does it make sense to look at cluster keys, bootstrap keys, and clients.

5. Cluster Keys, Bootstrap Keys, and OSD Lockbox Keys

In the real run, the helper provided the next step for cluster-internal keys:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply --rotate-cluster-keys

This covers more than simple client keys. Relevant identities include:

• bootstrap keys

• client.crash

• OSD lockbox keys

• other cluster-managed identities

OSD lockbox keys are especially critical. The helper updates not only CephX entries but also the related local metadata. Therefore, “just rotate all keys” is not a safe description of this step.

6. client.admin Is Special

client.admin deserves separate attention. In the project, the dry run showed:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --rotate-admin-key --verbose

more than 200 running client.admin sessions.

That does not mean that more than 200 VMs have to be running. QEMU/RBD clients can open multiple Ceph connections, and a running VM keeps existing Ceph sessions alive. The number of sessions is therefore not a one-to-one VM count.

The rotation is staged:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply --rotate-admin-key

The old key remains temporarily valid while the new key is distributed. Proxmox keyring copies are updated. Existing sessions can continue to live; new sessions must already be established with the new key. Only once all consumers have really been refreshed may the old key be removed finally.

7. CephFS Mounts Can Block the client.admin Refresh

In the project, the CephFS mount at:

/mnt/pve/cephfs

was busy. Diagnosis:

fuser -vm /mnt/pve/cephfs
lsof +D /mnt/pve/cephfs

The cause was running VMs with ISO images inserted directly from CephFS:

/mnt/pve/cephfs/template/iso/...

The fix was not to hard-stop VMs. Instead, the ISO images were removed from the virtual CD-ROM drives. Example:

qm set <VMID> --ide2 none,media=cdrom

The concrete slot must be checked first. Not every VM uses ide2. The key point is that no running VM should still keep an ISO open directly from the CephFS mount.

Then check again:

lsof +D /mnt/pve/cephfs

Once the mount is no longer busy, run the helper again.

8. Refresh Running QEMU/RBD Sessions

After the CephFS refresh, almost 200 old client.admin sessions remained in the example. These were refreshed through live migration of the VMs:

qm migrate <VMID> <TARGET_NODE> --online

A live migration is sufficient in this context because a new QEMU process is created on the target node and its Ceph connections are established with the new key.

For larger environments, a one-time script is useful that:

• records running VMs

• migrates each VM exactly once

• stores successful VMIDs in a state file

• logs errors separately

• selects target nodes based on current load or VM count

• checks the CephX state after each migration

A simple generic sketch:

#!/usr/bin/env bash
set -euo pipefail

STATE=/root/cephx-refreshed-vms.txt
touch "$STATE"
chmod 600 "$STATE"

for vmid in $(qm list | awk 'NR>1 && $3 == "running" {print $1}'); do
  if grep -qx "$vmid" "$STATE"; then
    continue
  fi

  current_node="$(hostname)"
  target_node="<choose-target-node>"

  echo "Migrating VM ${vmid} from ${current_node} to ${target_node}"
  qm migrate "$vmid" "$target_node" --online
  echo "$vmid" >> "$STATE"

  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
done

This is intentionally only a skeleton. In production, target node selection, error handling, and maintenance logic must fit the environment.

9. Confirm client.admin Finally

Only when the helper reports:

Ready for confirmation: client.admin

run:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply --confirm-all-clients-refreshed

Before doing this, also account for external copies of client.admin. These can be manual keyring copies, old automation, backup jobs, or other integrations.

10. External Rook/OKD Consumers

The key difference to a pure Proxmox cluster is that an external Rook cluster uses its own CephX identities. In the project, these were:

client.csi-rbd-node
client.csi-rbd-provisioner
client.csi-cephfs-node
client.csi-cephfs-provisioner
client.healthchecker

The Proxmox helper deliberately does not change these keys automatically. The helper output effectively said:

Left to whoever manages the client that reads them

That is correct. Proxmox does not know the secrets stored in Kubernetes. A CephX rotation on the Ceph side is only complete once the corresponding Kubernetes secrets in the consumer cluster are synchronized as well.

11. Identify the Rook External Cluster

In the OKD/Kubernetes cluster:

oc -n rook-ceph get cephcluster
oc get storageclass
oc -n rook-ceph get secrets

In the project:

CephCluster EXTERNAL=true
StorageClass rook-ceph-block
Provisioner rook-ceph.rbd.csi.ceph.com

The RBD secrets were:

rook-csi-rbd-node
rook-csi-rbd-provisioner

with:

userID
userKey

The important part is alignment: the CephX user on the Ceph side and userKey in the Kubernetes secret must match. Do not copy keys into tickets, chat, Git, or logs.

Before changing Kubernetes secrets, check whether the secret is managed by GitOps or a controller:

oc -n rook-ceph get secret <SECRET> -o json | jq '.metadata | {ownerReferences, annotations, labels}'

ownerReferences, ArgoCD tracking labels, and the annotation kubectl.kubernetes.io/last-applied-configuration are especially important. If that annotation contains old secret values, it can later be accidentally reapplied or become visible in backups or exports. In our run, those metadata fields were checked before changes, and stale last-applied annotations were removed deliberately.

12. Special Case: ceph-csi Version and AES256K

A real blocker appeared directly with an external RBD key. After rotating one RBD key, provisioning initially failed:

connecting failed: rados: ret=-22, Invalid argument

The error could be narrowed down to AES256K compatibility of the CSI stack in use. After updating to a compatible Rook/ceph-csi state, provisioning worked again. The verified state in the OKD cluster was:

Rook Operator: v1.20.7
Ceph-CSI: v3.17.1

Matching CSI sidecars were also used, including:

csi-provisioner v5.2.0
csi-attacher v4.8.1
csi-resizer v1.13.2
csi-snapshotter v8.2.1
csi-node-driver-registrar v2.13.0

Before rotating external CSI keys, verify that the deployed Rook/ceph-csi version supports the new key type. Otherwise the key may be rotated correctly while the Kubernetes consumer is locked out.

13. Rotate the RBD Provisioner Key

On the Ceph side, back up first and then rotate:

umask 077

ceph auth get client.csi-rbd-provisioner \
  > /root/client.csi-rbd-provisioner.before-aes256k.keyring

ceph auth rotate --key-type=aes256k client.csi-rbd-provisioner \
  > /root/client.csi-rbd-provisioner.aes256k.keyring

chmod 600 /root/client.csi-rbd-provisioner*.keyring

The new key is then safely written to the Kubernetes secret:

rook-csi-rbd-provisioner

Only update the secret value, not the userID. After ceph auth rotate, the old key is no longer the current key; therefore the secret update must follow immediately and in a controlled way before new provisioner connections are tested.

Important: after a rotation, do not copy back from an old local file, an old chat snippet, or a previous secret. The currently valid key is retrieved directly from Ceph:

ceph auth get-key client.csi-rbd-provisioner

If a key is accidentally rotated twice, the first new key is already obsolete again. In that case, synchronize exactly the currently valid key, not the key from the first rotation attempt.

After the update, the RBD provisioner was refreshed and a new PVC was provisioned. Only once a new PVC becomes Bound has the provisioner path been tested in practice.

14. Rotate the RBD Node Key

Analogously:

ceph auth rotate --key-type=aes256k client.csi-rbd-node

Kubernetes secret:

rook-csi-rbd-node

Then test the node plugin path in practice:

• new PVC

• pod mounts PVC

• write a file

• read the file

• delete the pod

• recreate the pod

• read the file again

In the project, this end-to-end test succeeded. That matters: provisioning alone does not prove that node staging, mounting, and pod access work.

15. Check Active Sessions

The working path for session inspection was:

ceph tell mon.<MON-NAME> sessions --format json-pretty

Do not use:

ceph sessions

With multiple MONs, ceph tell mon.* sessions can produce multiple JSON outputs. That is not necessarily friendly for direct jq processing. It is better to check per MON or split the output intentionally.

In the project, active RBD sessions could be observed with:

auth_key_type: aes256k

Missing active sessions are not proof that an identity is no longer needed. Provisioner accounts may only appear for new PVCs, CephFS credentials may be prepared even when no CephFS PVCs currently exist, and healthchecker access depends on Rook operator behavior.

16. CephFS CSI Keys

These accounts were also rotated:

client.csi-cephfs-provisioner
client.csi-cephfs-node

The corresponding secrets:

rook-csi-cephfs-provisioner
rook-csi-cephfs-node

were updated.

At that point, the concrete cluster had:

• no CephFS StorageClass

• no CephFS PVCs

• no running CephFS CSI pods

Nevertheless, the credentials were synchronized because they were part of the external Rook setup. “No active session right now” does not automatically mean “credential is no longer needed.” Deleting it would be a separate architecture decision, not a side step of a key migration.

17. client.healthchecker

The final external key was:

client.healthchecker

It was stored in the secret:

rook-ceph-mon

with the fields:

ceph-username
ceph-secret

The decoded username was:

client.healthchecker

Rook uses this user for external cluster health and monitor access. The operator logs confirmed this:

will use "client.healthchecker" to check health and monitor status

After rotation:

ceph auth rotate --key-type=aes256k client.healthchecker

only ceph-secret in the secret rook-ceph-mon was updated.

Immediately afterwards, the Rook CephCluster briefly reported:

HEALTH_ERR
RADOS permission denied

The running operator still had the old healthchecker access active. A targeted operator refresh was then required:

oc -n rook-ceph rollout restart deployment/rook-ceph-operator

After the restart, rook-ceph-external was back to:

Synced / Healthy

and the CephCluster warnings were reduced to the global cipher settings.

Before the patch, it was verified that the secret would not be reset automatically through OwnerReferences or ArgoCD. If GitOps manages this secret, the new value must be updated in the authoritative source, otherwise the cluster can later lose the correct key again.

18. Reconcile Externally Rotated Keys with the Proxmox Helper

Because the five external keys were rotated outside the Proxmox script, the helper reported that individual keys had been changed outside the script.

For each external consumer, a first measurement was triggered:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --confirm-clients-refreshed <CLIENT> --apply

On the first run, the expected message was effectively:

not accepting ... yet
complete measurement is now recorded

This first FAIL is not automatically a cluster error in this context. The helper does not yet have a complete measurement for that external consumer and records it first. Only after all relevant external consumers have been measured can the dry run show that confirmation is possible.

If the helper reports full inventory unavailable, its statement is also limited: then the helper cannot safely evaluate the complete consumer inventory. In that case, the final lockdown must not be derived from partial information alone.

After the first measurement for all five clients, the dry run reported:

Ready for confirmation:
client.csi-cephfs-node,
client.csi-cephfs-provisioner,
client.csi-rbd-node,
client.csi-rbd-provisioner,
client.healthchecker

Only then was the helper ready to confirm the consumers as refreshed.

19. Disable AES Finally

Hinweis: Important: only run the final cipher lockdown once the Proxmox helper suggests it after complete measurement and all external consumers demonstrably work with the new keys.

Only now did the helper provide the final command:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys \
  --apply \
  --confirm-all-clients-refreshed \
  --restrict-ciphers

From this point on, old aes keys are actually rejected. Anyone who missed an external consumer can lock it out of Ceph with this step.

The expected completion:

PASS: only the 'aes256k' cipher is allowed for authentication now
PASS: Cephx migration is complete. Authentication and new keys use only 'aes256k'.

20. Final Status

Final verification:

pveceph auth status
ceph -s
ceph health detail

Final state:

Cephx health checks
none active

auth_service_cipher: aes256k
auth_allowed_ciphers: aes256k
auth_preferred_cipher: aes256k

and:

HEALTH_OK

In the example setup:

mon: 4 daemons in quorum
osd: 16 up / 16 in
mds: healthy
209 PGs active+clean

21. Do Not Delete the Recovery Journal Too Early

The helper works with a journal:

/etc/pve/priv/cephx-key-migration.json

This file can contain recovery information and previous keys. Therefore:

• protect file permissions

• do not publish it

• do not copy it into repositories

• do not place it uncontrolled into less protected backups

• clean it up only after completed access verification according to the Proxmox documentation

22. Lessons Learned

The most important points from the run:

• First bring all Ceph daemons to a compatible version.

• Take helper dry runs seriously.

• Never rotate OSD lockbox keys manually.

• client.admin requires a consumer refresh.

• Live migration works well to refresh QEMU/RBD sessions.

• CephFS can remain busy because of inserted ISO images.

• External Rook/CSI keys are not maintained automatically by Proxmox.

• Kubernetes secrets must be synchronized immediately with the CephX keys.

• Check ceph-csi compatibility before rotating external CSI keys to AES256K.

• Run practical provisioning and mount tests.

• Disable aes only at the very end.

• Run the final --restrict-ciphers step only when the helper suggests it after complete measurement.

23. Security Notes

Hinweis: Do not copy these commands blindly into production.

Before every step, check:

• Ceph health

• backups

• consumer list

• external clusters

• Kubernetes secrets

• version compatibility

• maintenance window

• recoverability

• monitoring and alerting

The most important failure class is not the rotation itself, but a missed consumer. If old external consumers are overlooked and aes is disabled too early, those consumers can be locked out of Ceph.

Conclusion

The actual CephX migration inside the Proxmox cluster is only part of the work. External consumers such as Rook and Ceph-CSI are the decisive part. Only once their credentials have been updated, consumers refreshed, and storage functions practically tested may the old cipher be disabled.

In this project, the final state was clean: only aes256k allowed, Ceph HEALTH_OK, all OSDs up/in, all PGs active+clean, and the external Rook/OKD consumer back to Synced / Healthy.