Chaos Engineering · CHAOS-1407

Cross-Cluster Live Migration

Chaos & Resilience Testing

Darshan Jain

Chaos Engineering Team

01 What is CCLM and how it works
02 How we tested — vmshift-validator
03 Spec-driven agentic runs — cclm-test
04 Recommendations, bugs, next steps

Act 2 · Context

What is CCLM?

OpenShift Virtualization

Run VMs as pods on OpenShift — upstream is KubeVirt. VMs get all the Kubernetes advantages: scheduling, autoscaling, self-healing, monitoring — one platform for VMs and containers.

CCLM — a feature of OpenShift Virtualization

Cross-cluster live migration moves a running VM from one OpenShift cluster to another with no workload disruption — memory copied live over cross-cluster network, guest state preserved (same PIDs, no data gap), orchestrated by MTV/Forklift.

Where it's used

Keeping critical VMs online through infrastructure change — Tech Preview in OpenShift Virtualization 4.20.

cluster maintenance & evacuation hardware refreshes cluster consolidation regional workload mobility

Three ideas before the deep-dive: the platform, the feature, the reason it matters.

Press T for next step

Act 2 · Control plane

How CCLM works — assembling the flow

Blue · Source

Control plane3 masters

Workers10 · 13 nodes total

Per nodeR660 · 512 GB · 110 cores · 25GbE

StorageNFS-CSI (RWX)

OpenShift4.21.18 · CNV 4.21

Network198.18.0.0/16 · baremetal-L2

virt-controllerKubeVirt

virt-apiKubeVirt

virt-handlerproxy :9185

Running VMguest apps · file / sqlite / http

virt-launcher · SOURCE QEMUstatus: Running

Green · Target + MTV

Control plane3 masters

Workers10 · 13 nodes total

Per nodeR660 · 512 GB · 110 cores · 25GbE

StorageNFS-CSI (RWX shared)

OpenShift4.21.18 · MTV / Forklift

Network198.19.0.0/16 · br-migration L2

virt-controllerKubeVirt

virt-apiKubeVirt

virt-handlerproxy :9185

forklift-controllerMTV orchestrator

forklift-apiMTV

Plantype: live · Ready

Migration CR phases

Target VM skeletonempty virt-launcher

VMIMKubeVirt · memory job

Two OpenShift clusters. One VM will move live from Blue to Green.

Press T for next step

Act 2 · Data plane

Memory page transfer — Source → Target

BLUE · SOURCE virt-launcher

Guest Running

PHASE 1

Pre-copy

bulk RAM

PHASE 2

Dirty pages

rate → 0

PHASE 3

Cutover

~65 ms

GREEN · TARGET virt-launcher

Waiting

Disk is already shared. Only guest memory must move over br-migration.

Press T for next step

Act 3 · How we tested

vmshift-validator pipeline

Guest-level truth — CR Succeeded is not enough. PIDs, SQLite, logs, and domjobinfo prove live migration.

1/4

VERIFY workloads

SSH into source VM
Confirm file-writer, SQLite, HTTP up

Why: baseline health before migrate

→

2/4

PRE snapshot

Line counts, PIDs
SQLite integrity, Prometheus

Why: compare after cutover

→

3/4

MIGRATE + WAIT

Plan Ready → Migration
Prep → Sync → Done

Why: wait for completion

→

4/4

POST check

SSH on Green; compare pre vs post
domjobinfo metrics

Why: prove live (same PIDs)

→

OUT

summary.json PASS

PIDs unchanged
Data continued, no gap

Verdict artifact

github.com/redhat-chaos/vmshift-validator · cloud29 · Blue→Green · OCP 4.21.18

Independent guest validation is how we catch silent Forklift failures.

Press T for next step

Act 4 · Spec-driven

Chaos scenario catalog

One folder per ID · scenario-spec.md is the contract · optional reports/

Pod kills — virt-launcher, handler, controller, Forklift

Network — latency, loss, partition, port 9185, DNS, blackout

Resource stress — CPU / memory on workers & control plane

Storage — DataVolume disruption

Control plane — API, etcd, master CPU/mem

F / G

Node drain · IPMI power-off during VMIM

Scale — 5 / 20 / 50 VMs concurrent

Compound faults — dual kills, network + pod

37+ scenarios under cclm-chaos/scenarios — specs drive reproducible runs.

Press T for next step

Act 4 · cclm-test skill

Agent-driven testing workflow

.claude/skills/cclm-test · trigger: “test A1” / “run chaos B3” / free-form fault

Understand spec

Read scenario-spec.md
Krkn type + trigger gate

Analyze cluster

Connectivity + discover-vms
Forklift ready · krknctl

Prepare

Sync bastion · resolve params
Prefer chaos-trigger.sh

Confirm with user

Show test plan
Wait for human OK

Execute mig + chaos

Start migration
Inject at VMIM gate

Verify injection

Events / pod / tc evidence
Else: no trusted verdict

Analyze results

Outcome + timing grade
Guest integrity vs spec

Report + bugs

Report template
Bug file → bug-tracker

cclm-test · bastion ssh

cclm-test: natural language → disciplined chaos run.

Press T for next step

Act 4 · Templates

Predefined report & bug guidelines

report-guidelines.md · 14 sections

Executive Summary
Test Environment
Scenario Description
Migration Configuration (CRs)
Chaos Injection Method
How to Reproduce
Test Results
Performance Impact
Workload / Data Integrity
Bugs Found
Observations for Developer
Observations for Engineer
Verdict Summary
Artifact Paths

bug-guidelines.md

When to file (unexpected + reproduced + product)
Summary
Root Cause
Observed vs Expected
Evidence (timestamps, pod names)
Steps to Reproduce (copy-paste)
Per-iteration table
Suggested Fix → bug-tracker.md

Templates keep 168+ runs consistent and engineer-ready.

Press T for next step

Act 5 · Customer recommendations

What customers should do

1. Always verify VM health independently — do not trust Forklift status alone
2. Patch target runStrategy back to Always after migration
3. Monitor and clean orphaned target resources before retry
4. Batch Plan creation (groups of 10, 30s gap) for 50+ VMs
Network: Protect br-migration first (<5% loss). <10 ms RTT low risk; >15 ms budget extra time

Known customer concurrency (awels): 5 VMs/cluster, 2/node. Our scale & chaos testing far exceeds that headroom.

Act 5 · Product impact

Bugs we found — 6 OCPBUGS

Bug	Summary	Severity
OCPBUGS-99403	Forklift reports Succeeded when source VMIM Failed	Critical
OCPBUGS-101771	Target VM runStrategy stuck at Halted after CCLM	Critical
OCPBUGS-104616	VM runs on both clusters after CCLM + node drain	Critical
OCPBUGS-101763	Webhook rejects Plans at 50+ concurrent creations	Critical
OCPBUGS-99404	RBAC cleanup failure after failed CCLM	Normal
OCPBUGS-101764	Single-replica forklift-api limits horizontal scaling	Normal

Spotlight OCPBUGS-99403: WaitForStateTransfer → Completed without checking VMIM Succeeded — why guest validation exists.

Act 6 · Close

What we proved · What’s next

Proved: Network resilience PASS · E6/E7 control-plane stress PASS · guest-level live migration validation works
Impact: 37+ scenarios · 168+ runs · 6 OCPBUGS · recommendations ready for product docs

Agentic next

Agentic Scenario Discovery — AI-designed scenarios (storage · control plane · busy-neighbor IO) · test-index README · Prometheus migration-metrics RFC
Lead agent — one long-lived orchestrator: owns the plan and the verdict, dispatches work, patches the team
- Short-lived specialist workers, one per task: issue-triage · krkn-prepare · observer · recovery · report-writer · analyst
- Model per job: fast tier for polling/triage, mid tier for commands/reports, strongest for diagnosis & judgment
Shared state files — coordination through files on disk, not conversation
- run-state.md · design.md · postmortems/ — every agent reads and writes the same state
Self-improving — when a worker fails, the Lead learns from it
- Failure → postmortem → Lead patches that subagent's prompt → relaunch improved worker next run
- Seed already exists: the agent's 401 stale-token fix, saved to memory in the real run

Act 6 · Close

Thank you

Q&A

darjain@redhat.com