CCE Cluster Upgrade Planner

Overview

Plan and assess CCE (Cloud Container Engine) Kubernetes cluster version upgrades using hcloud CLI. Covers upgrade path validation, 76-item pre-check checklist, addon compatibility, version-specific breaking changes, deprecated APIs, upgrade window estimation, and execution preview with two-step confirmation.

Architecture: hcloud CLI → CCE OpenAPI → Cluster Info / Upgrade Paths / Upgrade Workflow / Addon Info / Node Pool Info

Nodes

Standard workflow:

1. Collect cluster current state (version, nodes, addons, node pools)
2. Query upgrade paths via ListClusterUpgradePaths
3. Run pre-check via ShowClusterUpgradeInfo (or CreateUpgradeWorkFlow)
4. Evaluate addon compatibility for each installed addon
5. Estimate upgrade window based on node count, addon count, batch strategy
6. Generate upgrade plan with specific hcloud CLI commands (execution preview)
7. Provide rollback strategy and post-upgrade verification checklist

Prerequisites

Prerequisite check: hcloud (KooCLI) >= 7.2.2 required Run hcloud version to verify, and hcloud configure list to check profile exists.

hcloud version
hcloud configure list

Security Constraints

Dangerous Operation Confirmation Mechanism

This skill strictly enforces a two-step confirmation mechanism for upgrade execution preview.

All upgrade execution previews require explicit user confirmation before any changes are made. The process:

Step 1: Preview — Show upgrade commands, target version, affected components, and risk warnings

Step 2: Confirm & Execute — Only after user explicitly confirms

Operations Requiring Confirmation

Credential Security

• Never expose AK/SK values in conversation, commands, or output
• Never ask user to input AK/SK directly in conversation
• Only use hcloud configure list to check credential status (presence only, not values)
• Prefer profile mode or environment variables over explicit AK/SK parameters

Command Format Standard

CCE upgrade follows standard hcloud format:

hcloud CCE \x3COperation> --param=value --cli-region=\x3Cregion> --cli-output=json

Upgrade-specific Parameter Rules

1. Cluster ID required: all upgrade operations need --cluster_id
2. Target version required: upgrade operations need --spec.clusterUpgradeAction.targetVersion=v1.XX
3. Addon upgrade uses array format: --spec.clusterUpgradeAction.addons.1.addonTemplateName=\x3Cname> --spec.clusterUpgradeAction.addons.1.version=\x3Cver>
4. Node pool priority uses key-value format: --spec.clusterUpgradeAction.nodePoolOrder.key1=value1
5. Node selector uses nested format: --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.key=\x3Clabel-key> --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.operator=In --spec.clusterUpgradeAction.nodeOrder.key1.1.nodeSelector.value.1=\x3Cval>
6. Upgrade strategy: --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate (only in-place supported)
7. Batch size: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=\x3C1-40> (default 20, recommended)
8. Batch scope: --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster or NodePool

⚠️ Critical: Before constructing any upgrade command, always run hcloud CCE \x3COperation> --help to verify exact parameter names. CCE upgrade APIs have hundreds of parameters; the help output is the authoritative source.

Scenario Routing

Core Commands

Step 1: Collect Cluster Current State

# Get cluster details (version, node count, addons)
hcloud CCE ShowCluster --cluster_id=\x3Ccluster-id> --detail=true --cli-region=\x3Cregion> --cli-output=json

# List installed addons
hcloud CCE ListAddonInstances --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# List node pools
hcloud CCE ListNodePools --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# List nodes
hcloud CCE ListNodes --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

Step 2: Query Upgrade Paths

# Get all available upgrade paths for the cluster
hcloud CCE ListClusterUpgradePaths --cli-region=\x3Cregion> --cli-output=json

Step 3: Pre-Upgrade Check

# Get upgrade info (includes pre-check status)
hcloud CCE ShowClusterUpgradeInfo --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# Create upgrade workflow (triggers pre-check automatically)
hcloud CCE CreateUpgradeWorkFlow \
  --cluster_id=\x3Ccluster-id> \
  --apiVersion=v3 \
  --kind=WorkFlowTask \
  --spec.targetVersion=\x3Ctarget-version> \
  --spec.clusterVersion=\x3Ccurrent-version> \
  --cli-region=\x3Cregion> \
  --cli-output=json

Step 4: Addon Compatibility Check

# Get addon details (check version compatibility)
hcloud CCE ListAddonInstances --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# List addon template versions (check available versions for target K8s)
hcloud CCE ListAddonTemplates --cluster_id=\x3Ccluster-id> --addon_template_name=\x3Caddon-name> --cli-region=\x3Cregion> --cli-output=json

Step 5: Upgrade Execution Preview

# Preview cluster upgrade (without confirm=true, no execution)
hcloud CCE UpgradeCluster \
  --cluster_id=\x3Ccluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=\x3Ctarget-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.scope=Cluster \
  --cli-region=\x3Cregion> \
  --cli-output=json

With addons (specify addon upgrade during cluster upgrade):

hcloud CCE UpgradeCluster \
  --cluster_id=\x3Ccluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=\x3Ctarget-version> \
  --spec.clusterUpgradeAction.strategy.type=inPlaceRollingUpdate \
  --spec.clusterUpgradeAction.strategy.inPlaceRollingUpdate.userDefinedStep=20 \
  --spec.clusterUpgradeAction.addons.1.addonTemplateName=coredns \
  --spec.clusterUpgradeAction.addons.1.operation=patch \
  --spec.clusterUpgradeAction.addons.1.version=\x3Ctarget-addon-version> \
  --cli-region=\x3Cregion> \
  --cli-output=json

With node pool priority (control upgrade order):

hcloud CCE UpgradeCluster \
  --cluster_id=\x3Ccluster-id> \
  --metadata.apiVersion=v3 \
  --metadata.kind=UpgradeTask \
  --spec.clusterUpgradeAction.targetVersion=\x3Ctarget-version> \
  --spec.clusterUpgradeAction.nodePoolOrder.\x3Cnodepool-id>=\x3Cpriority> \
  --cli-region=\x3Cregion> \
  --cli-output=json

Step 6: Monitor Upgrade Progress

# Check upgrade workflow status
hcloud CCE ShowUpgradeWorkFlow \
  --cluster_id=\x3Ccluster-id> \
  --upgrade_workflow_id=\x3Cworkflow-id> \
  --cli-region=\x3Cregion> \
  --cli-output=json

# List upgrade tasks
hcloud CCE ListUpgradeClusterTasks --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# Pause upgrade (if issues found)
hcloud CCE PauseUpgradeClusterTask --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# Continue paused upgrade
hcloud CCE ContinueUpgradeClusterTask --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# Retry failed upgrade
hcloud CCE RetryUpgradeClusterTask --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

Step 7: Cancel Upgrade Workflow

# Cancel upgrade workflow (status must not be Running/Success/Cancel)
hcloud CCE UpgradeWorkFlowUpdate \
  --cluster_id=\x3Ccluster-id> \
  --upgrade_workflow_id=\x3Cworkflow-id> \
  --status.phase=Cancel \
  --cli-region=\x3Cregion> \
  --cli-output=json

Upgrade Window Estimation Formula

See references/upgrade-window-estimation.md for detailed formula and examples.

Quick estimation:

T_total = T_control_plane + T_node_batch + T_addon + T_verify + T_buffer

T_control_plane = 10-30 minutes
T_node_batch = N_nodes / batch_size * (5-15 min/node)
T_addon = count(addons) * (5-15 min/addon)
T_verify = 30 minutes
T_buffer = 20% * (T_control_plane + T_node_batch + T_addon)

Example: 3-node-pool cluster (10 nodes, 5 addons) upgrading v1.23→v1.25:

• Control plane: 15 min
• Node batches: 10/4 = 3 batches (1+4+5 nodes), ~9 min each = 9 min total (first batch 1 node at 5 min, second 4 nodes at 8 min, third 5 nodes at 10 min)
• Addon upgrade: 5 * 10 min = 50 min
• Verification: 30 min
• Buffer (20%): ~19 min
• Total: ~123 min (2h 3min)

Upgrade Methods

Key Constraints

• No skip-version upgrades: must follow upgrade path step by step (e.g., v1.21→v1.23→v1.25→v1.27→v1.28)
• Patch version first: upgrade to latest patch before major version upgrade
• Control plane before nodes: upgrade control plane first, then node pools
• Addons after cluster: upgrade addons after cluster reaches target version
• v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes, IP addresses change
• Autoscaling paused: during control plane upgrade, autoscaling is paused; resumes after control plane done, but shrink waits until full completion

Parameter Reference

CCE Upgrade API Parameters

Upgrade Workflow Parameters

Output Format

Upgrade Assessment Report

{
  "cluster_id": "abc123",
  "current_version": "v1.23",
  "target_version": "v1.25",
  "upgrade_paths": ["v1.23 → v1.24 → v1.25"],
  "pre_check_status": "passed|failed|warning",
  "pre_check_items": { "total": 76, "passed": 72, "failed": 4 },
  "addon_compatibility": [
    { "addon": "coredns", "current": "1.23", "compatible": true }
  ],
  "estimated_window_minutes": 123,
  "risk_level": "low|medium|high",
  "recommended_strategy": "inPlaceRollingUpdate",
  "execution_commands": ["hcloud CCE UpgradeCluster ..."]
}

Upgrade Execution Preview

{
  "cluster_id": "abc123",
  "step": "cluster_upgrade_preview",
  "target_version": "v1.25",
  "strategy": "inPlaceRollingUpdate",
  "batch_size": 20,
  "addons_upgrade": ["coredns → 1.25"],
  "estimated_time": "2h 3min",
  "confirmation_required": true,
  "warning": "Control plane upgrade is irreversible once started"
}

Verification

Pre-Upgrade Verification

hcloud CCE ShowClusterUpgradeInfo --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

• Verify preCheckStatus is passed
• Check all 76 items in preCheckItems

Post-Upgrade Verification

# Check cluster version
hcloud CCE ShowCluster --cluster_id=\x3Ccluster-id> --detail=true --cli-region=\x3Cregion> --cli-output=json

# Check addon status
hcloud CCE ListAddonInstances --cluster_id=\x3Ccluster-id> --cli-region=\x3Cregion> --cli-output=json

# Check node status
kubectl get nodes

# Check workload health
kubectl get pods -A

Notes

• No skip-version upgrades: must follow upgrade path step by step
• In-place upgrade is the only supported strategy: inPlaceRollingUpdate only
• v1.28+ control node IP change: upgrading to v1.28+ creates new control nodes with different IP addresses
• Autoscaling paused during upgrade: cluster autoscaling pauses during control plane upgrade
• Addon upgrade must follow cluster upgrade: addons upgrade after cluster reaches target version
• Pre-check is mandatory: never skip ShowClusterUpgradeInfo or CreateUpgradeWorkFlow pre-check

Best Practices

1. Always run pre-check first — use CreateUpgradeWorkFlow or ShowClusterUpgradeInfo to validate readiness
2. Upgrade patch version first — ensure current patch is latest before major version upgrade
3. Business low-traffic window — schedule upgrades during low-traffic periods
4. Keep ≥2 nodes per node pool for redundancy during upgrade
5. Monitor upgrade progress — check ShowUpgradeWorkFlow status after each phase
6. Verify after each phase — run post-upgrade verification after control plane and node upgrades
7. Rollback plan — use backup data and PauseUpgradeClusterTask if issues arise

References