\r \r

Huawei Cloud CCE-CCI Bursting Deployer\r

\r

Overview\r

\r This skill configures CCE workloads to burst into CCI 2.0 serverless\r capacity for elastic scaling. It automates the full workflow: precheck\r cluster readiness including NodeCheck for physical-node headroom, ensure\r VPCEP dependencies for image pulling, install or update the\r virtual-kubelet addon, discover tenant-owned smoke images, deploy a\r smoke workload, verify the virtual node and pod status, and diagnose\r addon failures.\r \r Architecture: python3 scripts/huawei-cloud.py → CCE/VPCEP/VPC API →\r Precheck → NodeCheck → VPCEP creation → Addon install → Image discovery → Smoke deployment → Verification → Diagnosis\r \r Key Principle: Preview-first. Read-only checks (precheck,\r NodeCheck, verify, discover images, diagnose addon) run immediately,\r but VPCEP creation, addon installation, node pool changes, and smoke\r workload deployment require explicit user approval with confirm=true\r before execution.\r \r Related Skills:\r \r

• huawei-cloud-cce-cluster-management - Cluster lifecycle,\r addon listing, kubeconfig retrieval\r
• huawei-cloud-cce-pod-failure-diagnoser - Pod failure diagnosis\r when bursting pods fail to reach Running\r
• huawei-cloud-cce-network-failure-diagnoser - Network diagnosis\r for VPCEP connectivity issues\r \r

Prerequisites\r

\r

1. Python Requirements (MANDATORY)\r

\r

• Python 3 installed (version >= 3.8)\r
• Run python3 --version to verify installation\r
• huawei-cloud.py script available in the scripts directory\r
• Required packages: huaweicloudsdkcce, huaweicloudsdkcore, huaweicloudsdkvpc, huaweicloudsdkvpcep, kubernetes\r \r

2. Credential Configuration\r

\r

• Valid Huawei Cloud credentials (AK/SK mode)\r
• Security Rules:\r
  • Never expose AK/SK values in code, conversation, or commands\r
  • Never use echo commands to check credential environment variables\r
  • Use environment variables: HUAWEI_AK, HUAWEI_SK, HUAWEI_PROJECT_ID\r
  • Alternative env vars: HUAWEICLOUD_SDK_AK, HUAWEICLOUD_SDK_SK, HW_ACCESS_KEY, HW_SECRET_KEY\r
  • Prefer IAM users over root account for cloud operations\r
  • Never persist AK/SK in skill files, debug files, reports, or shell history\r \r Configuration Method (Environment Variables Only):\r \r

export HUAWEI_AK=\x3Cyour-ak>\r
export HUAWEI_SK=\x3Cyour-sk>\r
export HUAWEI_PROJECT_ID=\x3Cyour-project-id>\r
```\r
\r
### 3. IAM Permission Requirements\r
\r
| API Action | Permission | Purpose |\r
| --- | --- | --- |\r
| `cce:cluster:get` | Get cluster details | Read cluster network spec (VPC, subnet, ENI) |\r
| `cce:addon:list` | List addons | Check virtual-kubelet installation state |\r
| `cce:addon:create` | Install addon | Install virtual-kubelet addon |\r
| `cce:addon:update` | Update addon | Configure virtual-kubelet bursting parameters |\r
| `vpcep:endpoint:create` | Create VPCEP | Create SWR/OBS interface endpoints |\r
| `vpcep:endpoint:list` | List VPCEP | Check existing VPCEP endpoints |\r
| `vpcep:service:list` | List VPCEP services | Discover public service details |\r
| `vpc:subnet:list` | List subnets | Validate subnet IDs in cluster VPC |\r
| `vpc:routetable:list` | List route tables | Find route table IDs for OBS gateway VPCEP |\r
\r
**Permission Failure Handling**: When any command fails due to IAM\r
permission errors, verify the permissions listed above, guide the\r
user to create custom IAM policies, and pause execution until\r
permissions are confirmed.\r
\r
## Core Commands\r
\r
All tools are invoked through `python3 scripts/huawei-cloud.py` with key=value parameters.\r
\r
### 1. Precheck (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_precheck_cce_cci_bursting region=cn-north-4 cluster_id=\x3Ccluster-id> vpcep_subnet_id=\x3Cvpc-subnet-id>\r
```\r
\r
Inspects cluster networking, resolves subnet roles, checks addon state, and reports any blocking issues. The cluster must be a Turbo/ENI cluster for CCI bursting.\r
\r
### 2. Ensure VPCEP Dependencies\r
\r
```bash\r
# Preview (no mutation)\r
python3 scripts/huawei-cloud.py huawei_ensure_cce_cci_vpcep region=cn-north-4 cluster_id=\x3Ccluster-id> vpcep_subnet_id=\x3Cvpc-subnet-id>\r
\r
# Apply after user approval\r
python3 scripts/huawei-cloud.py huawei_ensure_cce_cci_vpcep region=cn-north-4 cluster_id=\x3Ccluster-id> vpcep_subnet_id=\x3Cvpc-subnet-id> confirm=true\r
```\r
\r
Creates or reuses SWR, SWR-API, and OBS-compatible VPCEP interface\r
endpoints in the cluster VPC. If OBS information is missing, pass\r
`obs_endpoint_service_name` obtained from the Huawei Cloud service\r
ticket (do not guess).\r
\r
### 3. Setup CCI Bursting\r
\r
```bash\r
# Preview\r
python3 scripts/huawei-cloud.py huawei_setup_cce_cci_bursting region=cn-north-4 cluster_id=\x3Ccluster-id> vpcep_subnet_id=\x3Cvpc-subnet-id>\r
\r
# Apply after user approval\r
python3 scripts/huawei-cloud.py huawei_setup_cce_cci_bursting region=cn-north-4 cluster_id=\x3Ccluster-id> vpcep_subnet_id=\x3Cvpc-subnet-id> confirm=true\r
```\r
\r
Ensures VPCEP dependencies, installs `virtual-kubelet` if absent, and configures CCI network parameters. Idempotent: updates existing addon configuration without uninstalling.\r
\r
### 4. Verify Bursting Readiness (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_verify_cce_cci_bursting region=cn-north-4 cluster_id=\x3Ccluster-id>\r
```\r
\r
Checks addon state, virtual node readiness, and optional workload pod status. Returns `ready=true` when the virtual node is Ready and all workload pods are Running on it.\r
\r
### 5. Deploy Smoke Workload\r
\r
```bash\r
# Preview\r
python3 scripts/huawei-cloud.py huawei_deploy_cce_cci_smoke_workload region=cn-north-4 cluster_id=\x3Ccluster-id> replicas=2\r
\r
# Apply after user approval\r
python3 scripts/huawei-cloud.py huawei_deploy_cce_cci_smoke_workload region=cn-north-4 cluster_id=\x3Ccluster-id> replicas=2 confirm=true\r
```\r
\r
Creates or patches a small Deployment forced onto CCI capacity using the `bursting.cci.io/burst-to-cci: enforce` label. Uses a regional SWR image by default.\r
\r
### 6. Verify Smoke Workload (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_verify_cce_cci_bursting region=cn-north-4 cluster_id=\x3Ccluster-id> namespace=cci2-burst-lab workload_name=cci2-burst-demo\r
```\r
\r
Confirms test pods reach `Running` on the virtual node (`bursting-node` or `virtual-kubelet`).\r
\r
### 7. Check Node Capacity (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_check_cce_cci_node_capacity region=cn-north-4 cluster_id=\x3Ccluster-id>\r
```\r
\r
Inspects physical-node addon headroom. The precheck includes NodeCheck automatically, warning when schedulable capacity is below the 2C/4GiB conservative baseline. Preview node pool expansion\r
when the baseline is not met.\r
\r
### 8. List Node Pools (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_list_cce_nodepools region=cn-north-4 cluster_id=\x3Ccluster-id>\r
```\r
\r
Lists CCE node pools in the cluster. Used after NodeCheck warns about insufficient headroom to identify existing pools for resize.\r
\r
### 9. Resize Node Pool\r
\r
```bash\r
# Preview\r
python3 scripts/huawei-cloud.py huawei_resize_cce_nodepool region=cn-north-4 cluster_id=\x3Ccluster-id> nodepool_id=\x3Cnodepool-id> node_count=3\r
\r
# Apply after user approval\r
python3 scripts/huawei-cloud.py huawei_resize_cce_nodepool region=cn-north-4 cluster_id=\x3Ccluster-id> nodepool_id=\x3Cnodepool-id> node_count=3 confirm=true\r
```\r
\r
Resizes an existing node pool to increase physical-node capacity. Apply only after explicit user approval with `confirm=true`.\r
\r
### 10. Create Node Pool\r
\r
```bash\r
# Preview\r
python3 scripts/huawei-cloud.py huawei_create_cce_nodepool region=cn-north-4 cluster_id=\x3Ccluster-id> nodepool_name=\x3Cname> flavor=\x3Cflavor> availability_zone=\x3Caz> root_volume_size=40 root_volume_type=SSD\r
\r
# Apply after user approval\r
python3 scripts/huawei-cloud.py huawei_create_cce_nodepool region=cn-north-4 cluster_id=\x3Ccluster-id> nodepool_name=\x3Cname> flavor=\x3Cflavor> availability_zone=\x3Caz> root_volume_size=40 root_volume_type=SSD confirm=true\r
```\r
\r
Creates a new node pool. Prefer SSH keypair authentication. Apply only after explicit user approval with `confirm=true`.\r
\r
### 11. Discover Smoke Images (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_discover_cce_cci_smoke_images region=cn-north-4\r
```\r
\r
Discover tenant-owned SWR basic images through namespace, repository, and tag queries. The smoke deployment automatically selects a discovered tenant-owned image when `image` is omitted.\r
\r
### 12. Diagnose Bursting Addon (Read-Only)\r
\r
```bash\r
python3 scripts/huawei-cloud.py huawei_diagnose_cce_cci_bursting_addon region=cn-north-4 cluster_id=\x3Ccluster-id>\r
```\r
\r
Returns read-only addon diagnostics when verification fails or the virtual node does not become Ready. Inspect the returned diagnostics to identify region mismatches, missing project IDs,\r
or configuration issues.\r
\r
## Parameter Reference\r
\r
### Common Parameters\r
\r
| Parameter | Required/Optional | Description | Default |\r
| --- | --- | --- | --- |\r
| `region` | Required | Huawei Cloud region ID | `HUAWEI_AK` env region |\r
| `cluster_id` | Required | CCE cluster ID | N/A |\r
| `ak` | Optional | Access Key (overrides env var) | `HUAWEI_AK` env |\r
| `sk` | Optional | Secret Key (overrides env var) | `HUAWEI_SK` env |\r
| `project_id` | Optional | Project ID (overrides env var) | Auto-resolved via IAM |\r
\r
### Precheck Parameters\r
\r
| Parameter | Required/Optional | Description | Notes |\r
| --- | --- | --- | --- |\r
| `vpcep_subnet_id` | Optional | VPC subnet ID for VPCEP placement | Defaults to cluster host subnet |\r
\r
### VPCEP Parameters\r
\r
| Parameter | Required/Optional | Description | Notes |\r
| --- | --- | --- | --- |\r
| `vpcep_subnet_id` | Optional | VPC subnet ID for VPCEP | Defaults from precheck |\r
| `obs_endpoint_service_name` | Optional | Exact OBS VPCEP service name | Obtain from Huawei Cloud service ticket; do not guess |\r
| `route_table_ids` | Optional | Route table IDs for OBS gateway | Auto-resolved if omitted; comma-separated |\r
| `confirm` | Required for mutation | Approve VPCEP creation | `true` to apply, omit to preview |\r
\r
### Setup Parameters\r
\r
| Parameter | Required/Optional | Description | Notes |\r
| --- | --- | --- | --- |\r
| `vpcep_subnet_id` | Optional | VPC subnet ID for VPCEP | Defaults from precheck |\r
| `cci_subnet_id` | Optional | Neutron subnet ID for addon | Defaults from `spec.eni_network` |\r
| `obs_endpoint_service_name` | Optional | OBS VPCEP service name | Required if precheck reports missing OBS |\r
| `route_table_ids` | Optional | Route table IDs for OBS gateway | Auto-resolved if omitted |\r
| `addon_version` | Optional | virtual-kubelet addon version | Defaults to `1.5.82` or existing version |\r
| `confirm` | Required for mutation | Approve setup | `true` to apply, omit to preview |\r
\r
### Smoke Workload Parameters\r
\r
| Parameter | Required/Optional | Description | Default |\r
| --- | --- | --- | --- |\r
| `namespace` | Optional | Smoke namespace | `cci2-burst-lab` |\r
| `workload_name` | Optional | Smoke Deployment name | `cci2-burst-demo` |\r
| `image` | Optional | Container image | Regional SWR nginx image |\r
| `replicas` | Optional | Pod replica count | `2` |\r
| `confirm` | Required for mutation | Approve deployment | `true` to apply, omit to preview |\r
\r
### Verify Parameters\r
\r
| Parameter | Required/Optional | Description | Notes |\r
| --- | --- | --- | --- |\r
| `namespace` | Optional | Workload namespace | Filter pods and deployments |\r
| `workload_name` | Optional | Workload name | Filter pods by `app` label |\r
\r
### Subnet Role Reference\r
\r
| Parameter | ID Type | Used By |\r
| --- | --- | --- |\r
| `cci_subnet_id` | Neutron subnet UUID | `virtual-kubelet` addon `networkID`, `subnet_id`, `subnets[].subnetID` |\r
| `vpcep_subnet_id` | VPC subnet UUID | VPCEP interface endpoint placement |\r
\r
**These are different ID namespaces. Never swap them.** For a\r
Turbo/ENI cluster, `huawei_precheck_cce_cci_bursting` resolves\r
`cci_subnet_id` from `spec.eni_network`. Pass `vpcep_subnet_id`\r
explicitly when a dedicated endpoint subnet is preferred.\r
\r
## Output Format\r
\r
All tools return JSON with the following structure:\r
\r
| Field | Description |\r
| --- | --- |\r
| `success` | Boolean: `true` if operation succeeded, `false` otherwise |\r
| `action` | Action name that was executed |\r
| `region` | Huawei Cloud region |\r
| `cluster_id` | CCE cluster ID |\r
\r
**Precheck Output**:\r
\r
| Field | Description |\r
| --- | --- |\r
| `network` | Cluster network context (VPC, subnets, ENI) |\r
| `subnet_roles` | Resolved `cci_subnet_id` and `vpcep_subnet_id` |\r
| `virtual_kubelet` | Existing addon info or `null` |\r
| `issues` | List of blocking issues (empty if ready) |\r
\r
**Verify Output**:\r
\r
| Field | Description |\r
| --- | --- |\r
| `ready` | Boolean: addon installed, virtual node Ready, workload pods Running |\r
| `addon` | virtual-kubelet addon details |\r
| `virtual_nodes` | List of virtual nodes in the cluster |\r
| `workload.phase_distribution` | Pod phase counts (Running, Pending, etc.) |\r
| `workload.node_distribution` | Pod node assignment counts |\r
| `warning_events` | Recent warning events for the workload |\r
\r
## Verification\r
\r
### Step-by-step Verification Checklist\r
\r
1. Verify AK/SK credentials are configured via environment variables\r
2. Run `huawei_precheck_cce_cci_bursting` and confirm `issues` is empty\r
3. Verify the cluster is Turbo/ENI type (`container_network_mode` = `eni`)\r
4. Run `huawei_ensure_cce_cci_vpcep` preview and confirm VPCEP plan is correct\r
5. After user approval, apply VPCEP with `confirm=true`\r
6. Run `huawei_setup_cce_cci_bursting` preview, then apply with `confirm=true`\r
7. Run `huawei_verify_cce_cci_bursting` and confirm `ready=true`\r
8. Deploy smoke workload with preview, then `confirm=true`\r
9. Run final verification with namespace and workload_name parameters\r
10. Confirm all pods are `Running` on `bursting-node` or `virtual-kubelet`\r
\r
## Best Practices\r
\r
1. **Always precheck first**: Run `huawei_precheck_cce_cci_bursting` before any mutation to identify blocking issues\r
2. **Preview before apply**: Always run mutation actions without `confirm=true` first, review the plan, then re-run with `confirm=true` after explicit user approval\r
3. **Never swap subnet IDs**: `cci_subnet_id` (Neutron UUID) and `vpcep_subnet_id` (VPC UUID) are different ID namespaces; swapping them causes addon failure\r
4. **Prefer tenant-owned SWR images**: Docker Hub images and public namespace images timeout in CCI capacity; prefer a tenant-owned regional SWR image via `huawei_discover_cce_cci_smoke_images`;\r
   public namespace images are fallback-only\r
5. **Obtain OBS service name from service ticket**: Never guess the `obs_endpoint_service_name` from a similar regional public service\r
6. **Verify after each change**: Run `huawei_verify_cce_cci_bursting` after each applied change to confirm progress\r
7. **Setup is idempotent**: `huawei_setup_cce_cci_bursting` updates existing addon configuration without uninstalling; safe to re-run\r
8. **Reuse existing VPCEPs**: The tool reuses accepted VPCEPs in the cluster VPC; no duplicate creation\r
9. **Do not delete resources automatically**: Never auto-delete VPCEPs, namespaces, workloads, addons, or addon ReplicaSets\r
10. **Diagnose addon failures**: When verification fails, run `huawei_diagnose_cce_cci_bursting_addon` for read-only diagnostics; never patch the internal `bursting-status` ConfigMap automatically\r
11. **Node headroom is a warning**: The 2C/4GiB NodeCheck is a conservative small-cluster warning, not a platform hard limit; size production resources with the official addon formula\r
12. **Explicit project ID when needed**: Pass `project_id` explicitly when IAM auto-resolution is unavailable\r
\r
## Reference Documents\r
\r
| Document | Description |\r
| --- | --- |\r
| [Workflow](references/workflow.md) | Action sequence, subnet roles, and command examples |\r
| [Risk Rules](references/risk-rules.md) | Preview-first constraints, billing scope, and safe defaults |\r
| [Troubleshooting](references/troubleshooting.md) | Symptom-cause-action table for common bursting failures |\r
\r
## Notes\r
\r
- **Preview-first by design** — VPCEP creation, addon installation, node pool changes, and workload deployment return a preview without `confirm=true`; apply only after explicit user approval\r
- **Idempotent setup** — `huawei_setup_cce_cci_bursting` may update the existing `virtual-kubelet` addon configuration but never uninstalls it; it resolves and writes the regional project ID\r
- **Turbo/ENI required** — CCE to CCI bursting requires a Turbo cluster with ENI container network mode\r
- **No credential persistence** — AK/SK exists only during API calls; never written to disk, logs, or reports\r
- **Cross-skill escalation** — If bursting pods show CrashLoopBackOff or ImagePullBackOff, hand off to `huawei-cloud-cce-pod-failure-diagnoser`; if VPCEP connectivity fails, hand off to `huawei-cloud-cce-network-failure-diagnoser`\r
- **Tenant-owned smoke images** — Prefer a tenant-owned SWR image discovered via `huawei_discover_cce_cci_smoke_images`; public namespace images are fallback-only because CCI pulling through\r
  VPCEP may fail\r
- **NodeCheck warning baseline** — The 2C/4GiB value is a conservative small-cluster warning; size production addon resources with the official formula\r
\r
## Common Pitfalls\r
\r
| Pitfall | Symptom | Quick Fix |\r
| --- | --- | --- |\r
| Swapped subnet IDs | virtual-kubelet restarts, virtual node never Ready | Run precheck; use `cci_subnet_id` from `spec.eni_network` (Neutron UUID), not the VPC subnet UUID |\r
| Missing SWR VPCEPs | CCI pod `ImagePullBackOff` or image pull timeout | Run `huawei_ensure_cce_cci_vpcep` to create SWR endpoints |\r
| Public namespace smoke image | CCI pod image pull timeout through VPCEP | Use `huawei_discover_cce_cci_smoke_images` to find a tenant-owned SWR image |\r
| Missing bursting label | Workload pods stay Pending on real nodes | Add `bursting.cci.io/burst-to-cci: enforce` label to pod template |\r
| Guessed OBS service name | OBS gateway VPCEP creation fails | Obtain exact `obs_endpoint_service_name` from Huawei Cloud service ticket |\r
  | Insufficient physical-node headroom | `bursting-cceaddon-*` Pods stay Pending | Run `huawei_check_cce_cci_node_capacity`; preview node pool resize or create |\r
  | Addon region mismatch | Logs report `northchina` or `southchina` | Run `huawei_diagnose_cce_cci_bursting_addon`; do not hard-code region mapping |\r
  | Missing project ID | Addon reports IAM denied or project ID missing | Pass `project_id` explicitly when auto-resolution unavailable |\r
  | Multiple active ReplicaSets | `bursting-cceaddon-*` has several ReplicaSets | Inspect Deployment rollout state; do not auto-delete ReplicaSets |\r
  | Non-ENI cluster | Precheck reports CCI bursting not supported | Use a Turbo/ENI cluster; overlay_l2 clusters cannot burst to CCI |\r
| Non-ENI cluster | Precheck reports CCI bursting not supported | Use a Turbo/ENI cluster; overlay_l2 clusters cannot burst to CCI |\r
| Insufficient physical-node headroom | `bursting-cceaddon-*` Pods stay Pending | Run `huawei_check_cce_cci_node_capacity`; preview node pool resize or create |\r
| Addon region mismatch | Logs report `northchina` or `southchina` | Run `huawei_diagnose_cce_cci_bursting_addon`; do not hard-code region mapping |\r
| Missing project ID | Addon reports IAM denied or project ID missing | Pass `project_id` explicitly when IAM auto-resolution unavailable |\r
| Multiple active ReplicaSets | `bursting-cceaddon-*` has several ReplicaSets | Inspect Deployment rollout state; do not auto-delete ReplicaSets |