Huawei Cloud CCE Alarm Correlation Engine

⚠️ Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is located at scripts/huawei-cloud.py within the skill directory
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview

This skill correlates Huawei Cloud AOM active and historical alarms for CCE, transforming raw alarm event streams into actionable alarm leads. The core principle is to consider both active and history alarms, avoiding omission of resource-type alarms that have already recovered but still impact diagnosis.

This skill has both read-only tools (alarm query, analysis, inspection, rule query) and mutation tools (alarm rule create/update/delete, action rule delete). All mutation operations require a two-step confirmation workflow with confirm=true.

Related Skills

Capabilities

1. Query active + history alarms merged and deduplicated (huawei_list_aom_alarms)
2. Query current active alarms only (huawei_list_aom_current_alarms)
3. Analyze alarms: deduplication, severity grouping, burst/steady identification (huawei_analyze_aom_alarms)
4. Query, create, update, delete, enable, disable AOM alarm rules (mutation requires confirm=true)
5. Create AOM event alarm rules referencing CCE event list (huawei_create_aom_event_alarm_rule)
6. Query and delete AOM action/notification rules (delete requires confirm=true)
7. Query AOM mute rules (huawei_list_aom_mute_rules)
8. CCE cluster alarm inspection with risk summary (huawei_aom_alarm_inspection)

Typical Use Cases

• Query all alarms (active + history) for a CCE cluster and group by severity
• Reduce alarm storm noise by deduplicating and classifying burst vs. steady alarms
• Inspect a CCE cluster for alarm health risks
• Create, update, or delete AOM alarm rules with preview + confirmation workflow
• Check whether alarms are suppressed by action rules or mute rules causing notification gaps
• Create event alarm rules for CCE workload, node, network, storage, or autoscaling events

Prerequisites

Python Dependencies

The dispatcher script requires Python >= 3.6 and the following packages:

• huaweicloudsdkcore
• huaweicloudsdkaom
• huaweicloudsdkiam

Credential Configuration

🚫 Never expose or log AK/SK values. Credentials exist only in the current request call stack and are released after each invocation. Do not write credentials to files, logs, or responses.

✅ Use environment variables HUAWEI_AK / HUAWEI_SK for authentication. The dispatcher reads them automatically.

Security rules for credentials:

1. No persistent storage — never write AK/SK, tokens, or certificates to disk files
2. No long-term memory cache — AK/SK exists only during the current API call and is released afterward
3. Project ID memory cache only — only non-sensitive project IDs may be cached in process memory (never written to disk)
4. No log leakage — never include AK/SK in logs, response output, or error messages
5. Output desensitization — output only alarm, resource, and rule information; never expose authentication credentials

AK/SK may be provided in two ways:

• Via environment variables HUAWEI_AK / HUAWEI_SK (recommended)
• Via per-call parameters ak and sk (not recommended for production)

IAM Permissions

Core Tools

All actions are invoked via the dispatcher script:

python3 scripts/huawei-cloud.py \x3Caction> region=\x3Cregion> [key=value ...]

Two-Step Confirmation Workflow for Mutation Operations

Mutation operations (create, update, delete, enable, disable alarm rules; delete action rules) require confirm=true to execute. Without confirm, the tool returns a preview and confirmation prompt only.

Step 1: Preview — call without confirm:

python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule metric_name=cpuUsage \
  namespace=PAAS.NODE comparison_operator='>' threshold=80 \
  period=60 evaluation_periods=3 statistic=average alarm_level=2

Returns: operation preview, target rule, rule fields, and confirmation example. No real creation is performed.

Step 2: Confirm execution — call again with confirm=true:

python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule metric_name=cpuUsage \
  namespace=PAAS.NODE comparison_operator='>' threshold=80 \
  period=60 evaluation_periods=3 statistic=average alarm_level=2 \
  confirm=true

Operations Requiring Confirmation

Prohibited Actions

If analysis results require scaling, rebooting, draining, vulnerability status changes, or other remediation actions, output recommendations only and hand off to huawei-cloud-cce-auto-remediation-runner for preview, confirmation, and post-verification.

Alarm Query and Correlation (Read-Only)

# Query active + history alarms in a region
python3 scripts/huawei-cloud.py huawei_list_aom_alarms region=cn-north-4

# Query alarms for a specific cluster
python3 scripts/huawei-cloud.py huawei_list_aom_alarms \
  region=cn-north-4 cluster_id=xxx

# Query current active alarms for a cluster
python3 scripts/huawei-cloud.py huawei_list_aom_current_alarms \
  region=cn-north-4 cluster_id=xxx

# Analyze alarms for a cluster (burst, attention, steady groups)
python3 scripts/huawei-cloud.py huawei_analyze_aom_alarms \
  region=cn-north-4 cluster_id=xxx

Alarm Rule Management (Mutation Requires confirm=true)

Event alarm rule constraints:

• When creating event alarm rules via huawei_create_aom_event_alarm_rule, the event_name must reference the event list and naming format in references/cce-event-list.md (recommended format: Chinese event description##Event name).
• Event alarm rules are created with alarm noise reduction enabled by default (route_group_enable=true).

Metric alarm rule constraints:

• When creating metric alarm rules via huawei_create_aom_alarm_rule, PromQL/metric thresholds should reference references/cce-prometheus-metric-alarms.md.

# Query alarm rules
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules region=cn-north-4

# Preview create alarm rule (no execution)
python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule metric_name=cpuUsage \
  namespace=PAAS.NODE comparison_operator='>' threshold=80 \
  period=60 evaluation_periods=3 statistic=average alarm_level=2

# Confirm create alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule metric_name=cpuUsage \
  namespace=PAAS.NODE comparison_operator='>' threshold=80 \
  period=60 evaluation_periods=3 statistic=average alarm_level=2 \
  confirm=true

# Preview update alarm rule
python3 scripts/huawei-cloud.py huawei_update_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule threshold=80

# Confirm update alarm rule
python3 scripts/huawei-cloud.py huawei_update_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule threshold=80 confirm=true

# Preview delete alarm rule
python3 scripts/huawei-cloud.py huawei_delete_aom_alarm_rule \
  region=cn-north-4 rule_name=xxx

# Confirm delete alarm rule
python3 scripts/huawei-cloud.py huawei_delete_aom_alarm_rule \
  region=cn-north-4 rule_name=xxx confirm=true

# Preview disable alarm rule
python3 scripts/huawei-cloud.py huawei_disable_aom_alarm_rule \
  region=cn-north-4 rule_id=xxx

# Confirm disable alarm rule
python3 scripts/huawei-cloud.py huawei_disable_aom_alarm_rule \
  region=cn-north-4 rule_id=xxx confirm=true

# Preview enable alarm rule
python3 scripts/huawei-cloud.py huawei_enable_aom_alarm_rule \
  region=cn-north-4 rule_id=xxx

# Confirm enable alarm rule
python3 scripts/huawei-cloud.py huawei_enable_aom_alarm_rule \
  region=cn-north-4 rule_id=xxx confirm=true

# Query action rules
python3 scripts/huawei-cloud.py huawei_list_aom_action_rules region=cn-north-4

# Preview delete action rule
python3 scripts/huawei-cloud.py huawei_delete_aom_action_rule \
  region=cn-north-4 rule_name=xxx

# Confirm delete action rule
python3 scripts/huawei-cloud.py huawei_delete_aom_action_rule \
  region=cn-north-4 rule_name=xxx confirm=true

# Query mute rules
python3 scripts/huawei-cloud.py huawei_list_aom_mute_rules region=cn-north-4

Cluster Alarm Inspection (Read-Only)

# Inspect alarms for a specific cluster
python3 scripts/huawei-cloud.py huawei_aom_alarm_inspection \
  region=cn-north-4 cluster_id=xxx

Parameter Reference

Alarm Query Parameters

Alarm Rule Mutation Parameters

Output Format

Alarm Summary

Output must include:

Resource Clues

For CCE alarms, prioritize the following resource dimensions:

Recommended Diagnosis Skills

See references/output-schema.md for the full JSON response schema.

Verification

1. Run the dispatcher with a known region and cluster to confirm connectivity:

   python3 scripts/huawei-cloud.py huawei_list_aom_alarms region=cn-north-4 cluster_id=\x3Ccluster_id>
   

2. Execute huawei_analyze_aom_alarms and verify that burst, attention, and steady groupings are returned correctly
3. Verify huawei_aom_alarm_inspection returns cluster alarm summary with risk items
4. Test mutation preview workflow: call huawei_create_aom_alarm_rule without confirm and verify it returns a preview only (no actual creation)
5. After a mutation operation with confirm=true, call huawei_list_aom_alarm_rules to verify the rule state change

Best Practices

1. Always query both active and history alarms via huawei_list_aom_alarms; never assume absence of active alarms means no problems — check history alarms
2. Use huawei_analyze_aom_alarms for alarm storm scenarios to deduplicate and identify burst vs. steady alarms
3. For mutation operations, always follow the two-step workflow: preview first (without confirm), then confirm (with confirm=true) only after explicit user approval
4. After creating event alarm rules, verify that event_name follows the format in references/cce-event-list.md
5. When checking notification gaps, query both action rules (huawei_list_aom_action_rules) and mute rules (huawei_list_aom_mute_rules)
6. Do not interpret absence of active alarms as "no problem"; always cross-reference history alarms
7. When creating metric alarm rules, reference references/cce-prometheus-metric-alarms.md for recommended PromQL expressions and thresholds

Reference Documents

Notes

1. This skill has both read-only and mutation tools — mutation operations (create, update, delete, enable, disable alarm rules; delete action rules) require confirm=true two-step confirmation
2. Never create, update, or delete action rules or mute rules — only query and delete action rules with confirmation
3. If remediation actions are needed (scale, reboot, drain), output recommendations only and hand off to huawei-cloud-cce-auto-remediation-runner
4. Never expose or log AK/SK or environment variable values
5. All actions are executed via python3 scripts/huawei-cloud.py \x3Caction>; do not use hcloud CLI or direct API calls
6. Do not interpret absence of active alarms as "no problem" — always verify with history alarms

Common Pitfalls