🎤 Digital Singer — NuwaAI Avatar Singing

Turn your NuwaAI digital avatar into a singing performer with lip-synced mouth movements.

How It Works

Audio files → FFmpeg PCM conversion → NuwaAI humanctrl avatar messages → Avatar sings with lip-sync
                                    + Browser \x3Caudio> plays accompaniment in sync

Key concept: Two audio streams work together:

• speech (vocal-only): Drives the avatar's mouth movements via NuwaAI avatar control messages
• music (accompaniment): Plays simultaneously via browser \x3Caudio> element

Battle Flow

1. Avatar greets user, lists available songs
2. User picks a song
3. Avatar sings the upper half (vocal drives lip-sync + accompaniment plays in sync)
4. Avatar says "your turn" → accompaniment for lower half plays → user sings along (ASR captures voice)
5. Battle scoring + blind box reward
6. Ask to continue

User Preparation (What You Need Before Using)

1. NuwaAI Account (Required)

Sign up at nuwaai.com (free) and create a digital avatar. You need:

• API Key — from your NuwaAI dashboard
• Avatar ID — the avatar you want to sing
• User ID — your NuwaAI user ID

Enter these in the browser interface, or pre-configure in .nuwa-config.json:

{
  "avatarId": "your-avatar-id",
  "apiKey": "your-api-key",
  "userId": "your-user-id"
}

2. LLM API (Required)

The singing host (conversation agent) needs an OpenAI-compatible LLM API. Configure in server.mjs or via environment variables:

• DASHSCOPE_BASE_URL — API endpoint (default: Dashscope)
• DASHSCOPE_API_KEY — API key
• QWEN_MODEL — Model name (default: qwen-plus)

3. Song Audio Files (Required)

Each song needs 3 audio files placed in the skill directory:

How to prepare these files:

• Use any audio editing tool (e.g. Audacity, Adobe Audition) to split songs into upper/lower halves
• Use vocal separation tools (e.g. UVR, Demucs) to extract a cappella (vocal-only) from the upper half
• Export accompaniment as MP3, vocal as WAV (any FFmpeg-supported format works)
• Place all files in the skill directory (same folder as server.mjs)

4. Song Registry (Required)

After preparing audio files, register each song in server.mjs SONGS object:

const SONGS = {
  "十年": {
    artist: "陈奕迅",
    acappella_upper: "十年高潮上清唱.wav",
    accomp_upper: "十年高潮上伴奏.MP3",
    accomp_lower: "十年高潮下伴奏.MP3",
  },
  // Add more songs...
};

5. FFmpeg (Required)

Install FFmpeg for audio format conversion:

# macOS
brew install ffmpeg
# Ubuntu/Debian
sudo apt install ffmpeg

6. Node.js 18+ (Required)

node --version  # must be >= 18

Quick Start

1. Complete all preparation steps above
2. Copy example song files from assets/songs/ to the skill directory:

   cp \x3Cskill-dir>/assets/songs/* \x3Cskill-dir>/
   

   This includes a ready-to-use demo song: 十年 (陈奕迅) with vocal, upper and lower accompaniment files.
3. Start the server:

   node \x3Cskill-dir>/server.mjs
   

4. Open http://localhost:3098 in browser
5. Enter NuwaAI credentials (if not pre-configured)
6. Pick a song and start singing!

Included Example Song

The skill ships with one demo song in assets/songs/:

• 十年高潮上清唱.wav — vocal (a cappella)
• 十年高潮上伴奏.MP3 — upper half accompaniment
• 十年高潮下伴奏.MP3 — lower half accompaniment

Copy them to the skill root directory to use. The song "十年" is pre-registered in server.mjs.

NuwaAI Integration

Uses humanctrl WebSocket with ASR enabled. Avatar control message format:

{
  "type": "avatar",
  "data": {
    "content": "",
    "audio": {
      "segment": 0,
      "speech": "\x3Cbase64 PCM>",
      "music": "\x3Cbase64 PCM>"
    }
  }
}

• speech: Vocal-only PCM driving lip-sync (≤10KB per chunk)
• music: Same as speech for a cappella mode
• segment: Chunk index, -1 = last chunk
• Audio: 16kHz mono PCM, base64 encoded

Features

• 🎤 Avatar lip-sync singing via NuwaAI humanctrl
• 🎵 Synchronized accompaniment playback
• 🗣️ ASR voice capture for user singing
• 🎯 Fun battle scoring system
• 🎁 Blind box rewards
• ⏸️ Interruptible speech (not during singing)
• 📱 Responsive web interface

Requirements

• Node.js 18+
• FFmpeg (for audio → PCM conversion)
• NuwaAI account with avatar
• Modern browser (WebRTC + microphone)