Long Document Processing Prompt Best Practices: Chunking Strategies, Recursive Summarization and RAG Retrieval Augmentation

Chapter 76: Multilingual and Localization: Making Claude Perfectly Serve Global Users

76.1 The Challenges of Multilingual AI Service

Expanding an AI service from a single language to global multilingual support is far more than "making the model answer in the target language." Genuine multilingual service involves four layers of challenge:

1. Language detection — Accurately identifying the language and dialect a user is employing
2. Code-switching handling — Managing users who mix multiple languages within a single message
3. Cultural adaptation — Different cultures have different communication norms, etiquette, and taboos
4. Consistency management — Ensuring outputs in different languages maintain equivalent quality and style

Claude has strong multilingual capabilities, supporting high-quality generation in 50+ languages. But a gap remains between "model capability" and "production-grade multilingual experience." This chapter focuses on engineering practices that bridge that gap.

76.2 Language Detection

76.2.1 Claude-Based Language Detection

For scenarios requiring 100% language coverage with maximum accuracy, Claude itself can serve as the detector:

from anthropic import Anthropic

client = Anthropic()

LANGUAGE_DETECTOR_PROMPT = """You are a language detection expert. For any given text, identify:
1. The primary language (ISO 639-1 code, e.g., zh, en, ja, ko, es, fr, de, ar)
2. If multiple languages are present, list all detected languages with proportion estimates
3. The language variant (e.g., Simplified/Traditional Chinese, regional Spanish)
4. Confidence level (high/medium/low)

Output as JSON:
{
  "primary_language": "zh",
  "primary_language_name": "Chinese",
  "variant": "simplified",
  "all_languages": [{"code": "zh", "proportion": 0.9}, {"code": "en", "proportion": 0.1}],
  "is_code_switching": false,
  "confidence": "high"
}"""

def detect_language(text: str) -> dict:
    if not text or len(text.strip()) < 3:
        return {"primary_language": "unknown", "confidence": "low"}
    
    response = client.messages.create(
        model="claude-haiku-4-5",  # lightweight model to reduce cost
        max_tokens=200,
        system=LANGUAGE_DETECTOR_PROMPT,
        messages=[{"role": "user", "content": f"Text: {text[:500]}"}]
    )
    
    import json
    try:
        return json.loads(response.content[0].text)
    except json.JSONDecodeError:
        return {"primary_language": "unknown", "confidence": "low"}

76.2.2 A Hybrid Detection Strategy

In production, a layered strategy balances speed and accuracy:

import langdetect

class LanguageDetector:
    """
    Multi-layer language detector:
    - Layer 1: Unicode character range rules (instant, free)
    - Layer 2: langdetect library (fast, free)
    - Layer 3: Claude detection (accurate, costs tokens)
    """
    
    SCRIPT_RANGES = {
        "zh": [(0x4E00, 0x9FFF), (0x3400, 0x4DBF)],
        "ja": [(0x3040, 0x309F), (0x30A0, 0x30FF)],
        "ko": [(0xAC00, 0xD7AF)],
        "ar": [(0x0600, 0x06FF)],
        "th": [(0x0E00, 0x0E7F)],
    }
    
    def detect(self, text: str, use_llm_for_ambiguous: bool = True) -> dict:
        # Layer 1: script-based detection
        script_result = self._detect_by_script(text)
        if script_result and script_result["confidence"] == "high":
            return script_result
        
        # Layer 2: langdetect
        try:
            lang = langdetect.detect(text)
            probs = langdetect.detect_langs(text)
            top_prob = max(probs, key=lambda x: x.prob)
            if top_prob.prob > 0.9:
                return {"primary_language": lang, "confidence": "high", "method": "langdetect"}
        except Exception:
            pass
        
        # Layer 3: Claude (only for ambiguous cases)
        if use_llm_for_ambiguous:
            return detect_language(text)
        
        return {"primary_language": "en", "confidence": "low", "method": "fallback"}
    
    def _detect_by_script(self, text: str) -> dict:
        char_counts = {}
        for char in text:
            code = ord(char)
            for lang, ranges in self.SCRIPT_RANGES.items():
                for start, end in ranges:
                    if start <= code <= end:
                        char_counts[lang] = char_counts.get(lang, 0) + 1
        
        if char_counts:
            total_script = sum(char_counts.values())
            total_chars = len([c for c in text if c.strip()])
            if total_script / max(total_chars, 1) > 0.3:
                dominant = max(char_counts, key=char_counts.get)
                return {"primary_language": dominant, "confidence": "high", "method": "script"}
        return None

76.3 Code-Switching Handling

Code-switching — using multiple languages within the same message — is extremely common in bilingual and multilingual communities.

76.3.1 Common Code-Switching Patterns

CODESWITCHING_EXAMPLES = {
    "technical_embedding": "I'm working on a machine learning project using Python",
    "emotional_emphasis": "这个 bug 真的太 tricky 了，完全 debug 不出来",  # Chinese-English mix
    "singlish": "Today weather not good lah, we cancel that meeting can or not?",
    "taiwanese_mandarin": "這個 feature 我 implement 完了，你 check 一下有沒有問題"
}

76.3.2 Determining the Response Language

class CodeSwitchingHandler:
    
    def determine_response_language(
        self,
        user_message: str,
        detection_result: dict,
        user_preference: str = None,
        platform_default: str = "en"
    ) -> str:
        """
        Priority order:
        1. Explicitly stored user language preference
        2. Primary non-English language in the message (>60% proportion)
        3. Platform default
        """
        if user_preference:
            return user_preference
        
        if detection_result.get("is_code_switching"):
            # In code-switching, technical terms are often English,
            # but the user's actual language is their native tongue
            all_langs = detection_result.get("all_languages", [])
            non_english = [l for l in all_langs if l["code"] != "en"]
            if non_english:
                return max(non_english, key=lambda x: x["proportion"])["code"]
        
        primary = detection_result.get("primary_language", platform_default)
        confidence = detection_result.get("confidence", "low")
        
        if confidence in ["high", "medium"]:
            return primary
        
        return platform_default

76.3.3 Multilingual System Prompt Construction

def build_multilingual_system_prompt(
    base_instructions: str,
    response_language_policy: str = "match_user"
) -> str:
    language_rules = {
        "match_user": """Language policy:
- Automatically detect the user's primary language
- Respond in the same language the user is using
- In code-switching scenarios, respond in the user's native language (the non-English portion)
- Technical terms may remain in English; explanatory text should use the user's language""",
        
        "always_en": """Language policy:
- Always respond in English regardless of the user's input language""",
        
        "zh_then_en": """Language policy:
- If Chinese is detected as the primary language, respond in Chinese
- For all other languages, respond in English"""
    }
    
    lang_rule = language_rules.get(response_language_policy, language_rules["match_user"])
    return f"""{base_instructions}\n\n{lang_rule}"""

76.4 Cultural Adaptation

Cultural adaptation is the deepest and most frequently overlooked dimension of multilingual service. It is not translation — it is adjusting the way content is expressed to align with the target culture's cognitive patterns and social norms.

76.4.1 Core Cultural Dimensions

76.4.2 Culture-Aware Prompt Templates

CULTURE_AWARE_TEMPLATES = {
    "zh_cn": {
        "customer_service": """You are a professional Chinese customer service representative.

Communication style:
- Use polite but not overly formal tone
- Address users as "您" (formal you)
- Provide complete responses; overly brief replies feel dismissive
- If the issue cannot be immediately resolved, give a concrete time commitment
- Use appropriate courtesy phrases ("感谢您的联系", "给您带来不便深感抱歉")
- Avoid any politically sensitive vocabulary

Format:
- Keep paragraphs short (3-4 lines)
- Use numbered lists for multiple steps
- End with an invitation for further questions"""
    },
    
    "ja": {
        "customer_service": """あなたはプロフェッショナルな日本語カスタマーサービス担当者です。

コミュニケーションスタイル：
- 丁寧語を使用し、相手を「お客様」と呼ぶ
- 問題解決の前に、まずお詫びと共感を示す
- 直接的な「いいえ」を避け、代替案を提示する
- 文章の末尾に「〜でございます」「〜いたします」を使用
- 不確かな情報は提供せず、確認後に回答する旨を伝える"""
    },
    
    "ar": {
        "customer_service": """أنت ممثل خدمة عملاء محترف يتحدث اللغة العربية.

أسلوب التواصل:
- ابدأ بالتحية المناسبة للسياق
- استخدم أسلوباً محترماً وودياً في آنٍ معاً
- اعترف بالمشكلة قبل الانتقال إلى الحلول
- تجنب الإلحاح أو الضغط المباشر
- أشر إلى التزامك بمساعدة العميل"""
    }
}

76.4.3 Automated Cultural Appropriateness Check

def check_cultural_appropriateness(content: str, target_culture: str) -> dict:
    cultural_check_prompt = f"""Evaluate the cultural appropriateness of the following content 
for a {target_culture} audience.

Content:
{content}

Check the following dimensions:
1. Are there culturally insensitive expressions or phrases likely to cause misunderstanding?
2. Is the communication style (direct/indirect, formal/informal) appropriate for this culture?
3. Are there metaphors, idioms, or cultural references that need localization?
4. Does the content touch on sensitive topics for this culture?

Output as JSON:
{{
  "is_appropriate": true/false,
  "issues": [
    {{"type": "formality", "description": "...", "suggestion": "..."}}
  ],
  "overall_assessment": "...",
  "localization_needed": true/false
}}"""
    
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1000,
        messages=[{"role": "user", "content": cultural_check_prompt}]
    )
    
    import json
    try:
        return json.loads(response.content[0].text)
    except:
        return {"raw_assessment": response.content[0].text}

76.5 Multilingual Consistency Management

76.5.1 Cross-Language Quality Comparison

def evaluate_multilingual_consistency(source_text, source_lang, translations) -> dict:
    """
    translations: {"zh": "...", "ja": "...", "fr": "..."}
    """
    consistency_prompt = f"""Evaluate the consistency and quality of these translations 
against the source text.

Source ({source_lang}):
{source_text}

Translations:
{chr(10).join([f"{lang}: {text}" for lang, text in translations.items()])}

For each translation, evaluate:
1. Semantic accuracy (is the core meaning preserved?)
2. Tone consistency (does it maintain the same formal/informal register?)
3. Cultural appropriateness (is it suitable for the target audience?)
4. Fluency (does it read naturally, not like machine translation?)

Output scores (1-10) and specific issues per language as JSON."""
    
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=2000,
        messages=[{"role": "user", "content": consistency_prompt}]
    )
    return {"assessment": response.content[0].text}

76.5.2 Multilingual Prompt Version Management

class MultilingualPromptManager:
    
    def __init__(self):
        self.prompts = {}  # {task_name: {lang_code: prompt_text}}
        self.base_language = "en"
    
    def set_base_prompt(self, task_name: str, prompt: str):
        if task_name not in self.prompts:
            self.prompts[task_name] = {}
        self.prompts[task_name][self.base_language] = prompt
    
    def generate_localized_version(self, task_name: str, target_lang: str) -> str:
        base_prompt = self.prompts.get(task_name, {}).get(self.base_language)
        if not base_prompt:
            raise ValueError(f"Base prompt not set for task: {task_name}")
        
        localization_request = f"""Localize the following system prompt for a {target_lang} audience.

This is NOT just translation — you must:
1. Adjust formality level to match {target_lang} cultural norms
2. Adapt examples and references to be meaningful for {target_lang} users
3. Preserve all core functional instructions
4. Ensure output format requirements remain clear after localization

Original English prompt:

{base_prompt}

Output the {target_lang} localized version directly:"""
        
        response = client.messages.create(
            model="claude-opus-4-5",
            max_tokens=2000,
            messages=[{"role": "user", "content": localization_request}]
        )
        
        localized = response.content[0].text
        self.prompts[task_name][target_lang] = localized
        return localized
    
    def get_prompt(self, task_name: str, lang: str) -> str:
        task_prompts = self.prompts.get(task_name, {})
        return task_prompts.get(lang, task_prompts.get(self.base_language, ""))

76.6 Production Architecture for Multilingual Deployment

76.6.1 Language Routing Layer

class MultilingualRouter:
    
    def __init__(self, prompt_manager: MultilingualPromptManager):
        self.detector = LanguageDetector()
        self.prompt_manager = prompt_manager
    
    def route(self, user_message: str, task_name: str, user_locale: str = None) -> dict:
        if user_locale:
            lang_code = user_locale.split("_")[0]
            detection = {"primary_language": lang_code, "confidence": "high"}
        else:
            detection = self.detector.detect(user_message)
            lang_code = detection.get("primary_language", "en")
        
        prompt = self.prompt_manager.get_prompt(task_name, lang_code)
        
        return {
            "detected_language": lang_code,
            "detection_confidence": detection.get("confidence"),
            "system_prompt": prompt,
            "user_locale": user_locale or lang_code
        }

76.6.2 Multilingual Test Matrix

A production multilingual system needs a test matrix covering all supported languages:

MULTILINGUAL_TEST_MATRIX = [
    {
        "test_id": "ml_001",
        "scenario": "language_switching",
        "inputs": {
            "zh": "如何重置我的密码？",
            "en": "How do I reset my password?",
            "ja": "パスワードをリセットするにはどうすればいいですか？",
            "ko": "비밀번호를 어떻게 재설정하나요?",
            "es": "¿Cómo puedo restablecer mi contraseña?"
        },
        "expected_behavior": "Respond in corresponding language with equivalent core information",
        "checks": {
            "zh": {"contains": ["密码", "重置"]},
            "en": {"contains": ["password", "reset"]},
            "ja": {"contains": ["パスワード", "リセット"]}
        }
    }
]

Summary

Multilingual localization is the critical capability for taking an AI product to global markets. On the technical side, three core engineering modules are required: language detection (a hybrid layered strategy), code-switching handling (identifying the user's native language and responding in it), and cultural adaptation (adjusting tone, etiquette, and content). On the management side, multilingual prompt version management and consistency evaluation ensure users across all languages receive equivalent quality experiences.

Cultural adaptation is the hardest dimension to quantify but the most important for user experience. An AI assistant that performs excellently in English may alienate users in Japan or the Arab world due to inappropriate formality or communication style, even with perfect translations. The tools and framework in this chapter provide the engineering foundation for building truly global AI services.