功能描述

Monitor bioRxiv/medRxiv preprints and academic discussions to identify emerging research hotspots before they appear in mainstream journals

使用说明 (SKILL.md)

Emerging Topic Scout

Name: Emerging Topic Scout
Author: aipoch-ai

A real-time monitoring system for identifying "incubation period" research hotspots in biological and medical sciences before they are defined by mainstream journals.

Overview

This skill continuously monitors:

bioRxiv: Biology preprints via RSS/API ⚠️ Currently blocked by Cloudflare
medRxiv: Medicine preprints via RSS/API ⚠️ Currently blocked by Cloudflare
arXiv: Quantitative Biology preprints via RSS ✅ Recommended alternative
Academic discussions: Social media and forum mentions

It uses trend analysis algorithms to detect sudden spikes in topic frequency, cross-platform mentions, and emerging keyword clusters.

⚠️ Network Access Notice

bioRxiv and medRxiv are currently protected by Cloudflare JavaScript Challenge, which prevents programmatic RSS access. As a workaround, this skill now supports arXiv q-bio (Quantitative Biology) as an alternative data source.

Recommended usage:

# Use arXiv for reliable data fetching
python scripts/main.py --sources arxiv --days 30

# bioRxiv/medRxiv may return 0 results due to Cloudflare protection
python scripts/main.py --sources biorxiv medrxiv --days 30  # May not work

Installation

cd /Users/z04030865/.openclaw/workspace/skills/emerging-topic-scout
pip install -r scripts/requirements.txt

Usage

Basic Scan (Recommended: Use arXiv)

python scripts/main.py --sources arxiv --days 7 --output json

Legacy bioRxiv/medRxiv (May not work due to Cloudflare)

python scripts/main.py --sources biorxiv medrxiv --days 7 --output json

Advanced Configuration (arXiv Recommended)

python scripts/main.py \
  --sources arxiv \
  --keywords "CRISPR,gene editing,machine learning" \
  --days 14 \
  --min-score 0.7 \
  --output markdown \
  --notify

Legacy Configuration (bioRxiv/medRxiv - May not work)

python scripts/main.py \
  --sources biorxiv medrxiv \
  --keywords "CRISPR,gene editing,long COVID" \
  --days 14 \
  --min-score 0.7 \
  --output markdown \
  --notify
# Note: bioRxiv/medRxiv may return 0 results due to Cloudflare protection

## Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--sources` | list | `arxiv` | Data sources to monitor (arxiv recommended due to Cloudflare issues with biorxiv/medrxiv) |
| `--keywords` | string | (auto-detect) | Comma-separated keywords to track |
| `--days` | int | `7` | Lookback period in days |
| `--min-score` | float | `0.6` | Minimum trending score (0-1) |
| `--max-topics` | int | `20` | Maximum topics to return |
| `--output` | string | `markdown` | Output format: `json`, `markdown`, `csv` |
| `--notify` | flag | `false` | Send notification for high-priority topics |
| `--config` | path | `config.yaml` | Path to configuration file |

## Output Format

### JSON Output

```json
{
  "scan_date": "2026-02-06T05:57:00Z",
  "sources": ["biorxiv", "medrxiv"],
  "hot_topics": [
    {
      "topic": "gene editing therapy",
      "keywords": ["CRISPR", "base editing", "prime editing"],
      "trending_score": 0.89,
      "velocity": "rapid",
      "preprint_count": 34,
      "cross_platform_mentions": 127,
      "related_papers": [
        {
          "title": "New CRISPR variant shows promise",
          "authors": ["Smith J.", "Lee K."],
          "doi": "10.1101/2026.01.15.xxxxx",
          "source": "biorxiv",
          "published": "2026-01-15",
          "abstract_summary": "..."
        }
      ],
      "emerging_since": "2026-01-20"
    }
  ],
  "summary": {
    "total_papers_analyzed": 1247,
    "new_topics_detected": 8,
    "high_priority_alerts": 2
  }
}

Markdown Output

# Emerging Topics Report - 2026-02-06

## 🔥 High Priority Topics

### 1. Gene Editing Therapy (Score: 0.89)
- **Keywords**: CRISPR, base editing, prime editing
- **Growth Rate**: Rapid (+145% vs last week)
- **Preprints**: 34 papers
- **Cross-platform mentions**: 127

#### Key Papers
1. "New CRISPR variant shows promise" - Smith J. et al.
   - DOI: 10.1101/2026.01.15.xxxxx
   - Source: bioRxiv

Configuration File

Create config.yaml for persistent settings:

sources:
  arxiv:
    enabled: true
    rss_url: "https://export.arxiv.org/rss/q-bio"
    description: "arXiv Quantitative Biology - Recommended (no Cloudflare)"
  biorxiv:
    enabled: false  # Disabled due to Cloudflare protection
    rss_url: "https://www.biorxiv.org/rss/recent.rss"
    api_endpoint: "https://api.biorxiv.org/details/"
    note: "Currently blocked by Cloudflare JavaScript Challenge"
  medrxiv:
    enabled: false  # Disabled due to Cloudflare protection
    rss_url: "https://www.medrxiv.org/rss/recent.rss"
    api_endpoint: "https://api.medrxiv.org/details/"
    note: "Currently blocked by Cloudflare JavaScript Challenge"

trending:
  min_papers_threshold: 5
  velocity_window_days: 3
  novelty_weight: 0.4
  momentum_weight: 0.6

keywords:
  auto_detect: true
  custom_trackers:
    - "artificial intelligence"
    - "machine learning"
    - "single cell"
    - "spatial transcriptomics"

output:
  default_format: markdown
  save_history: true
  history_path: "./data/history.json"

notifications:
  enabled: false
  high_score_threshold: 0.8

Trending Score Algorithm

The trending score (0-1) is calculated using:

Score = (Novelty × 0.4) + (Momentum × 0.4) + (CrossRef × 0.2)

Where:
- Novelty: Inverse frequency of topic in historical data
- Momentum: Rate of increase in mentions over velocity window
- CrossRef: Mentions across multiple platforms

API Endpoints

bioRxiv API

Base: https://api.biorxiv.org/
Details: /details/[server]/[DOI]/[format]
Publication: /pub/[DOI]/[format]

medRxiv API

Same structure as bioRxiv

Data Storage

Historical data is stored in data/history.json for:

Trend comparison
Velocity calculation
Duplicate detection

Examples

Example 1: Quick Daily Scan (arXiv - Recommended)

python scripts/main.py --sources arxiv --days 1 --output markdown

Example 2: Daily Scan with bioRxiv (May not work)

python scripts/main.py --sources biorxiv --days 1 --output markdown
# Note: May return 0 results due to Cloudflare protection

### Example 2: Weekly Deep Analysis

```bash
python scripts/main.py \
  --days 7 \
  --min-score 0.7 \
  --max-topics 50 \
  --output json \
  > weekly_report.json

Example 3: Track Specific Research Area

python scripts/main.py \
  --keywords "Alzheimer,neurodegeneration,amyloid" \
  --days 30 \
  --min-score 0.5

Known Issues

bioRxiv/medRxiv Cloudflare Protection

Status: ❌ Blocked
Issue: bioRxiv and medRxiv RSS feeds are protected by Cloudflare JavaScript Challenge, which prevents programmatic access. The site returns an HTML page requiring JavaScript execution and cookie validation.

Attempted Solutions:

✅ Added browser User-Agent headers → Failed (Cloudflare detects bot)
✅ Added complete browser headers (Accept, Accept-Language, etc.) → Failed
❌ Browser automation (Selenium/Playwright) → Not implemented (complex, heavy dependency)

Workaround: ✅ Use arXiv instead

arXiv q-bio (Quantitative Biology) RSS is accessible without protection
Contains computational biology, bioinformatics, and quantitative biology papers
Successfully tested: 35+ papers fetched in 30-day window

Usage:

# Recommended: Use arXiv
python scripts/main.py --sources arxiv --days 30

# Not working: bioRxiv/medRxiv
python scripts/main.py --sources biorxiv medrxiv --days 30  # Returns 0 papers

Troubleshooting

Rate Limiting

If you encounter rate limits, increase the --delay parameter (default: 1s between requests).

Missing Papers (0 results from bioRxiv/medRxiv)

This is expected due to Cloudflare protection. Use --sources arxiv instead.

RSS Feed Access Denied

Some institutional firewalls may block preprint servers. Ensure you can access:

✅ https://export.arxiv.org/rss/q-bio (should work)
❌ https://www.biorxiv.org/rss/recent.rss (Cloudflare blocked)

Low Trending Scores

For niche topics, lower --min-score threshold or increase --days for more data.

References

See references/README.md for:

API documentation links
Research papers on trend detection
Related tools and resources

License

MIT License - Part of OpenClaw Skills Collection

Risk Assessment

Risk Indicator	Assessment	Level
Code Execution	Python scripts with tools	High
Network Access	External API calls	High
File System Access	Read/write data	Medium
Instruction Tampering	Standard prompt guidelines	Low
Data Exposure	Data handled securely	Medium

Security Checklist

No hardcoded credentials or API keys
No unauthorized file system access (../)
Output does not expose sensitive information
Prompt injection protections in place
API requests use HTTPS only
Input validated against allowed patterns
API timeout and retry mechanisms implemented
Output directory restricted to workspace
Script execution in sandboxed environment
Error messages sanitized (no internal paths exposed)
Dependencies audited
No exposure of internal service architecture

Prerequisites

# Python dependencies
pip install -r requirements.txt

Evaluation Criteria

Success Metrics

Successfully executes main functionality
Output meets quality standards
Handles edge cases gracefully
Performance is acceptable

Test Cases

Basic Functionality: Standard input → Expected output
Edge Case: Invalid input → Graceful error handling
Performance: Large dataset → Acceptable processing time

Lifecycle Status

Current Stage: Draft
Next Review Date: 2026-03-06
Known Issues:
- ⚠️ bioRxiv/medRxiv blocked by Cloudflare (use arXiv as workaround)
- Network access limitations for some RSS feeds
Planned Improvements:
- Investigate bioRxiv/medRxiv API alternatives
- Consider browser automation for Cloudflare bypass
- Add more arXiv categories (q-bio subcategories)
- Performance optimization

安全使用建议

This skill is internally consistent with its stated purpose, but before installing: (1) inspect the remainder of scripts/main.py (especially the notification code triggered by --notify) to confirm it doesn't post data to unexpected external endpoints or expect credentials; (2) run pip install inside a virtualenv or container; (3) be aware that the script uses browser-like headers to avoid simple bot-blocking (it will not bypass Cloudflare JS challenges); (4) confirm that writing history.json in the skill workspace is acceptable for your environment and that no sensitive files will be read; and (5) if you plan to enable continuous or automated runs, ensure rate-limiting and robots.txt compliance to avoid abuse of target services.

功能分析

Type: OpenClaw Skill Name: emerging-topic-scout Version: 0.1.0 The 'emerging-topic-scout' skill is a legitimate research tool designed to monitor academic preprint servers (arXiv, bioRxiv, medRxiv) for trending topics. The core logic in `scripts/main.py` uses standard libraries like `feedparser` and `requests` to fetch RSS feeds and API data, performing keyword extraction and trend scoring locally. While the skill requires network access and file system writes (to `data/history.json`), these actions are transparently documented in `SKILL.md` and are essential for its stated purpose. No evidence of data exfiltration, malicious execution, or prompt injection was found; the code is well-structured and aligns with the provided documentation.

能力评估

✓ Purpose & Capability

Name/description match the code and SKILL.md: the package fetches RSS/API feeds (arXiv, bioRxiv, medRxiv where possible), extracts text, runs NLP/topic-detection, and stores history. No unrelated credentials, binaries, or system accesses are requested.

ℹ Instruction Scope

Runtime instructions are narrowly scoped to fetching feeds, running analysis, and emitting reports. The SKILL.md and code deliberately note Cloudflare protection for bioRxiv/medRxiv and recommend arXiv. One behavioral note: the code uses browser-like headers to avoid simple bot detection; this is understandable for scraping RSS but is a boundary behavior you should be aware of. The SKILL.md references a --notify flag but does not document the notification endpoint or mechanism in the excerpt — review the implementation before enabling notifications.

✓ Install Mechanism

No install spec in the registry; SKILL.md instructs pip installing the included requirements files. All dependencies are typical Python packages available on PyPI. There are no downloads from untrusted URLs or archive extraction steps.

✓ Credentials

The skill declares no required environment variables or credentials. The code performs network requests to public RSS/APIs and writes a local history.json inside the skill workspace — this is proportionate to its purpose. There is no request for unrelated secrets.

✓ Persistence & Privilege

The skill does not request always:true and does not modify system-wide agent configs. It writes/reads history.json within its own data directory (expected for a local history/cache) and otherwise runs on demand.

版本历史

v0.1.0

- Initial release of Emerging Topic Scout for real-time monitoring of emerging research topics in biological and medical sciences. - Supports trend detection by analyzing preprint servers (arXiv q-bio recommended) and social/academic discussions. - bioRxiv and medRxiv programmatic access currently blocked by Cloudflare; arXiv remains fully supported. - Provides configurable scanning, keyword tracking, trending score calculation, and multiple output formats (JSON, Markdown, CSV). - Example configurations and output formats included; recommends arXiv as a reliable data source. - Documentation covers usage instructions, configuration, output, and known issues.

元数据

Slug emerging-topic-scout

版本 0.1.0

许可证 MIT-0

累计安装 0

当前安装数 0

历史版本数 1

常见问题