TrendRadar

Crawls 11+ heterogeneous platforms (Zhihu, Weibo, Bilibili, Twitter, Reddit, HackerNews, etc.) and RSS feeds using platform-specific scrapers, normalizes disparate data schemas into a unified NewsItem model, and deduplicates content across sources using fuzzy title matching and URL canonicalization. The system maintains platform-specific metadata (rank, heat value, engagement metrics) while presenting a single normalized feed, enabling cross-platform trend detection that would be invisible within individual platform silos.

Implements a multi-stage filtering pipeline that matches news items against user-defined keywords using regex patterns, required word lists, and excluded word lists. The system applies frequency-based scoring (keyword occurrence count) combined with platform hotspot weights to rank filtered results. Configuration is stored in frequency_words.txt with support for regex patterns, AND/OR/NOT boolean operators, and per-keyword weighting. Filtering occurs at collection time (reducing storage) and again at report generation time (enabling dynamic reconfiguration without re-crawling).

Detects newly emerged topics by comparing current crawl results against historical data stored in the system. Topics are marked as 🆕 (new) if they appear for the first time in the current crawl or if their hotspot rank increased significantly compared to previous crawls. The system tracks topic emergence velocity (how quickly a topic rises in rankings) and flags topics with unusual acceleration. New topic detection is performed at report generation time, enabling dynamic detection without re-crawling. The system maintains a historical hotspot index for comparison.

Provides a web-based UI for editing TrendRadar configuration files (config.yaml, frequency_words.txt, timeline.yaml) with real-time validation and preview. The editor supports: (1) syntax highlighting for YAML and regex, (2) validation of keyword patterns (regex compilation check), (3) preview of filtered results based on current keyword configuration, (4) drag-and-drop channel configuration, (5) schedule preview (shows next 10 execution times). Changes are validated before saving, preventing configuration errors. The editor is optional; users can edit config files directly.

Integrates LiteLLM to provide vendor-agnostic AI analysis and summarization of filtered news items. Users configure their preferred LLM provider (OpenAI, Anthropic, Ollama, local models, etc.) once in config.yaml, and the system automatically routes analysis requests to that provider. The AI analysis capability includes: (1) automated summarization of long articles into key points, (2) sentiment analysis (positive/negative/neutral), (3) trend prediction based on historical patterns, and (4) custom analysis prompts. Analysis results are cached to avoid redundant API calls and can be pushed directly to notification channels.

Leverages LiteLLM to translate news content from source languages (primarily Chinese) to target languages (English, etc.) on-demand. The system detects source language automatically (via langdetect or similar), caches translations to avoid re-translating identical content, and batches translation requests to reduce API calls. Translations are stored alongside original content, enabling bilingual reports and multi-language notification delivery. Translation can be triggered at collection time (all news) or report time (only filtered news).

Implements a notification abstraction layer supporting 9+ delivery channels (WeChat, WeWork, Feishu, Telegram, Email, ntfy, Bark, Slack, etc.). Each channel has a provider-specific formatter that converts normalized news items into channel-appropriate messages (e.g., WeChat card format, Telegram markdown, email HTML). The system batches notifications atomically—all news items for a report are sent as a single batch to each channel, ensuring consistency and reducing API calls. Message formatting respects channel constraints (character limits, attachment limits, etc.) and supports templating for customization.

Exposes TrendRadar's data and analysis capabilities as an MCP server, enabling AI agents and LLM applications to query trends, perform analysis, and generate insights through natural language. The MCP server implements tools for: (1) querying filtered news by keyword/date/platform, (2) retrieving trend statistics and hotspot rankings, (3) running custom analysis on news subsets, (4) generating reports in various formats. Clients (Claude, other LLM agents) can invoke these tools via MCP protocol, enabling conversational exploration of trends without direct database access. The server maintains state across multiple requests, allowing multi-turn conversations about trends.

Generates customizable trend reports in multiple formats (JSON, CSV, Markdown, HTML, plain text) with configurable content sections (summary, hotspots, new topics, platform breakdown, sentiment analysis, etc.). The system supports three report modes: (1) SIMPLE mode (top N items with basic metadata), (2) DETAILED mode (full content + analysis + sentiment), (3) ANALYSIS mode (AI-generated insights + trend predictions). Reports can be filtered by keyword, date range, platform, and hotspot threshold. Report generation is decoupled from notification delivery, enabling on-demand report generation and archival.

Implements a scheduler that executes the entire TrendRadar pipeline (crawl → filter → analyze → report → notify) on a configurable schedule defined in timeline.yaml. The scheduler supports multiple execution modes: (1) fixed intervals (e.g., every 6 hours), (2) cron expressions (e.g., '0 9 * * MON-FRI'), (3) one-time execution. Each scheduled job can have different configurations (keywords, report mode, notification channels), enabling multiple independent workflows. The scheduler handles job queuing, error recovery, and execution logging. Execution can be triggered via GitHub Actions (cloud), Docker (containerized), or local Python process.

Supports three distinct deployment models with identical functionality but different operational characteristics: (1) GitHub Actions deployment (serverless, cloud storage, no infrastructure management), (2) Docker deployment (containerized, local storage, self-hosted), (3) Local Python deployment (direct execution, minimal setup). Each deployment model uses the same core TrendRadar code but with environment-specific configuration (storage backend, scheduler implementation, notification credentials). The system abstracts storage (local filesystem, S3, cloud storage) and scheduler (GitHub Actions cron, Docker cron, APScheduler) behind interfaces, enabling seamless switching between deployment models.

Implements a storage abstraction layer supporting multiple backends: local filesystem (for development/testing), AWS S3 (for cloud deployment), and other cloud storage services. The system stores: (1) raw crawled news items (for deduplication and historical analysis), (2) filtered reports (for archival and audit), (3) analysis results (for caching and reuse), (4) execution logs (for monitoring and debugging). Storage is organized by date and source for efficient querying. The abstraction enables switching backends via configuration without code changes. Data can be stored locally (for privacy) or in cloud (for accessibility and backup).