multi-platform trending topic aggregation with unified normalization
Crawls 11+ heterogeneous platforms (Zhihu, Weibo, Bilibili, Twitter, Reddit, HackerNews, etc.) and RSS feeds using platform-specific scrapers, normalizes disparate data schemas into a unified NewsItem model, and deduplicates content across sources using fuzzy title matching and URL canonicalization. The system maintains platform-specific metadata (rank, heat value, engagement metrics) while presenting a single normalized feed, enabling cross-platform trend detection that would be invisible within individual platform silos.
Unique: Implements platform-specific crawler modules with unified NewsItem schema and fuzzy deduplication across 11+ heterogeneous sources (Chinese + international), rather than relying on single-platform APIs or generic RSS parsing. Maintains platform-specific metadata (rank × 0.6 + frequency × 0.3 + platform hot value × 0.1) for weighted hotspot scoring.
vs alternatives: Covers more platforms (especially Chinese social media) with deeper metadata extraction than generic RSS aggregators, and provides unified deduplication across sources unlike single-platform monitoring tools.
keyword-based content filtering with regex and boolean logic
Implements a multi-stage filtering pipeline that matches news items against user-defined keywords using regex patterns, required word lists, and excluded word lists. The system applies frequency-based scoring (keyword occurrence count) combined with platform hotspot weights to rank filtered results. Configuration is stored in frequency_words.txt with support for regex patterns, AND/OR/NOT boolean operators, and per-keyword weighting. Filtering occurs at collection time (reducing storage) and again at report generation time (enabling dynamic reconfiguration without re-crawling).
Unique: Combines regex pattern matching with frequency-based scoring and platform hotspot weighting (rank × 0.6 + frequency × 0.3 + platform hot value × 0.1) in a two-stage pipeline (collection-time and report-time filtering). Supports dynamic reconfiguration without re-crawling by applying filters at report generation.
vs alternatives: More flexible than simple keyword matching (supports regex and boolean logic) and more efficient than semantic filtering (no LLM overhead), making it suitable for real-time filtering at scale.
real-time new topic detection with 🆕 markers and trend emergence tracking
Detects newly emerged topics by comparing current crawl results against historical data stored in the system. Topics are marked as 🆕 (new) if they appear for the first time in the current crawl or if their hotspot rank increased significantly compared to previous crawls. The system tracks topic emergence velocity (how quickly a topic rises in rankings) and flags topics with unusual acceleration. New topic detection is performed at report generation time, enabling dynamic detection without re-crawling. The system maintains a historical hotspot index for comparison.
Unique: Detects new topics by comparing current hotspot rankings against historical data, marking topics with significant rank increases as 🆕. Tracks emergence velocity to distinguish breaking news from sustained trends.
vs alternatives: More efficient than semantic similarity detection (no LLM overhead) and more accurate than simple first-appearance detection (accounts for re-emerging topics), but requires historical baseline data.
visual configuration editor with real-time validation and preview
Provides a web-based UI for editing TrendRadar configuration files (config.yaml, frequency_words.txt, timeline.yaml) with real-time validation and preview. The editor supports: (1) syntax highlighting for YAML and regex, (2) validation of keyword patterns (regex compilation check), (3) preview of filtered results based on current keyword configuration, (4) drag-and-drop channel configuration, (5) schedule preview (shows next 10 execution times). Changes are validated before saving, preventing configuration errors. The editor is optional; users can edit config files directly.
Unique: Provides web-based configuration editor with real-time validation, regex preview, and schedule visualization. Enables non-technical users to configure TrendRadar without editing YAML files.
vs alternatives: More user-friendly than manual YAML editing and provides validation feedback, but adds operational complexity compared to file-based configuration.
ai-powered news analysis and summarization via litellm multi-provider abstraction
Integrates LiteLLM to provide vendor-agnostic AI analysis and summarization of filtered news items. Users configure their preferred LLM provider (OpenAI, Anthropic, Ollama, local models, etc.) once in config.yaml, and the system automatically routes analysis requests to that provider. The AI analysis capability includes: (1) automated summarization of long articles into key points, (2) sentiment analysis (positive/negative/neutral), (3) trend prediction based on historical patterns, and (4) custom analysis prompts. Analysis results are cached to avoid redundant API calls and can be pushed directly to notification channels.
Unique: Uses LiteLLM abstraction layer to support any LLM provider (OpenAI, Anthropic, Ollama, local models) with single configuration, enabling provider switching without code changes. Caches analysis results to reduce redundant API calls and costs.
vs alternatives: More flexible than hardcoded OpenAI integration (supports any LiteLLM provider) and cheaper than dedicated sentiment analysis APIs (can use local models), but slower than rule-based sentiment analysis.
ai-powered translation with multi-language support and caching
Leverages LiteLLM to translate news content from source languages (primarily Chinese) to target languages (English, etc.) on-demand. The system detects source language automatically (via langdetect or similar), caches translations to avoid re-translating identical content, and batches translation requests to reduce API calls. Translations are stored alongside original content, enabling bilingual reports and multi-language notification delivery. Translation can be triggered at collection time (all news) or report time (only filtered news).
Unique: Implements provider-agnostic translation via LiteLLM with automatic language detection, content-based caching, and batch request optimization. Stores translations alongside originals for bilingual report generation.
vs alternatives: More flexible than dedicated translation APIs (supports any LiteLLM provider) and cheaper than commercial translation services when using local models, but slower than specialized translation APIs.
multi-channel notification delivery with atomic message batching
Implements a notification abstraction layer supporting 9+ delivery channels (WeChat, WeWork, Feishu, Telegram, Email, ntfy, Bark, Slack, etc.). Each channel has a provider-specific formatter that converts normalized news items into channel-appropriate messages (e.g., WeChat card format, Telegram markdown, email HTML). The system batches notifications atomically—all news items for a report are sent as a single batch to each channel, ensuring consistency and reducing API calls. Message formatting respects channel constraints (character limits, attachment limits, etc.) and supports templating for customization.
Unique: Implements atomic message batching across 9+ heterogeneous channels with provider-specific formatters and constraint-aware truncation. Single configuration enables simultaneous delivery to WeChat, WeWork, Feishu, Telegram, Email, ntfy, Bark, Slack, etc. without code changes.
vs alternatives: Supports more channels (especially Chinese platforms like WeWork, Feishu) than generic notification services, and batching reduces API calls and spam compared to per-item notifications.
model context protocol (mcp) server for conversational trend analysis
Exposes TrendRadar's data and analysis capabilities as an MCP server, enabling AI agents and LLM applications to query trends, perform analysis, and generate insights through natural language. The MCP server implements tools for: (1) querying filtered news by keyword/date/platform, (2) retrieving trend statistics and hotspot rankings, (3) running custom analysis on news subsets, (4) generating reports in various formats. Clients (Claude, other LLM agents) can invoke these tools via MCP protocol, enabling conversational exploration of trends without direct database access. The server maintains state across multiple requests, allowing multi-turn conversations about trends.
Unique: Implements full MCP server exposing trend data and analysis tools to LLM agents, enabling conversational queries and multi-turn analysis workflows. Maintains state across requests and supports complex tool invocations (filtering, analysis, report generation).
vs alternatives: Enables conversational access to trends (vs. API-only access) and integrates with LLM agent workflows (vs. standalone tools), but adds operational complexity compared to simple REST APIs.
+4 more capabilities