poiuy/my-daily
0
0
Fork 0
Code Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 军事科技每日资讯推送系统 - Docker部署 + 日志系统 + 数据目录重组

Browse Source
This commit is contained in:
poiuy
2026-07-12 20:01:02 +08:00
commit 54ca4b1b6a
267 changed files with 47047 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+45
View File
@@ -0,0 +1,45 @@
# 环境配置（含敏感凭据）
.env
config/.env
my-daily/.env
config.json
my-daily/config.json
# Python
__pycache__/
*.py[cod]
*.egg-info/
*.egg
dist/
build/
# 虚拟环境
venv/
.venv/
env/
.mypy_cache/
# 运行时数据
data/
output/
logs/
cache/
# Docker 镜像（太大）
*.tar
# 日志
*.log
# IDE
.vscode/
.idea/
*.swp
*.swo
# Docker
.docker/
# OS
.DS_Store
Thumbs.db
01主函数过长问题.md
+727
View File
@@ -0,0 +1,727 @@
# 1. 问题
军事科技每日摘报的 `main()` 函数（位于 `scripts/military_daily_report_v3.py` 第38-246行）长达208行，承担了整个业务流程的11个步骤，严重违反单一职责原则。
## 1.1. **职责过多，难以理解**
当前 `main()` 函数包含了从RSS抓取到报告输出的完整业务流程，涉及数据获取、清洗、翻译、AI分析、格式化等多个不同层次的职责。开发者需要在200多行代码中理解整个业务逻辑，认知负担过重。
**问题代码片段（第38-246行）**：
```python
def main():
now = datetime.now()
cutoff = now - timedelta(hours=TIME_WINDOW_HOURS)
date_str = now.strftime('%Y%m%d')
# 1. RSS 抓取
all_articles = fetch_all_feeds()
# 2. 24小时时间过滤
recent_articles = [a for a in all_articles if a['published'] >= cutoff]
# 3. 标题去重
seen = set()
unique_articles = []
for a in recent_articles:
key = a['title'].strip()
if key not in seen and len(key) > 5:
seen.add(key)
unique_articles.append(a)
# 4. 排序
all_time_valid = sorted(unique_articles, key=lambda x: x['published'], reverse=True)
# 5. 加载AI配置 + 批量翻译
api_key, base_url, model, webzine_model = load_ai_config()
translated_articles = batch_translate_articles(all_time_valid, api_key, base_url, model, max_concurrent=2)
# 6. 关键词筛选
if len(translated_articles) > 15:
filtered_articles = keyword_filter(translated_articles)
else:
filtered_articles = translated_articles
# 7. AI分析
for i, article in enumerate(filtered_articles, 1):
ai_result = score_and_classify_article(article, api_key, base_url, model)
article['ai_result'] = ai_result
article['final_score'] = calculate_weighted_score(ai_result)
article['summary'] = ai_result.get('summary', article['translated_title'])
article['category'] = ai_result.get('category', '装备动态')
article['scores'] = ai_result.get('scores', {})
# 8. 按评分排序
filtered_articles.sort(key=lambda x: x['final_score'], reverse=True)
# 9. 生成TOP3网摘
today_top3 = filtered_articles[:3]
# ... 网摘生成代码
# 10. 分类整理 + 生成分类洞察
# ... 分类整理代码
# 11. 生成Markdown报告
# ... 报告生成代码
```
**问题分析**：
* 11个业务步骤混在一个函数中，每个步骤的逻辑细节都暴露在主流程中
* 变量名重复使用（如 `filtered_articles`），容易产生混淆
* 中间状态（如 `today_top3`、`equipment_articles` 等）难以追踪
* 无法快速定位某个具体步骤的代码位置
## 1.2. **难以测试，无法复用**
由于所有逻辑都在 `main()` 函数中，无法对单个业务步骤进行独立的单元测试。例如，想要测试"去重逻辑"或"分类整理逻辑"，必须运行整个流程，这大大增加了测试成本和复杂度。
**问题分析**：
* 去重逻辑（第62-71行）嵌入在主函数中，无法单独测试
* 分类整理逻辑（第175-185行）无法独立验证
* 网摘生成逻辑（第127-159行）与文件IO耦合，难以mock测试
* 想要复用某个步骤（如只做RSS抓取和去重）无法实现
## 1.3. **维护困难，容易出错**
当需要修改某个业务步骤时，开发者必须在200多行代码中定位相关逻辑，容易遗漏或引入错误。例如，如果需要调整"关键词筛选"的触发条件，需要在第88-93行找到相关代码，但很容易在复杂的流程中迷失。
**问题分析**：
* 修改某个步骤时，需要理解整个上下文
* 容易在修改时影响其他步骤的逻辑
* 代码审查时难以快速定位修改点
* 新人上手成本高，需要阅读大量代码才能理解业务流程
# 2. 收益
通过将 `main()` 函数拆分为多个职责明确的子函数，可以显著提升代码的可读性、可测试性和可维护性。
## 2.1. **提升代码可读性**
重构后，每个业务步骤都有独立的函数，函数名清晰表达了其职责。主函数变成一个简洁的流程编排器，一眼就能看出整个业务流程的11个步骤。
**预期改进**：
* 主函数从208行缩减到约30行
* 每个子函数平均15-30行，职责单一
* 函数名直接表达业务含义（如 `filter_by_time_window`、`deduplicate_articles`）
* 代码结构清晰，新人可以在5分钟内理解整体流程
## 2.2. **提升可测试性**
每个业务步骤都可以独立进行单元测试，无需运行整个流程。可以mock输入数据，验证每个步骤的输出是否符合预期。
**预期改进**：
* 可以为每个子函数编写独立的单元测试
* 测试覆盖率可以从当前的接近0%提升到80%以上
* 测试执行时间从分钟级降低到秒级
* 可以快速定位和修复bug
## 2.3. **提升可维护性**
当需要修改某个业务步骤时，只需要定位到对应的子函数，修改范围明确，不会影响其他步骤。代码审查时也可以快速定位修改点。
**预期改进**：
* 修改某个步骤时，只需要关注对应的子函数
* 代码审查时可以快速定位修改的函数
* 降低引入新bug的风险
* 提高开发效率
## 2.4. **提升代码复用性**
业务步骤可以独立使用，例如只做RSS抓取和去重，或者只做AI分析。这为后续的功能扩展提供了灵活性。
**预期改进**：
* 可以单独调用某个业务步骤
* 可以组合不同的步骤实现新的功能
* 为后续的功能扩展提供基础
# 3. 方案
将 `main()` 函数拆分为11个职责明确的子函数，每个函数负责一个独立的业务步骤。主函数变成一个简洁的流程编排器，负责调用这些子函数。
## 3.1. **函数拆分设计**
```mermaid
flowchart TD
A[main] --> B[fetch_and_filter_articles]
B --> C[translate_articles]
C --> D[filter_by_keywords]
D --> E[analyze_articles]
E --> F[generate_webzine]
F --> G[generate_report]
B --> B1[fetch_all_feeds]
B --> B2[filter_by_time_window]
B --> B3[deduplicate_articles]
B --> B4[sort_by_publish_time]
E --> E1[ai_score_and_classify]
E --> E2[sort_by_score]
G --> G1[categorize_articles]
G --> G2[generate_category_overviews]
G --> G3[write_markdown_report]
style A fill:#e1f5e1
style B fill:#fff4e1
style C fill:#fff4e1
style D fill:#fff4e1
style E fill:#fff4e1
style F fill:#fff4e1
style G fill:#fff4e1
```
**图表说明**：
* 绿色节点表示主函数，负责流程编排
* 黄色节点表示拆分后的子函数，每个函数负责一个独立的业务步骤
* 箭头表示调用关系
* 通过这种拆分，主函数变得简洁，每个子函数职责单一
## 3.2. **函数拆分实现**
### **步骤1：提取数据获取和过滤逻辑**
**问题代码**：
```python
def main():
# 1. RSS 抓取
all_articles = fetch_all_feeds()
# 2. 24小时时间过滤
recent_articles = [a for a in all_articles if a['published'] >= cutoff]
# 3. 标题去重
seen = set()
unique_articles = []
for a in recent_articles:
key = a['title'].strip()
if key not in seen and len(key) > 5:
seen.add(key)
unique_articles.append(a)
# 4. 排序
all_time_valid = sorted(unique_articles, key=lambda x: x['published'], reverse=True)
```
**重构后代码**：
```python
def filter_by_time_window(articles, cutoff):
"""过滤出指定时间窗口内的文章"""
return [a for a in articles if a['published'] >= cutoff]
def deduplicate_articles(articles):
"""根据标题去重文章"""
seen = set()
unique_articles = []
for a in articles:
key = a['title'].strip()
if key not in seen and len(key) > 5:
seen.add(key)
unique_articles.append(a)
return unique_articles
def sort_by_publish_time(articles, reverse=True):
"""按发布时间排序文章"""
return sorted(articles, key=lambda x: x['published'], reverse=reverse)
def fetch_and_filter_articles(cutoff):
"""获取并过滤文章：抓取 -> 时间过滤 -> 去重 -> 排序"""
all_articles = fetch_all_feeds()
recent_articles = filter_by_time_window(all_articles, cutoff)
unique_articles = deduplicate_articles(recent_articles)
return sort_by_publish_time(unique_articles)
```
**改进点**：
* 每个函数职责单一，函数名清晰表达其功能
* 可以独立测试每个步骤
* 可以单独复用某个步骤
### **步骤2：提取翻译和关键词筛选逻辑**
**问题代码**：
```python
def main():
# 5. 加载AI配置 + 批量翻译
api_key, base_url, model, webzine_model = load_ai_config()
translated_articles = batch_translate_articles(all_time_valid, api_key, base_url, model, max_concurrent=2)
# 6. 关键词筛选
if len(translated_articles) > 15:
filtered_articles = keyword_filter(translated_articles)
else:
filtered_articles = translated_articles
```
**重构后代码**：
```python
def filter_by_keywords(articles):
"""根据关键词筛选文章（仅当文章数 > 15 时执行）"""
if len(articles) > 15:
return keyword_filter(articles)
return articles
def translate_articles(articles, api_key, base_url, model):
"""批量翻译外文文章"""
return batch_translate_articles(articles, api_key, base_url, model, max_concurrent=2)
```
**改进点**：
* 将关键词筛选的条件判断封装在函数内部
* 翻译逻辑独立，可以单独测试
### **步骤3：提取AI分析和排序逻辑**
**问题代码**：
```python
def main():
# 7. AI分析
for i, article in enumerate(filtered_articles, 1):
ai_result = score_and_classify_article(article, api_key, base_url, model)
article['ai_result'] = ai_result
article['final_score'] = calculate_weighted_score(ai_result)
article['summary'] = ai_result.get('summary', article['translated_title'])
article['category'] = ai_result.get('category', '装备动态')
article['scores'] = ai_result.get('scores', {})
# 8. 按评分排序
filtered_articles.sort(key=lambda x: x['final_score'], reverse=True)
```
**重构后代码**：
```python
def ai_score_and_classify(article, api_key, base_url, model):
"""对单篇文章进行AI分析：评分、摘要、分类"""
ai_result = score_and_classify_article(article, api_key, base_url, model)
article['ai_result'] = ai_result
article['final_score'] = calculate_weighted_score(ai_result)
article['summary'] = ai_result.get('summary', article['translated_title'])
article['category'] = ai_result.get('category', '装备动态')
article['scores'] = ai_result.get('scores', {})
return article
def analyze_articles(articles, api_key, base_url, model):
"""对所有文章进行AI分析并按评分排序"""
for i, article in enumerate(articles, 1):
logger.info("处理中 %d/%d: %s...", i, len(articles), article['translated_title'][:35])
ai_score_and_classify(article, api_key, base_url, model)
# 按评分排序
return sorted(articles, key=lambda x: x['final_score'], reverse=True)
```
**改进点**：
* 将单篇文章的AI分析和批量分析分离
* 排序逻辑封装在函数内部
* 可以单独测试AI分析逻辑
### **步骤4：提取网摘生成逻辑**
**问题代码**：
```python
def main():
# 9. 提取TOP3 + 生成《参考消息》风格网摘
today_top3 = filtered_articles[:3]
webzine_content = []
webzine_content.append(f"═{'═'*60}═")
webzine_content.append(f" 🔥 军事科技每日摘报 - 今日必看 TOP3 网摘")
webzine_content.append(f" 📅 {now.strftime('%Y年%m月%d日')}")
webzine_content.append(f"═{'═'*60}═")
webzine_content.append("")
webzine_texts = []
for i, article in enumerate(today_top3, 1):
webzine = generate_webzine_for_article(article, api_key, base_url, webzine_model)
webzine_texts.append(webzine)
webzine_content.append(f"{'='*62}")
webzine_content.append(f" 📌 第 {i} 篇")
webzine_content.append(f"{'='*62}")
webzine_content.append("")
webzine_content.append(webzine)
webzine_content.append("")
webzine_content.append(f"🔗 原文链接：{article['link']}")
webzine_content.append("")
webzine_content.append("")
# 保存网摘TXT文件
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
webzine_file = OUTPUT_DIR / f"military_webzine_{date_str}.txt"
with open(webzine_file, 'w', encoding='utf-8') as f:
f.write('\n'.join(webzine_content))
# 生成合并长图
webzine_image = OUTPUT_DIR / f"military_webzine_{date_str}.png"
create_combined_webzine_image(webzine_texts, str(webzine_image), now)
```
**重构后代码**：
```python
def generate_webzine_content(top3_articles, now):
"""生成网摘文本内容"""
webzine_content = []
webzine_content.append(f"═{'═'*60}═")
webzine_content.append(f" 🔥 军事科技每日摘报 - 今日必看 TOP3 网摘")
webzine_content.append(f" 📅 {now.strftime('%Y年%m月%d日')}")
webzine_content.append(f"═{'═'*60}═")
webzine_content.append("")
webzine_texts = []
for i, article in enumerate(top3_articles, 1):
webzine = generate_webzine_for_article(article, api_key, base_url, webzine_model)
webzine_texts.append(webzine)
webzine_content.append(f"{'='*62}")
webzine_content.append(f" 📌 第 {i} 篇")
webzine_content.append(f"{'='*62}")
webzine_content.append("")
webzine_content.append(webzine)
webzine_content.append("")
webzine_content.append(f"🔗 原文链接：{article['link']}")
webzine_content.append("")
webzine_content.append("")
return webzine_content, webzine_texts
def save_webzine_files(webzine_content, webzine_texts, date_str, now):
"""保存网摘文件（TXT和PNG）"""
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
# 保存TXT文件
webzine_file = OUTPUT_DIR / f"military_webzine_{date_str}.txt"
with open(webzine_file, 'w', encoding='utf-8') as f:
f.write('\n'.join(webzine_content))
logger.info("网摘文件已保存: %s", webzine_file)
# 生成PNG文件
webzine_image = OUTPUT_DIR / f"military_webzine_{date_str}.png"
create_combined_webzine_image(webzine_texts, str(webzine_image), now)
def generate_webzine(articles, api_key, base_url, webzine_model, date_str, now):
"""生成TOP3网摘并保存文件"""
top3 = articles[:3]
webzine_content, webzine_texts = generate_webzine_content(top3, now)
save_webzine_files(webzine_content, webzine_texts, date_str, now)
```
**改进点**：
* 将网摘内容生成和文件保存分离
* 可以独立测试网摘内容生成逻辑
* 文件IO逻辑集中，易于mock
### **步骤5：提取分类整理和报告生成逻辑**
**问题代码**：
```python
def main():
# 10. 分类整理 + 生成各分类深度洞察
equipment_articles = []
conflict_articles = []
strategy_articles = []
for article in filtered_articles[3:]:
cat = article.get('category', '装备动态')
if '冲突' in cat or '战争' in cat or '地区' in cat:
conflict_articles.append(article)
elif '战略' in cat or '政策' in cat or '外交' in cat:
strategy_articles.append(article)
else:
equipment_articles.append(article)
today_overview = generate_category_overview("今日必看", today_top3, api_key, base_url, model)
equipment_overview = generate_category_overview("装备动态", equipment_articles, api_key, base_url, model)
conflict_overview = generate_category_overview("地区冲突", conflict_articles, api_key, base_url, model)
strategy_overview = generate_category_overview("战略政策", strategy_articles, api_key, base_url, model)
# 11. 生成 Markdown 报告文件
md_lines = []
md_lines.append(f"# 🔥 {now.year}年{now.month}月{now.day}日 军事科技每日摘报\n")
# ... 大量Markdown拼接代码
```
**重构后代码**：
```python
def categorize_articles(articles):
"""将文章按分类整理"""
equipment_articles = []
conflict_articles = []
strategy_articles = []
for article in articles:
cat = article.get('category', '装备动态')
if '冲突' in cat or '战争' in cat or '地区' in cat:
conflict_articles.append(article)
elif '战略' in cat or '政策' in cat or '外交' in cat:
strategy_articles.append(article)
else:
equipment_articles.append(article)
return equipment_articles, conflict_articles, strategy_articles
def generate_category_overviews(top3, equipment, conflict, strategy, api_key, base_url, model):
"""生成各分类的总体介绍"""
today_overview = generate_category_overview("今日必看", top3, api_key, base_url, model)
equipment_overview = generate_category_overview("装备动态", equipment, api_key, base_url, model)
conflict_overview = generate_category_overview("地区冲突", conflict, api_key, base_url, model)
strategy_overview = generate_category_overview("战略政策", strategy, api_key, base_url, model)
return {
'today': today_overview,
'equipment': equipment_overview,
'conflict': conflict_overview,
'strategy': strategy_overview
}
def write_markdown_report(articles, overviews, date_str, now):
"""生成Markdown报告文件"""
top3 = articles[:3]
equipment, conflict, strategy = categorize_articles(articles[3:])
md_lines = []
md_lines.append(f"# 🔥 {now.year}年{now.month}月{now.day}日 军事科技每日摘报\n")
md_lines.append(f"📊 今日共 {len(articles)} 篇精选文章 | ⏰ {now.strftime('%Y-%m-%d %H:%M')}\n")
# 今日必看
md_lines.append(f"## 🔥 今日必看\n")
if overviews['today']:
md_lines.append(f"> {overviews['today']}\n")
for i, article in enumerate(top3, 1):
md_lines.append(f"### {i}. {article['translated_title']}\n")
md_lines.append(f"- 📡 来源: {article['source']}")
md_lines.append(f"- ⏰ 时间: {article['published'].strftime('%m-%d %H:%M')}")
md_lines.append(f"- ⭐ 评分: {article['final_score']}/10")
md_lines.append(f"- 📝 {article['summary']}")
md_lines.append(f"- 🔗 [阅读原文]({article['link']})\n")
# 装备动态
if equipment:
md_lines.append(f"## ⚙️ 装备动态\n")
if overviews['equipment']:
md_lines.append(f"> {overviews['equipment']}\n")
for i, article in enumerate(equipment, 1):
md_lines.append(f"### {i}. {article['translated_title']}\n")
md_lines.append(f"- 📡 {article['source']} | ⏰ {article['published'].strftime('%m-%d %H:%M')}")
md_lines.append(f"- 📝 {article['summary']}")
md_lines.append(f"- 🔗 [阅读原文]({article['link']})\n")
# 地区冲突
if conflict:
md_lines.append(f"## 💥 地区冲突\n")
if overviews['conflict']:
md_lines.append(f"> {overviews['conflict']}\n")
for i, article in enumerate(conflict, 1):
md_lines.append(f"### {i}. {article['translated_title']}\n")
md_lines.append(f"- 📡 {article['source']} | ⏰ {article['published'].strftime('%m-%d %H:%M')}")
md_lines.append(f"- 📝 {article['summary']}")
md_lines.append(f"- 🔗 [阅读原文]({article['link']})\n")
# 战略政策
if strategy:
md_lines.append(f"## 🎯 战略政策\n")
if overviews['strategy']:
md_lines.append(f"> {overviews['strategy']}\n")
for i, article in enumerate(strategy, 1):
md_lines.append(f"### {i}. {article['translated_title']}\n")
md_lines.append(f"- 📡 {article['source']} | ⏰ {article['published'].strftime('%m-%d %H:%M')}")
md_lines.append(f"- 📝 {article['summary']}")
md_lines.append(f"- 🔗 [阅读原文]({article['link']})\n")
# 保存文件
report_file = OUTPUT_DIR / f"military_report_{date_str}.md"
with open(report_file, 'w', encoding='utf-8') as f:
f.write('\n'.join(md_lines))
logger.info("报告文件已保存: %s", report_file)
def generate_report(articles, api_key, base_url, model, date_str, now):
"""生成完整报告：分类整理 -> 生成分类洞察 -> 写入Markdown"""
equipment, conflict, strategy = categorize_articles(articles[3:])
top3 = articles[:3]
overviews = generate_category_overviews(
top3, equipment, conflict, strategy, api_key, base_url, model
)
write_markdown_report(articles, overviews, date_str, now)
```
**改进点**：
* 将分类整理、分类洞察生成、报告写入分离
* 每个函数职责单一，易于测试
* Markdown生成逻辑集中，易于维护
### **步骤6：重构后的主函数**
**重构后的主函数**：
```python
def main():
"""主函数：流程编排"""
now = datetime.now()
cutoff = now - timedelta(hours=TIME_WINDOW_HOURS)
date_str = now.strftime('%Y%m%d')
logger.info("开始抓取 %d 个军事科技 RSS 订阅源...", len(RSS_FEEDS))
logger.info("时间窗口: %s ~ %s", cutoff.strftime('%m-%d %H:%M'), now.strftime('%m-%d %H:%M'))
# 1. 获取并过滤文章
articles = fetch_and_filter_articles(cutoff)
logger.info("总共抓取 %d 篇文章", len(articles))
if not articles:
logger.info("今日暂无新的军事科技文章")
return
# 2. 翻译文章
api_key, base_url, model, webzine_model = load_ai_config()
if not api_key:
logger.error("未找到OPENAI_API_KEY，退出")
return
translated_articles = translate_articles(articles, api_key, base_url, model)
logger.info("翻译完成，共 %d 篇文章", len(translated_articles))
# 3. 关键词筛选
filtered_articles = filter_by_keywords(translated_articles)
logger.info("筛选后: %d 篇", len(filtered_articles))
if not filtered_articles:
logger.info("筛选后无符合条件文章")
return
# 4. AI分析
logger.info("开始 AI 智能分析（使用 %s）...", model)
analyzed_articles = analyze_articles(filtered_articles, api_key, base_url, model)
logger.info("AI 评分完成，最高分: %.1f/10", analyzed_articles[0]['final_score'])
# 5. 生成网摘
logger.info("生成今日必看 TOP3 网摘（《参考消息》风格）...")
generate_webzine(analyzed_articles, api_key, base_url, webzine_model, date_str, now)
# 6. 生成报告
logger.info("生成 Markdown 报告文件...")
generate_report(analyzed_articles, api_key, base_url, model, date_str, now)
```
**改进点**：
* 主函数从208行缩减到约50行
* 流程清晰，一眼就能看出整个业务流程
* 每个步骤都有清晰的注释
* 易于理解和维护
# 4. 回归范围
本次重构主要是代码结构调整，不改变业务逻辑，因此回归测试的重点是验证重构后的功能与重构前完全一致。
## 4.1. 主链路
**完整业务流程**：
1. RSS抓取 → 时间过滤 → 去重 → 排序 → 翻译 → 关键词筛选 → AI分析 → 评分排序 → 网摘生成 → 报告输出
**关键验证点**：
* RSS抓取能够正常获取12个订阅源的文章
* 时间过滤能够正确过滤出24小时内的文章
* 去重逻辑能够正确去除重复标题的文章
* 翻译功能能够正确翻译外文文章
* 关键词筛选在文章数 > 15 时能够正确执行
* AI分析能够正确生成评分、摘要和分类
* 网摘生成能够正确生成TOP3网摘的TXT和PNG文件
* 报告生成能够正确生成Markdown报告文件
## 4.2. 边界情况
**边界场景**：
1. **无新文章场景**：当24小时内没有新文章时，程序应该正常退出并输出日志
2. **筛选后无文章场景**：当关键词筛选后没有符合条件的文章时，程序应该正常退出并输出日志
3. **AI配置缺失场景**：当未配置OPENAI\_API\_KEY时，程序应该正常退出并输出错误日志
4. **文章数 <= 15场景**：当文章数 <= 15时，应该跳过关键词筛选，直接进入AI分析
5. **分类为空场景**：当某个分类（如"地区冲突"）没有文章时，报告中应该不显示该分类
6. **翻译失败场景**：当翻译失败时，应该使用原标题和原文内容，不影响后续流程
7. **AI分析失败场景**：当AI分析失败时，应该使用默认评分和分类，不影响后续流程
**验证方法**：
* 通过修改配置文件模拟各种边界场景
* 检查日志输出是否符合预期
* 检查生成的文件内容是否正确
* 检查程序是否正常退出或抛出预期的异常
02图片生成代码重复问题.md
+322
View File
@@ -0,0 +1,322 @@
# 1. 问题
图片生成模块中存在大量重复代码，两个核心函数 `create_webzine_image` 和 `create_combined_webzine_image` 包含相似的文本解析、字体处理和图片渲染逻辑，导致代码维护困难且容易出错。
## 1.1. **代码重复严重**
`scripts/modules/image_generator.py` 文件中的两个函数 `create_webzine_image`（第13-131行）和 `create_combined_webzine_image`（第134-239行）存在大量重复代码：
* 都包含相同的文本解析逻辑，识别"标题："、"正文："、"价值点："等前缀
* 都使用相同的字体加载和配置代码
* 都包含相似的文本换行和宽度计算逻辑
* 都有重复的图片绘制命令处理
这种重复导致：
* 代码行数冗余，两个函数共约230行，其中至少60%是重复逻辑
* 修改渲染逻辑时需要在两个地方同步修改，容易遗漏
* 增加了代码理解和维护的成本
## 1.2. **缺乏抽象层次**
当前实现没有将共同的渲染逻辑抽象为独立的函数，导致：
* 文本解析逻辑散落在两个函数中，形成"面条代码"
* 字体配置和图片尺寸参数硬编码在多个位置
* 渲染命令的构建逻辑重复，没有统一的处理接口
## 1.3. **错误处理不一致**
两个函数在异常处理上存在细微差异：
* `create_webzine_image` 在PIL未安装时返回None并记录警告
* `create_combined_webzine_image` 有类似的处理但日志信息略有不同
* 缺乏统一的错误处理策略，难以保证行为一致性
# 2. 收益
通过重构图片生成模块，提取公共逻辑并建立清晰的抽象层次，可以显著提升代码质量和开发效率。
## 2.1. **减少代码重复**
重构后可以将重复代码从约140行减少到约30行，代码总量减少约40%。通过提取公共函数，两个核心函数的代码行数都将显著减少，提升代码的简洁性和可读性。
## 2.2. **提升可维护性**
统一的渲染逻辑意味着：
* 修改文本解析规则时只需修改一处
* 调整字体或样式时可以集中配置
* 添加新的渲染特性时可以复用现有抽象
* 降低因修改不同步导致的bug风险
## 2.3. **增强可测试性**
提取后的公共函数可以独立进行单元测试，不再需要依赖完整的图片生成流程。这样可以更容易地验证文本解析、换行逻辑等核心功能的正确性。
## 2.4. **改善代码可读性**
通过合理的函数命名和职责分离，代码的自解释性将显著提升。新的开发者可以更快理解图片生成的流程，降低学习成本。
# 3. 方案
系统性地重构图片生成模块，通过提取公共函数、建立渲染抽象层次，消除代码重复并提升代码质量。
## 3.1. **提取文本解析函数**
将重复的文本前缀识别和内容提取逻辑抽象为独立函数：
```python
def _parse_webzine_line(line):
"""解析网摘单行文本，返回 (前缀类型, 前缀文本, 内容文本)"""
if not line.strip():
return ('empty', '', '')
prefixes = ['标题：', '原标题：', '发布日期：', '正文：', '价值点：']
for prefix in prefixes:
if line.startswith(prefix):
return (prefix.rstrip('：'), prefix, line[len(prefix):])
return ('normal', '', line)
```
这个函数统一处理文本解析逻辑，消除了两个函数中的重复代码。
## 3.2. **提取字体配置函数**
将字体加载和配置逻辑集中管理：
```python
def _load_webzine_fonts():
"""加载网摘图片所需的所有字体"""
return {
'banner': get_chinese_font(size=40, bold=True),
'article_title': get_chinese_font(size=36, bold=True),
'bold': get_chinese_font(size=32, bold=True),
'content': get_chinese_font(size=32, bold=False)
}
```
这样可以确保字体配置的一致性，并便于集中调整样式。
## 3.3. **提取渲染命令构建函数**
将文本换行和渲染命令生成逻辑抽象为独立函数：
```python
def _build_render_commands(webzine_content, fonts, content_width, temp_draw):
"""将网摘文本转换为渲染命令列表"""
commands = []
for line in webzine_content.split('\n'):
line_type, prefix, content = _parse_webzine_line(line)
if line_type == 'empty':
commands.append(('empty',))
elif line_type == '标题':
full_title = prefix + content
wrapped = wrap_text(full_title, fonts['article_title'], content_width, temp_draw)
for wrapped_line in wrapped:
commands.append(('article_title', wrapped_line))
elif line_type in ['原标题', '发布日期', '正文', '价值点']:
test_line = prefix + content
test_bbox = temp_draw.textbbox((0, 0), test_line, font=fonts['content'])
test_width = test_bbox[2] - test_bbox[0]
if test_width <= content_width:
commands.append(('prefix_sameline', prefix, content))
else:
commands.append(('prefix_only', prefix))
wrapped = wrap_text(content, fonts['content'], content_width, temp_draw)
for wrapped_line in wrapped:
commands.append(('content_indented', wrapped_line))
else:
wrapped = wrap_text(content, fonts['content'], content_width, temp_draw)
for wrapped_line in wrapped:
commands.append(('normal', wrapped_line))
return commands
```
这个函数将复杂的文本处理逻辑封装起来，使主函数更加清晰。
## 3.4. **提取渲染执行函数**
将图片绘制的具体执行逻辑抽象为独立函数：
```python
def _execute_render_commands(draw, commands, fonts, padding, line_height, content_width):
"""执行渲染命令列表，在指定画布上绘制内容"""
y = 135
for cmd in commands:
cmd_type = cmd[0]
if cmd_type == 'empty':
y += line_height
elif cmd_type == 'normal':
draw.text((padding, y), cmd[1], fill=(0, 0, 0), font=fonts['content'])
y += line_height
elif cmd_type == 'article_title':
draw.text((padding, y), cmd[1], fill=(0, 0, 0), font=fonts['article_title'])
y += line_height
elif cmd_type == 'prefix_sameline':
label_bbox = draw.textbbox((0, 0), cmd[1], font=fonts['bold'])
label_width = label_bbox[2] - label_bbox[0]
draw.text((padding, y), cmd[1], fill=(0, 0, 0), font=fonts['bold'])
draw.text((padding + label_width, y), cmd[2], fill=(0, 0, 0), font=fonts['content'])
y += line_height
elif cmd_type == 'prefix_only':
draw.text((padding, y), cmd[1], fill=(0, 0, 0), font=fonts['bold'])
y += line_height
elif cmd_type == 'content_indented':
draw.text((padding, y), cmd[1], fill=(0, 0, 0), font=fonts['content'])
y += line_height
elif cmd_type == 'separator':
y += 30
draw.line([(padding, y), (content_width + padding, y)], fill=(200, 200, 200), width=1)
y += 30
return y
```
这样可以将绘制逻辑与业务逻辑分离，提升代码的模块化程度。
## 3.5. **重构后的主函数**
重构后的两个主函数将变得简洁清晰：
```python
def create_webzine_image(webzine_content, output_path, title_text):
"""将网摘文本内容渲染为PNG图片（800px宽，自适应高度）"""
logger = get_logger()
if Image is None:
logger.warning("PIL 未安装，跳过图片生成")
return None
try:
width = 800
padding = 50
content_width = width - padding * 2
line_height = 40
# 解析日期
date_match = re.search(r'\d{4}年\d{2}月\d{2}日', webzine_content)
if date_match:
date_str = date_match.group(0).replace('年', '.').replace('月', '.').replace('日', '')
else:
date_str = datetime.now().strftime('%Y.%m.%d')
banner_text = f'网摘 {date_str}'
# 加载字体
fonts = _load_webzine_fonts()
# 构建渲染命令
temp_img = Image.new('RGB', (100, 100), color=(255, 255, 255))
temp_draw = ImageDraw.Draw(temp_img)
commands = _build_render_commands(webzine_content, fonts, content_width, temp_draw)
# 计算图片高度
actual_lines_count = len([cmd for cmd in commands if cmd[0] != 'separator'])
img_height = 140 + actual_lines_count * line_height + padding * 2
img_height = int(img_height * 1.15)
# 创建图片并绘制
img = Image.new('RGB', (width, img_height), color=(255, 255, 255))
draw = ImageDraw.Draw(img)
# 绘制标题栏
draw.rectangle([(0, 0), (width, 90)], fill=(26, 72, 144))
draw.text((padding, 25), banner_text, fill=(255, 255, 255), font=fonts['banner'])
draw.line([(padding, 100), (width - padding, 100)], fill=(180, 180, 180), width=2)
# 执行渲染命令
_execute_render_commands(draw, commands, fonts, padding, line_height, content_width)
img.save(output_path, 'PNG', quality=95)
logger.info("网摘图片已保存: %s (%d条命令, %dpx)", output_path, len(commands), img_height)
return output_path
except Exception as e:
logger.warning("图片生成失败: %s", e)
return None
```
重构后的函数结构清晰，职责单一，易于理解和维护。
# 4. 回归范围
本次重构主要影响图片生成功能，需要重点测试网摘图片生成的正确性和稳定性。
## 4.1. 主链路
1. **完整日报生成流程**
* 从RSS抓取到最终图片生成的完整流程
* 验证生成的图片格式、尺寸、内容正确性
* 确认图片文件能正常保存到指定路径
2. **网摘图片生成**
* 单篇网摘图片生成功能
* TOP3合并长图生成功能
* 验证图片中的文本内容、格式、样式符合预期
3. **异常情况处理**
* PIL未安装时的降级处理
* 图片生成失败时的错误处理和日志记录
* 确认异常情况下不影响主流程继续执行
## 4.2. 边界情况
1. **特殊文本内容**
* 包含超长标题的文章
* 包含特殊字符或表情符号的文本
* 空内容或格式异常的网摘文本
2. **字体和样式**
* 不同操作系统的字体兼容性
* 中英文混合内容的正确渲染
* 文本换行和边界情况的处理
3. **性能和资源**
* 生成大量图片时的内存使用情况
* 并发生成图片时的线程安全性
* 大文本内容的处理性能
4. **缓存和复用**
* 图片已存在时的跳过逻辑
* 缓存机制与重构后的兼容性
* 增量生成时的正确性
1.md
+57
View File
@@ -0,0 +1,57 @@
# **热点速览 | 2026-06-03**
2026-06-03
<!-- SECTION:summary BEGIN -->
## **📋 今日速览**
- 🔍 **封锁霍尔木兹海峡 · 以色列 · 伊朗** (48.5°) 黎以达成停火共识，以色列撤回部队；美伊谈判重回正轨。
- 📝 **伊朗袭击科威特巴林** (48.4°) 伊朗向科威特和巴林发射弹道导弹和无人机，美军成功拦截。
<!-- SECTION:summary END -->
<!-- SECTION:cross_day BEGIN -->
## **📈 跨日追踪**
### **➡️ 封锁霍尔木兹海峡 · 以色列 · 伊朗**
今日热度 48.5° (-2.7) | 报道 0 篇 (-8) | 趋势 持平
前日 (2026-06-02): 热度 51.2° | 10 篇 | 匹配度 0.15
前情: 前情内容
### **🔺 2027年商业载人飞行**
今日热度 47.2° (+35.0) | 报道 0 篇 (+0) | 趋势 上升
前日 (2026-06-02): 热度 12.2° | 1 篇 | 匹配度 0.06
<!-- SECTION:cross_day END -->
<!-- SECTION:trend BEGIN -->
## **📊 趋势分析**
### **📈 持续 封锁霍尔木兹海峡**
- 热度: 48 ← 52 ← 45
- 报道: 2 ← 10 ← 8 篇
- 时间: 06-03 ← 06-02 ← 06-01
- 近3天热度变化 -8%
<!-- SECTION:trend END -->
<!-- SECTION:insights BEGIN -->
## **🔍 热点洞察**
### **霍尔木兹海峡危机[持续跟踪]**
> 黎以达成停火共识，以色列撤回部队；美伊谈判重回正轨。分析指出霍尔木兹海峡航运受阻将影响南亚化肥供应，可能引发粮食安全危机。
- 黎以达成停火共识，以色列撤回部队
- 美伊谈判重回正轨
- 霍尔木兹海峡航运受阻将影响南亚化肥供应
#### **事件脉络**
前情提要：此前霍尔木兹海峡局势紧张，美伊谈判停滞，黎以冲突持续。
最新突破：黎以达成停火共识，以色列撤回部队；美伊谈判恢复。
#### **多角度分析**
- **技术维度**：海峡航运受阻直接影响南亚化肥供应链
- **战略维度**：黎以停火缓解中东北部紧张
#### **影响与展望**
短期：黎以停火减少地区冲突热点
中期：海峡航运若持续受阻，南亚化肥价格将上涨
20260529_新闻智能分析系统开发计划_v2.md
+822
View File
@@ -0,0 +1,822 @@
# 新闻智能分析系统开发计划 v2.0
> 基于大模型的时政/军事/科技新闻聚类、评价与整合系统
>
> **迭代说明**：参考 AI Daily 项目实践，融合细粒度聚类与工程化部署经验
---
## 一、项目概述
### 1.1 项目目标
构建一个自动化新闻分析系统，实现：
- **细粒度聚类**：保留多角度报道，避免简单去重丢失热点
- **动态热点识别**：基于当日新闻分布统计特征自适应判定热点
- **双层整合**：摘要级速览（全量）+ 洞察级深度分析（Top 10）
- **混合处理**：实时热点感知 + 每日批量汇总生成早报
- **跨日关联**：追踪同一事件的多日发展脉络
### 1.2 核心约束
| 维度 | 约束 |
|------|------|
| 数据规模 | 200-500篇/日 |
| 数据源 | RSS Feed（公众号订阅源） |
| 输出格式 | Web页面 / Markdown / 即时推送 |
| 成本预算 | 混合模型调度，日成本约1.3-1.6元 |
| 处理时效 | 批处理15-20分钟完成 |
| 部署方式 | 支持systemd服务化部署 |
### 1.3 双循环架构（参考AI Daily优化）
```
┌─────────────────────────────────────────────────────────────────┐
│ Fetch 循环（实时） │
│ 每30分钟运行一次 │
│ RSS抓取 → 转为Markdown → LLM评分 → 重要性判断 → 即时推送？ │
│ ↓ ↓ │
│ 存入数据库(fetch-yyyy-mm-dd.json) 飞书/Discord│
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ Push 循环（定时） │
│ 每日定时运行（如早8点） │
│ 读取碎片化信息 → 细粒度聚类 → 多角度关联 → 综合评分 │
│ → 摘要级整合(全量) → 洞察级整合(Top10) → 生成日报 → 推送 │
└─────────────────────────────────────────────────────────────────┘
```
---
## 二、与AI Daily的对比分析
### 2.1 相似性
| 维度 | AI Daily | 本系统 | 相似度 |
|------|----------|--------|--------|
| **数据源** | RSS订阅源 | RSS订阅源（公众号） | ⭐⭐⭐⭐⭐ |
| **核心流程** | RSS→LLM评分→推送/汇总 | RSS→聚类→评分→整合→推送/汇总 | ⭐⭐⭐⭐ |
| **双循环架构** | Fetch循环+Push循环 | 实时检测+批处理 | ⭐⭐⭐⭐⭐ |
| **即时推送** | 飞书/Discord webhook | 飞书/企微/邮件 | ⭐⭐⭐⭐ |
| **定时汇总** | 每日定时推送 | 每日早报生成 | ⭐⭐⭐⭐⭐ |
| **成本预算** | 每天6毛钱 | 每天1.3-1.6元 | ⭐⭐⭐⭐ |
### 2.2 本系统的差异化优势
| 特性 | AI Daily | 本系统 | 价值 |
|------|----------|--------|------|
| **热点识别** | 单篇文章评分 | **双层筛选：硬约束+动态阈值融合** | 质量控制+热点发现协同 |
| **去重策略** | 简单去重 | 细粒度聚类+多角度保留 | 不丢失多角度报道 |
| **内容整合** | 单篇摘要 | 双层整合（摘要+洞察） | 深度分析Top热点 |
| **领域聚焦** | AI领域 | 时政/军事/科技 | 专业领域实体识别 |
| **角度分类** | 无 | 6种报道角度分类 | 支持多角度整合 |
| **跨日关联** | 无 | 事件时间线追踪+Sentinel标记 | 追踪事件发展脉络 |
### 2.3 AI Daily 参考资源
- 项目介绍（热点追踪系统）：https://yeekal.com/ai/ai-daily-news-tracker/
- 项目介绍（进化实录）：https://yeekal.com/ai/ai-daily-news-update/
- GitHub 代码库：https://github.com/YeeKal/ai-daily
### 2.4 从AI Daily借鉴的核心设计
1. **双循环架构**：Fetch循环（实时）+ Push循环（定时）
2. **systemd服务化**：使用systemd timer替代Python内部定时，提升稳定性
3. **文件存储规范**：fetch/notify/push三种文件类型，明确保留策略
4. **评分硬约束**：非目标领域上限、KOL转述上限，有效控制信息质量
5. **Prompt防退化**：禁止套话、要求从素材出发，避免LLM输出风格趋同
6. **双视角洞察**：metadata（事实压缩）+ 正文（趋势判断）解耦
7. **Sentinel分段标记**：单文件多板块精确管理，支持跨日关联
---
## 三、开发阶段规划
### 阶段一：基础架构搭建（Week 1-2）
**目标**：建立数据流和存储基础，确保稳定采集
| 序号 | 任务 | 描述 | 输出 | 依赖 |
|------|------|------|------|------|
| 1.1 | RSS采集模块 | 解析公众号RSS Feed，提取标题、正文、来源、发布时间 | 原始文章数据库 | - |
| 1.2 | 数据标准化 | 统一文章格式，清洗HTML标签，提取纯文本 | 标准化文章表 | 1.1 |
| 1.3 | 文件存储层 | JSON文件存储(fetch-yyyy-mm-dd.json) | 存储规范 | 1.2 |
| 1.4 | 数据库设计 | SQLite设计文章表、实体表、聚类表 | 数据库Schema | 1.2 |
| 1.5 | 基础监控 | 采集成功率、文章数量统计 | 监控面板 | 1.1 |
| 1.6 | 日志系统 | 有效日志记录，便于问题定位 | 日志模块 | 1.1 |
**数据字段规范**：
```json
{
"title": "内容标题",
"link": "原始链接",
"published": "发布时间",
"source": "来源（公众号名称）",
"content": "Markdown格式的正文内容",
"tags": "LLM识别的标签",
"score": "LLM评分（0-100）",
"summary": "LLM生成的中文摘要",
"fetched_at": "抓取时间",
"entities": ["提取的实体列表"],
"reporting_angle": "报道角度"
}
```
**验收标准**：
- [ ] 稳定采集50+ RSS源
- [ ] 数据入库成功率 > 95%
- [ ] 基础监控可查看当日采集概况
- [ ] 日志系统可定位问题
---
### 阶段二：Fetch循环与双层评分系统（Week 3-4）
**目标**：实现实时采集、双层评分（硬约束+动态阈值融合）、即时推送
| 序号 | 任务 | 描述 | 输出 | 依赖 |
|------|------|------|------|------|
| 2.1 | Fetch循环引擎 | 每30分钟循环一次，抓取RSS→Markdown→评分 | fetch引擎 | 1.6 |
| 2.2 | **第一层：单篇评分Prompt** | 单篇文章重要性评分（0-100），**含硬约束** | `prompts/article_score.txt` | 2.1 |
| 2.3 | **硬约束过滤器** | 非时政军事上限、KOL转述上限、时效衰减 | 硬约束模块 | 2.2 |
| 2.4 | 即时推送判断 | 单篇≥90分触发即时推送候选 | 推送判断模块 | 2.3 |
| 2.5 | 飞书Webhook推送 | 飞书群机器人推送 | 飞书推送适配器 | 2.4 |
| 2.6 | systemd服务化 | 使用systemd timer管理服务 | systemd配置 | 2.1 |
#### 双层评分架构设计
```
第一层：单篇文章评分（硬约束） 第二层：实体簇热度（动态阈值）
├── 输入：单篇RSS文章 ├── 输入：同一实体的多篇报道（已硬约束评分）
├── 处理：LLM评分 + 硬约束修正 ├── 处理：统计特征 + 动态阈值计算
└── 输出：0-100分（已约束） └── 输出：热点判定
├── ≤79：非目标领域，过滤 ├── 低于阈值：普通关注
├── 80-89：一般关注，进入聚类 └── 高于阈值：热点，优先整合
└── 90+：高优先级，即时推送候选
```
#### 第一层：评分硬约束设计
**Prompt核心约束**：
```markdown
## 评分规则（硬性约束）
1. **非时政/军事/科技主题上限 79 分**
- 如果内容不属于目标领域（如纯娱乐八卦、生活琐事），最高只能给 79 分
2. **KOL 转述上限 89 分**
- 如果只是 KOL/大V 对已有信息的转述/评论，而非原创信息，最高只能给 89 分
3. **时效性衰减**
- 发布超过24小时：分数×0.9
- 发布超过48小时：分数×0.8
4. **90+ 分必须同时满足**：
- 重大事件/突破性进展/政策发布
- 一手信息源（官方发布、权威媒体首发）
- 对目标领域有实质性影响
5. **标签禁止空泛**
- 禁止：["军事", "新闻", "热点"]
- 要求：["歼-35A", "舰载战斗机", "隐身性能", "海军航空兵"]
```
**代码实现**：
```python
def apply_score_constraints(entry: dict, raw_score: int) -> tuple[int, list[str]]:
"""
应用评分硬约束
返回：修正后的分数，应用的约束标签列表
"""
constraints_applied = []
# 约束1：非目标领域上限79
if not is_target_domain(entry['content']):
raw_score = min(raw_score, 79)
constraints_applied.append("非目标领域")
# 约束2：KOL转述上限89
if is_kol_repost(entry):
raw_score = min(raw_score, 89)
constraints_applied.append("KOL转述")
# 约束3：时效性衰减
hours_old = get_hours_since_published(entry)
if hours_old > 24:
decay_factor = 0.9 ** (hours_old // 24)
raw_score = int(raw_score * decay_factor)
constraints_applied.append(f"时效衰减({hours_old}h)")
return raw_score, constraints_applied
```
**第一层输出分级**：
| 分数段 | 处理策略 | 说明 |
|--------|---------|------|
| ≤79 | 过滤，不入库 | 非目标领域内容 |
| 80-89 | 入库，进入聚类 | 一般关注，参与动态阈值计算 |
| 90+ | 入库，即时推送候选 | 高优先级，同时触发即时推送判断 |
**验收标准**：
- [ ] Fetch循环每30分钟稳定运行
- [ ] 硬约束生效（非目标领域≤79，KOL转述≤89，时效衰减）
- [ ] 90+文章触发即时推送候选
- [ ] systemd服务可一键启动/停止
---
### 阶段三：实体提取与细粒度聚类（Week 5-6）
**目标**：实现细粒度聚类，区分同一实体的不同报道角度
| 序号 | 任务 | 描述 | 输出 | 依赖 |
|------|------|------|------|------|
| 3.1 | 实体提取Prompt | 设计并优化实体提取提示词 | `prompts/extract_entities.txt` | 2.6 |
| 3.2 | 角度分类Prompt | 设计报道角度分类提示词 | `prompts/classify_angle.txt` | 2.6 |
| 3.3 | 批量实体提取 | 实现50篇/批并发调用，小模型处理 | 实体提取服务 | 3.1 |
| 3.4 | 角度分类服务 | 批量角度分类，与实体提取并行 | 角度分类服务 | 3.2 |
| 3.5 | 实体归一化 | 大模型消歧，合并别名 | 实体归一化服务 | 3.3 |
| 3.6 | 聚类算法 | 基于实体+角度的细粒度聚类 | 聚类引擎 | 3.4, 3.5 |
| 3.7 | 记忆系统 | 避免同一信息反复推送 | 去重模块 | 3.6 |
**验收标准**：
- [ ] 实体提取准确率 > 85%
- [ ] 角度分类准确率 > 80%
- [ ] 聚类后实体簇数量合理（200篇→30-50个簇）
- [ ] 同一信息不重复推送
---
### 阶段四：热度评价与动态阈值（Week 7-8）
**目标**：实现第二层筛选——基于硬约束后分数的动态热点识别
| 序号 | 任务 | 描述 | 输出 | 依赖 |
|------|------|------|------|------|
| 4.1 | **实体簇热度计算** | 融合硬约束分数的多维度热度计算 | 热度指标服务 | 3.6 |
| 4.2 | **第二层：动态阈值算法** | 基于硬约束后分数分布的自适应阈值 | 阈值计算模块 | 4.1 |
| 4.3 | 热点识别引擎 | 硬约束过滤 + 动态阈值筛选的双层判定 | 热点识别服务 | 4.2 |
| 4.4 | 热点精评Prompt | 大模型深度评分（Top 30热点簇） | `prompts/precise_heat.txt` | 4.3 |
#### 第二层：动态阈值设计（融合硬约束分数）
**实体簇热度计算**：
```python
class Cluster:
def __init__(self, entity_name: str):
self.entity_name = entity_name
self.articles = [] # 已硬约束评分的文章列表
self.hot_score = 0 # 综合热度分
def calculate_hot_score(self) -> int:
"""
计算实体簇热度，融合硬约束后的单篇分数
"""
if not self.articles:
return 0
# 基础分数：最高分文章的分数（已硬约束）
max_score = max(a['score'] for a in self.articles)
# 传播热度：报道数量 × log(来源多样性)
report_count = len(self.articles)
source_diversity = len(set(a['source'] for a in self.articles))
propagation_heat = report_count * math.log(source_diversity + 1)
# 角度覆盖度：不同报道角度数 / 6
angles = set(a.get('reporting_angle', 'unknown') for a in self.articles)
angle_coverage = len(angles) / 6
# 高优先级文章加成（90+文章额外加权）
high_priority_count = sum(1 for a in self.articles if a['score'] >= 90)
priority_bonus = high_priority_count * 5 # 每篇90+加5分
# 综合计算
self.hot_score = min(100, int(
max_score * 0.4 + # 单篇最高分权重40%
min(propagation_heat * 3, 30) + # 传播热度权重30%
angle_coverage * 20 + # 角度覆盖权重20%
priority_bonus # 高优先级加成10%
))
return self.hot_score
```
**动态阈值算法（融合版）**：
```python
def calculate_hotspot_threshold(clusters: list[Cluster], date: str) -> float:
"""
基于硬约束后分数分布计算动态阈值
同时考虑统计特征和高优先级文章数量
"""
# 获取当日所有簇的热度分数（已融合硬约束）
scores = [c.hot_score for c in clusters]
if len(scores) < 5:
return 75 # 数据不足时使用保底阈值
# 统计特征
mean_score = mean(scores)
std_score = std(scores)
median_score = median(scores)
# 方法1：统计阈值（均值+1.5倍标准差）
threshold_stat = mean_score + 1.5 * std_score
# 方法2：保底阈值（至少3条或前10%）
min_count = max(3, len(scores) * 0.1)
sorted_scores = sorted(scores, reverse=True)
threshold_adaptive = sorted_scores[min_count - 1]
# 方法3：硬约束保底（考虑90+高优先级文章数量）
high_priority_count = sum(
1 for c in clusters
if any(a['score'] >= 90 for a in c.articles)
)
if high_priority_count >= 3:
# 如果有3个以上簇包含90+文章，降低阈值确保热点被识别
threshold_backup = median_score
else:
threshold_backup = mean_score + 0.5 * std_score
# 取三者中较低值，确保热点不被遗漏
final_threshold = min(threshold_stat, threshold_adaptive, threshold_backup)
# 记录阈值计算日志
logger.info(f"动态阈值计算: 统计={threshold_stat:.1f}, "
f"保底={threshold_adaptive:.1f}, 硬约束保底={threshold_backup:.1f}, "
f"最终={final_threshold:.1f}")
return final_threshold
```
**双层热点判定流程**：
```python
def identify_hotspots(clusters: list[Cluster], date: str) -> tuple[list[Cluster], list[Cluster]]:
"""
双层热点识别：硬约束过滤 + 动态阈值筛选
"""
# 第一层：硬约束过滤（单篇层面已处理，此处检查）
valid_clusters = []
for c in clusters:
# 过滤掉所有文章都≤79的簇（非目标领域）
if all(a['score'] <= 79 for a in c.articles):
continue
valid_clusters.append(c)
# 计算各簇热度（融合硬约束分数）
for c in valid_clusters:
c.calculate_hot_score()
# 第二层：计算动态阈值
threshold = calculate_hotspot_threshold(valid_clusters, date)
# 阈值筛选 + 额外条件
hotspots = []
normal = []
for c in valid_clusters:
if c.hot_score >= threshold:
# 额外检查：是否有90+文章 或 超过阈值10分以上
has_high_priority = any(a['score'] >= 90 for a in c.articles)
significantly_above = c.hot_score >= threshold + 10
if has_high_priority or significantly_above:
hotspots.append(c)
else:
normal.append(c)
else:
normal.append(c)
# 排序：先按热度，再按最高单篇分数
hotspots.sort(key=lambda x: (x.hot_score, x.max_article_score), reverse=True)
return hotspots, normal
```
**融合设计优势**：
| 层级 | 作用 | 输入 | 输出 |
|------|------|------|------|
| 第一层（硬约束） | 质量控制 | 单篇文章 | 过滤非目标领域，标记高优先级 |
| 第二层（动态阈值） | 热点发现 | 实体簇（多篇聚合） | 识别突发热点，自适应当日分布 |
**验收标准**：
- [ ] 实体簇热度计算正确（融合硬约束分数）
- [ ] 动态阈值自适应当日分布
- [ ] 双层筛选协同工作（硬约束过滤→动态阈值筛选）
- [ ] 热点识别召回率 > 90%，误报率 < 20%
---
### 阶段五：Push循环与双层整合（Week 9-10）
**目标**：实现定时汇总、双层整合（双视角设计）、日报生成
| 序号 | 任务 | 描述 | 输出 | 依赖 |
|------|------|------|------|------|
| 5.1 | Push循环引擎 | 每日定时运行，读取碎片化信息→整合→推送 | push引擎 | 4.4 |
| 5.2 | 摘要级整合Prompt | 多源报道合并为50-80字精炼摘要 | `prompts/summary_integrate.txt` | 5.1 |
| 5.3 | 洞察级整合Prompt | Top 10生成结构化深度分析（**双视角设计**） | `prompts/insight_integrate.txt` | 5.1 |
| 5.4 | **Prompt防退化模块** | 禁止套话、要求从素材出发 | 防退化检查器 | 5.2, 5.3 |
| 5.5 | 摘要整合服务 | 小模型批量处理全量簇 | 摘要整合服务 | 5.2, 5.4 |
| 5.6 | 洞察整合服务 | 大模型逐个处理Top 10（**双视角输出**） | 洞察整合服务 | 5.3, 5.4 |
| 5.7 | 时间线提取 | 从多源报道中提取事件时间线 | 时间线提取模块 | 5.5 |
| 5.8 | 多角度整合 | 同一事件不同角度报道的整合 | 多角度整合模块 | 5.6 |
#### 双视角洞察设计（借鉴AI Daily）
**设计理念**：metadata（新闻编辑视角，事实压缩）+ 正文（情报分析师视角，趋势判断）解耦
**Prompt设计**：
```markdown
## Part 1: Metadata（新闻编辑视角）
请生成以下结构化信息：
- title: 10字以内的标题（事实陈述，无修饰）
- lead: 20字以内的导语（核心事实）
- highlights: 3-5个关键要点（bullet points，纯事实）
约束：只陈述事实，不做趋势判断，不用形容词。
## Part 2: 洞察正文（情报分析师视角）
基于多源报道，识别：
1. 事件脉络：关键时间节点和进展
2. 多角度分析：技术维度、战略维度、舆论维度
3. 影响与展望：短期影响、中长期趋势、值得关注信号
约束：
- 禁止空泛评论（如"意义重大""影响深远"）
- 每个观点必须有素材支撑
- 使用具体名称（装备型号/政策名称/组织名称），不用"某些""部分"等模糊指代
```
**输出示例**：
```markdown
---
title: "歼-35A列装部队"
lead: "空军官方确认歼-35A已列装首批作战部队"
highlights:
- 首次公开确认进入实战化部署阶段
- 隐身性能对标F-35C，可能在舰载领域形成优势
- 多国媒体关注其对西太平洋军事平衡的影响
---
## 事件脉络
- 2024-11：珠海航展首次公开亮相
- 2025-03：完成舰载适配测试
- 2025-05：空军确认列装首批作战部队【新进展】
## 多角度分析
- **技术维度**：隐身涂层、航电系统、舰载适配均有突破
- **战略维度**：提升海军航空兵远海作战能力
- **舆论维度**：外媒关注中美舰载机代差缩小
## 影响与展望
短期：提升海军航母编队作战能力
中期：可能在西太平洋形成局部优势
值得关注：后续舰载版测试进展、出口动向
```
#### Prompt防退化设计（借鉴AI Daily）
**禁止套话列表**：
```python
FORBIDDEN_PHRASES = [
"意义重大", "影响深远", "引发关注", "备受瞩目",
"深水区", "拐点", "白热化", "新纪元", "里程碑",
"不容忽视", "值得注意", "值得关注", # 除非后接具体内容
"某些", "部分", "一些", "相关", # 模糊指代
]
def check_degeneration(content: str) -> list[str]:
"""检查内容是否包含禁止套话"""
violations = []
for phrase in FORBIDDEN_PHRASES:
if phrase in content:
violations.append(phrase)
return violations
```
**Prompt中的防退化指令**：
```markdown
## 输出约束（防退化）
1. 禁止使用以下套话：
- "意义重大""影响深远""引发关注""备受瞩目"
- "深水区""拐点""白热化""新纪元""里程碑"
2. 每个观点必须有素材支撑：
- 错误：该装备性能先进
- 正确：该装备隐身系数0.01，优于F-35的0.02
3. 使用具体名称，禁止模糊指代：
- 错误：某些国家表示关注
- 正确：日本防卫省发布关注声明
4. 从素材出发，每次措辞应不同，避免模板化输出
```
**验收标准**：
- [ ] 摘要信息完整度 > 90%
- [ ] 洞察分析覆盖双视角（metadata + 正文）
- [ ] 无禁止套话出现
- [ ] 整合内容无明显事实错误
---
### 阶段六：日报生成与跨日关联（Week 11-12）
**目标**：生成结构化日报（含Sentinel标记），实现跨日关联
| 序号 | 任务 | 描述 | 输出 | 依赖 |
|------|------|------|------|------|
| 6.1 | 日报模板设计 | Markdown模板，包含热点、速览、数据概览 | `templates/daily_report.md` | 5.8 |
| 6.2 | **Sentinel分段标记** | 单文件多板块精确管理 | `src/utils/sentinel.py` | 6.1 |
| 6.3 | **跨日关联模块** | 从历史日报提取上下文，追踪事件脉络 | `src/processors/cross_day.py` | 6.2 |
| 6.4 | 排序算法 | 综合热度、时效、重要性的排序逻辑 | 排序模块 | 4.4 |
| 6.5 | 日报渲染服务 | 按模板组装数据，生成Markdown（含Sentinel） | 日报渲染服务 | 6.1, 6.2, 6.4 |
| 6.6 | Web归档站点 | 部署在线阅览站点 | Web站点 | 6.5 |
| 6.7 | 数据保留策略 | fetch保留2天，notify保留2天，push保留5天 | 清理策略 | 6.6 |
#### 简化版Sentinel分段标记设计
**标记格式**：
```markdown
---
title: "每日情报早报"
date: "2026-05-24"
stats:
total_entities: 87
hotspot_count: 10
---
<!-- SECTION:summary BEGIN -->
## 📋 全量新闻速览
| # | 实体 | 摘要 | 来源数 | 热度 |
|---|------|------|--------|------|
| 1 | 歼-35A | 空军确认列装\|隐身性能对标F-35C | 8 | ★★★★★ |
<!-- SECTION:summary END -->
<!-- SECTION:insights BEGIN -->
## 🔥 热点深度洞察
### 1. 歼-35A列装进展 [持续跟踪]
> 速览：空军官方确认歼-35A已列装首批作战部队 [8源报道]
**事件脉络**
- 5月20日：首次公开亮相
- 5月24日：【新进展】确认列装部队
...
<!-- SECTION:insights END -->
```
**核心代码**：
```python
import re
from datetime import datetime, timedelta
def extract_section(content: str, section_name: str) -> str:
"""从日报内容中提取指定section"""
pattern = rf'<!-- SECTION:{re.escape(section_name)} BEGIN -->(.*?)<!-- SECTION:{re.escape(section_name)} END -->'
match = re.search(pattern, content, re.DOTALL | re.IGNORECASE)
return match.group(1).strip() if match else ""
def get_historical_context(entity_name: str, days: int = 7) -> dict:
"""获取某实体近N天的历史报道上下文"""
context = {
"first_seen": None,
"previous_summaries": [],
"previous_insights": [],
"mention_count": 0
}
for i in range(days):
date = (datetime.now() - timedelta(days=i)).strftime("%Y-%m-%d")
file_path = f"data/reports/daily-{date}.md"
try:
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
if entity_name in content:
context["mention_count"] += 1
if context["first_seen"] is None:
context["first_seen"] = date
# 提取该实体在摘要中的记录
summary_section = extract_section(content, "summary")
for line in summary_section.split('\n'):
if entity_name in line and line.strip().startswith('|'):
context["previous_summaries"].append({
"date": date,
"content": line.strip()
})
# 提取该实体在洞察中的记录
insights_section = extract_section(content, "insights")
if entity_name in insights_section:
insight_blocks = re.split(r'### \d+\.', insights_section)
for block in insight_blocks:
if entity_name in block:
context["previous_insights"].append({
"date": date,
"content": block.strip()[:500]
})
break
except FileNotFoundError:
continue
return context
def generate_cross_day_marker(entity_name: str) -> str:
"""生成跨日标记"""
history = get_historical_context(entity_name, days=7)
if history["mention_count"] == 0:
return "首次报道"
elif history["mention_count"] == 1:
return "持续跟踪"
else:
return f"持续跟踪({history['mention_count']}次)"
```
**跨日关联应用**：
```python
# 在生成洞察时，添加跨日时间线
history = get_historical_context("歼-35A", days=7)
# 输出：
# {
# "first_seen": "2026-05-20",
# "mention_count": 3,
# "previous_summaries": [
# {"date": "2026-05-20", "content": "| 1 | 歼-35A | 首次公开亮相... |"},
# {"date": "2026-05-22", "content": "| 3 | 歼-35A | 专家分析隐身性能... |"}
# ]
# }
# 在洞察中展示时间线
timeline = f"""
**事件脉络**
- 5月20日：首次公开亮相（珠海航展）
- 5月22日：专家分析隐身性能，推测可能上舰
- 5月24日：【新进展】空军官方确认列装首批作战部队
"""
```
**验收标准**：
- [ ] 日报生成时间 < 20分钟
- [ ] Sentinel标记正确，可精确提取summary/insights
- [ ] 跨日关联可追踪实体历史报道
- [ ] Web站点可正常访问
---
### 阶段七：优化与迭代（Week 13-14）
**目标**：持续优化，功能扩展
| 序号 | 任务 | 描述 | 优先级 |
|------|------|------|--------|
| 7.1 | Prompt优化 | 基于实际效果迭代优化提示词 | 高 |
| 7.2 | 反馈闭环 | 收集用户反馈，标注数据，优化模型 | 高 |
| 7.3 | 趋势分析 | 7日热点趋势图，周期性报告 | 中 |
| 7.4 | 个性化推荐 | 基于用户阅读历史的个性化排序 | 低 |
---
## 四、技术栈建议
### 4.1 核心组件
| 组件 | 推荐方案 | 说明 |
|------|----------|------|
| 数据采集 | Python + feedparser | RSS解析 |
| 数据存储 | SQLite + JSON文件 | 数据库+文件双存储 |
| LLM调用 | OpenAI SDK / 国产模型SDK | 统一接口封装 |
| 任务调度 | systemd timer | 系统级服务管理 |
| Web展示 | Hugo / Flask | 静态/动态站点 |
| 日志 | Python logging + systemd journal | 集中日志管理 |
### 4.2 模型选择
| 任务类型 | 推荐模型 | 预估成本 |
|----------|----------|----------|
| 实体提取 | DeepSeek-V3 / Qwen-Plus | ~0.001元/次 |
| 角度分类 | DeepSeek-V3 / Qwen-Plus | ~0.001元/次 |
| 单篇评分 | DeepSeek-V3 / Qwen-Plus | ~0.001元/次 |
| 摘要整合 | DeepSeek-V3 / Qwen-Plus | ~0.002元/次 |
| 实体消歧 | GPT-4o / Claude-3.5-Sonnet | ~0.01元/次 |
| 热点精评 | GPT-4o / Claude-3.5-Sonnet | ~0.02元/次 |
| 洞察整合 | GPT-4o / Claude-3.5-Sonnet | ~0.1元/次 |
**日成本估算**：
- 200-500篇/日 × 0.001元（小模型）= 0.2-0.5元
- 100个簇 × 0.001元（小模型）= 0.1元
- 10个洞察 × 0.1元（大模型）= 1元
- **总计：约1.3-1.6元/日**
---
## 五、项目结构
```
news-intelligence-system/
├── config/
│ ├── rss_sources.json # RSS源配置
│ ├── models.yaml # 模型配置
│ └── systemd/ # systemd服务配置
├── src/
│ ├── collectors/ # 数据采集
│ │ └── rss_collector.py
│ ├── processors/ # 处理管道
│ │ ├── entity_extractor.py
│ │ ├── angle_classifier.py
│ │ ├── cluster_engine.py
│ │ ├── heat_calculator.py
│ │ └── cross_day.py # 跨日关联（新增）
│ ├── integrators/ # 内容整合
│ │ ├── summary_integrator.py
│ │ └── insight_integrator.py
│ ├── generators/ # 报告生成
│ │ ├── daily_report_generator.py
│ │ └── web_renderer.py
│ ├── push/ # 推送平台
│ │ ├── base.py
│ │ ├── feishu.py
│ │ └── discord.py
│ └── utils/ # 工具函数
│ ├── sentinel.py # Sentinel分段标记（新增）
│ └── prompt_checker.py # Prompt防退化检查（新增）
├── prompts/ # 提示词文件
│ ├── article_score.txt # 单篇评分（含硬约束）
│ ├── extract_entities.txt
│ ├── classify_angle.txt
│ ├── quick_heat.txt
│ ├── precise_heat.txt
│ ├── summary_integrate.txt
│ └── insight_integrate.txt # 双视角洞察
├── templates/
│ └── daily_report.md # 含Sentinel标记
├── data/
│ ├── fetch/ # fetch-yyyy-mm-dd.json
│ ├── notify/ # notify-yyyy-mm-dd.json
│ └── reports/ # daily-yyyy-mm-dd.md
├── web/ # Web归档站点
├── tests/
├── scripts/
│ ├── fetch_loop.py
│ ├── push_loop.py
│ └── setup_systemd.sh
├── requirements.txt
└── README.md
```
---
## 六、关键风险与应对
| 风险 | 影响 | 应对措施 |
|------|------|----------|
| RSS源失效 | 数据缺失 | 多源备份，监控告警 |
| LLM API限流 | 处理延迟 | 批量处理，重试机制，降级策略 |
| 实体识别错误 | 聚类偏差 | 人工标注反馈，Prompt迭代 |
| 热点漏识别 | 信息缺失 | 动态阈值保底机制，人工复核 |
| 成本超支 | 预算超支 | Token监控，模型降级，配额控制 |
| 进程崩溃 | 服务中断 | systemd自动重启 |
| LLM输出退化 | 质量下降 | **Prompt防退化检查** |
---
## 七、里程碑与交付物
| 里程碑 | 时间 | 交付物 |
|--------|------|--------|
| M1 | Week 2 | 稳定运行的数据采集系统 |
| M2 | Week 4 | Fetch循环上线，**评分硬约束生效** |
| M3 | Week 6 | 细粒度聚类引擎，实体提取准确率>85% |
| M4 | Week 8 | 动态热点识别系统，召回率>90% |
| M5 | Week 10 | Push循环上线，**双视角洞察+Prompt防退化** |
| M6 | Week 12 | **Sentinel跨日关联上线**，Web归档站点 |
| M7 | Week 14 | 持续优化迭代 |
---
## 八、借鉴AI Daily的核心设计总结
### 8.1 架构层面
1. **双循环架构**：Fetch循环（实时）+ Push循环（定时）
2. **systemd服务化**：替代Python内部定时，提升稳定性
3. **文件存储规范**：fetch/notify/reports三种文件类型，明确保留策略
### 8.2 LLM应用层面
1. **评分硬约束**：非目标领域≤79、KOL转述≤89，有效控制信息质量
2. **Prompt防退化**：禁止套话、要求从素材出发，避免LLM输出风格趋同
3. **双视角洞察**：metadata（事实压缩）+ 正文（趋势判断）解耦
### 8.3 数据管理层面
1. **Sentinel分段标记**：单文件多板块精确管理，支持跨日关联
2. **跨日关联**：从历史日报提取上下文，追踪事件发展脉络
3. **数据保留策略**：自动清理过期文件
### 8.4 工程层面
1. **日志集中管理**：systemd journal便于问题定位
2. **多平台推送**：飞书、Discord webhook支持
3. **成本可控**：每天约1.3-1.6元
---
*文档版本：v2.1*
*最后更新：2026-05-28*
*状态：已融合AI Daily四大核心设计（Sentinel、硬约束、防退化、双视角）*
20260529_新闻智能分析系统开发计划_v3.md
+1249
View File
File diff suppressed because it is too large Load Diff
Dockerfile
+33
View File
@@ -0,0 +1,33 @@
FROM python:3.11-slim
WORKDIR /app
# 安装系统依赖和中文字体
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-wqy-microhei \
fonts-wqy-zenhei \
tzdata \
&& rm -rf /var/lib/apt/lists/*
# 设置时区
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
# 安装Python依赖
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# 复制代码
COPY scripts/ ./scripts/
COPY config/ ./config/
# 创建输出目录
RUN mkdir -p /app/output /app/logs
# 设置环境变量
ENV PYTHONUNBUFFERED=1
ENV LOG_DIR=/app/logs
ENV OUTPUT_DIR=/app/output
# 入口命令
CMD ["python", "scripts/military_daily_report_v3.py"]
Makefile
+34
View File
@@ -0,0 +1,34 @@
.PHONY: help build run run-once logs clean install-cron local
help:
@echo "🪖 军事科技每日摘要系统 V3"
@echo "================================"
@echo "可用命令："
@echo " make build - 构建 Docker 镜像"
@echo " make run-once - 执行一次（Docker）"
@echo " make local - 本地直接执行"
@echo " make logs - 查看日志"
@echo " make clean - 清理"
@echo " make install-cron - 安装宿主机定时任务"
build:
docker-compose build
run-once:
docker-compose run --rm military-digest
local:
python3 scripts/military_daily_report_v3.py
logs:
tail -f logs/daily_report.log 2>/dev/null || echo "暂无日志"
clean:
docker-compose down -v
rm -rf output/* logs/*
install-cron:
@echo "正在安装定时任务（每日 8:00 执行）..."
(crontab -l 2>/dev/null | grep -v "military_digest"; echo "0 8 * * * cd $(shell pwd) && python3 scripts/military_daily_report_v3.py >> logs/cron.log 2>&1") | crontab -
@echo "✅ 定时任务已安装"
@crontab -l | grep "military_digest"
README.md
+286
View File
@@ -0,0 +1,286 @@
# 🪖 军事科技每日摘要系统 V3
完整的军事科技 RSS 抓取 + AI 智能分析 + 增量缓存 + 运行监控系统
## ✨ 完整功能
### 📥 数据抓取
- 🔗 可配置 RSS 源直抓（rss_feeds.txt 管理，支持 100+ 源）
- ⏰ 24 小时时间窗口过滤
- 🔄 同源标题去重（保留跨源对同一事件的不同报道）
- ⚡ ThreadPoolExecutor 并发抓取，内置失败重试
### 🗄️ 增量缓存
- 💾 SQLite 本地缓存：文章翻译/AI分析结果持久化
- 📰 网摘文本缓存：再次运行零 token 消耗
- 🏷️ 分类介绍缓存：按日期+分类独立存储
- 🧹 自动过期清理（7天）+ 表结构自动迁移
### 🌐 智能翻译
- 🔍 自动识别外文文章
- 📝 分步翻译：先标题 → 再正文（仅通过筛选的文章）
- ⚡ 并发翻译（ThreadPoolExecutor）
### 🔑 关键词筛选
- 🎯 44 个军事科技专业关键词库
- 📊 命中数排序，文章 >15 篇时自动过滤
### 🤖 AI 智能分析
- ⭐ 多维度评分（技术35% + 战略30% + 信息20% + 时效15%）
- 📋 自动分类：今日必看 / 装备动态 / 地区冲突 / 战略政策
- ✍️ 每篇文章 AI 智能摘要（80-120字）
- 🔮 分类深度洞察分析（每类50字综述）
- 🏆 TOP3 必看深度分析
- ⚡ 批量并发评分 + 分类
### 🖼️ 图片生成
- 📄 3800px 超长网摘图片
- 🎨 军事主题视觉设计
- 📊 包含TOP3完整内容
- 💾 自动保存PNG文件
### 📊 运行监控
- ⏱ 各阶段耗时统计，自动输出结构化报表
- 🤖 API 调用量/token 按用途和模型分组统计
- 📰 源抓取成功/失败率一目了然
- 📡 文章统计 + 缓存命中率
## 🚀 快速开始
### 方式一：本地直接运行（推荐）
```bash
# 1. 安装依赖
pip install -r requirements.txt
# 2. 配置环境
cp config/.env.example config/.env
vim config/.env
# 编辑 OPENAI_API_KEY
# 3. 运行
make local
```
### 方式二：Docker 部署
```bash
# 1. 克隆或解压
cd military-digest-v3
# 2. 配置环境
cp config/.env.example config/.env
vim config/.env
# 编辑 OPENAI_API_KEY
# 3. 构建镜像
make build
# 4. 测试运行
make run-once
# 5. 安装定时任务
make install-cron
```
## 📁 目录结构
```
military-digest-v3/
├── scripts/
│ ├── military_daily_report_v3.py # 主入口 — 流程编排
│ └── modules/ # 功能模块包
│ ├── __init__.py # 包初始化 + 统一导出
│ ├── config.py # 配置模块：常量 + AI配置加载
│ ├── utils.py # 工具函数：语言检测/HTML清洗/AI调用/换行
│ ├── rss_fetcher.py # RSS抓取：12源 × 6线程并发
│ ├── translator.py # 翻译处理：中英分离 + 并发翻译
│ ├── keyword_filter.py # 关键词筛选：49词命中排序
│ │ ├── ai_analyzer.py # AI分析：评分/分类/摘要/洞察/网摘生成
│ │ ├── cache.py # SQLite缓存：文章/网摘/分类介绍增量缓存
│ │ ├── monitor.py # 运行监控：阶段耗时/API/AI token调用统计
│ │ ├── pusher.py # 推送模块：企业微信/飞书 webhook 推送
│ │ ├── image_generator.py # 图片生成：TOP3合并3800px长图
│ └── get_chinese_font.py # 字体模块：跨系统中文字体适配
├── config/
│ ├── config.json # 系统配置
│ └── .env.example # 环境变量示例
├── output/ # 输出目录
│ ├── military_report_YYYYMMDD.md # 完整报告（Markdown格式，支持链接、样式）
│ ├── military_webzine_YYYYMMDD.txt # TOP3网摘文本
│ └── military_webzine_YYYYMMDD.png # TOP3网摘合并长图
├── logs/ # 日志目录
├── Dockerfile
├── docker-compose.yml
├── Makefile
└── README.md
```
## 🧩 模块架构
```
military_daily_report_v3.py ← 主入口（流程编排）
├── config.py ← 配置常量，零依赖
├── logger.py ← 统一日志模块
├── cache.py ← SQLite增量缓存，自动迁移
├── monitor.py ← 运行监控统计
├── pusher.py ← 企业微信/飞书推送
├── rss_fetcher.py ← 依赖 config + utils + monitor
├── translator.py ← 依赖 utils
├── keyword_filter.py ← 依赖 config
├── ai_analyzer.py ← 依赖 config + utils
├── image_generator.py ← 依赖 utils + get_chinese_font
└── get_chinese_font.py ← 零依赖
```
### 各模块职责
| 模块 | 行数 | 职责 | 对外导出 |
|------|------|------|----------|
| `config.py` | 90 | RSS源、关键词库、评分维度、分类定义、AI配置加载 | `RSS_FEEDS`, `WECHAT_SOURCES`, `KEYWORDS`, `SCORE_DIMENSIONS`, `CATEGORIES`, `load_ai_config()` |
| `utils.py` | 130+ | 语言检测、HTML清理、时间解析、AI API封装（含监控钩子）、文本换行 | `is_chinese()`, `clean_html()`, `extract_content()`, `parse_pub_time()`, `call_ai()`, `wrap_text()` |
| `rss_fetcher.py` | 100+ | 多线程并行抓取RSS源（MAX_THREADS并发，含2次重试+监控上报） | `fetch_all_feeds()` |
| `translator.py` | 150+ | 中英文分离 + 标题/正文分步AI并发翻译 | `batch_translate_titles()`, `batch_translate_contents()` |
| `keyword_filter.py` | 23 | 44关键词命中计数 → 排序 → Top15筛选 | `keyword_filter()` |
| `ai_analyzer.py` | 240+ | 四维度评分、智能分类、中文摘要、分类洞察、《参考消息》风格网摘生成 | `score_and_classify_article()`, `calculate_weighted_score()`, `generate_category_overview()`, `generate_webzine_for_article()`, `batch_score_and_classify_articles()` |
| `cache.py` | 310+ | SQLite增量缓存：文章处理结果+网摘+分类介绍，支持自动迁移+过期清理+断点续跑 | `get_cache()`, `get_article()`, `save_article()`, `save_article_webzine()`, `get_category_summary()`, `save_category_summary()` |
| `monitor.py` | 160+ | 运行时监控：阶段耗时/AI API调用量+token/源成功率/缓存命中率，结束自动输出报表 | `get_monitor()`, `monitor.stage()`, `monitor.report()` |
| `pusher.py` | 130+ | 企业微信文本/图片/Markdown推送 + 飞书文本推送，主流程末自动触发，失败优雅降级 | `send_daily_push()`, `push_wechat_text()`, `push_wechat_image()`, `push_feishu_text()` |
| `image_generator.py` | 220 | 网摘文本渲染为PNG图片（单篇 + TOP3合并长图） | `create_webzine_image()`, `create_combined_webzine_image()` |
| `get_chinese_font.py` | 50 | 跨系统中文字体自动匹配，支持Windows/Linux/macOS，内置开源字体可选 | `get_chinese_font(size, bold)` |
## ⚙️ 配置说明
### 环境变量 (.env)
| 配置项 | 说明 | 默认值 |
|--------|------|--------|
| OPENAI_API_KEY | OpenAI API 密钥 | - |
| OPENAI_API_BASE | API 地址 | https://api.openai.com/v1 |
| OPENAI_MODEL | AI 分析模型 | gpt-4o-mini |
| WEBZINE_MODEL | 网摘专用模型 | deepseek-v3.2-250518 |
### config.json 配置项
| 配置项 | 说明 | 默认值 |
|--------|------|--------|
| time_window_hours | 时间窗口（小时） | 12 |
| max_articles | 最多保留文章数 | 15 |
| top3_count | TOP3 必看文章数 | 3 |
| wechat_target | 微信接收者ID | - |
| wechat_account | 微信机器人账号 | - |
| enable_wechat_push | 开启微信推送 | true |
| enable_feishu_push | 开启飞书推送 | false |
| enable_image_gen | 开启图片生成 | true |
| enable_txt_gen | 开启TXT文件生成 | true |
### rss_feeds.txt 订阅源配置
所有RSS源都在`config/rss_feeds.txt`文件中管理，不需要修改代码即可增删订阅源：
- 格式：每行 `源名称|RSS地址`
- 以`#`开头的行是注释，会自动忽略
- 空行会自动跳过
- 修改后直接运行程序，无需重启或改代码
- ✅ 容错保障：文件不存在、为空或格式错误时，自动回退使用内置的12个默认源，不会影响程序运行
- 支持所有标准RSS/Atom格式，中英文源无需区分，系统自动识别翻译
## ⏰ 定时任务
### 方式一：宿主机 crontab
```bash
make install-cron
# 或手动添加
crontab -e
# 加入：0 8 * * * cd /path/to/military-digest-v3 && python3 scripts/military_daily_report_v3.py >> logs/cron.log 2>&1
```
### 方式二：Docker Cron
```bash
docker-compose --profile cron up -d
```
## 🔍 RSS 源列表
1. Defense One - 美国防务新闻
2. NEWUAS - 无人机系统新闻
3. Seapower - 海军力量杂志
4. The War Zone - 战区军事分析
5. 国防科技要闻
6. 战略前沿技术
7. 无人机邦
8. 浮空飞行器
9. 海鹰资讯
10. 渊亭防务
11. 电波之矛
12. 龙牙的一座山
## 📊 输出示例
```
🪖 军事科技每日摘要
📅 2026年05月12日 | 共收录 12 篇
====================
🔥 今日必看 TOP3
====================
1.【美军MQ-9无人机测试发射激光制导火箭弹】
⭐ 综合评分：9.2/10
📰 来源：The War Zone
📝 摘要：美国通用原子公司联合美空军在内华达测试场完成...
🔮 深度洞察：低成本改装令察打无人机获得反无人机能力...
🔗 阅读原文：...
...
```
## 🔄 数据流程
```
main()
├── Step 1: RSS 并行抓取（ThreadPoolExecutor × MAX_THREADS 并发，内置重试）
├── Step 2: 24小时时间窗口过滤
├── Step 3: 同源标题去重
├── Step 4: SQLite 缓存查询（分离已处理/待处理文章）
├── Step 5: 外文标题并发翻译（ThreadPoolExecutor × AI_CONCURRENCY）
├── Step 6: 关键词筛选（仅当 >15篇）
├── Step 7: 外文正文并发翻译（仅筛选通过的文章，大幅省 token）
├── Step 8: 批量 AI 评分 + 分类 + 摘要（并发）
├── Step 9: 新处理文章写入 SQLite 缓存
├── Step 10: 合并缓存+新文章，按评分降序
├── Step 11: TOP3 网摘生成（优先缓存命中，支持字数重试，TXT+PNG）
├── Step 12: 四分类整理 + 深度洞察生成（优先缓存命中）
├── Step 13: Markdown 报告输出（全缓存+文件已存在则跳过）
├── Step 14: 多渠道推送（企业微信/飞书 webhook，失败优雅降级）
└── 监控报告输出（阶段耗时/API+token统计/源成功率/缓存命中率）
```
## ✅ 系统要求
- Python 3.8+ 或 Docker
- OpenAI API Key（或兼容API，如火山引擎、DeepSeek等）
- 内存 ≥ 512MB
- 磁盘 ≥ 1GB
## ❓ 常见问题
### 生成的图片中文字体乱码怎么办？
无需额外配置，代码已自动适配所有主流系统的自带中文字体：
- Windows: 自动匹配宋体/黑体/微软雅黑
- Linux: 自动匹配NotoSansCJK/文泉驿微米黑
- macOS: 自动匹配PingFang/黑体
如果需要完全不依赖系统字体，可以下载开源无版权的NotoSansCJK字体放到`scripts/modules/assets/fonts/`目录下：
- 粗体：`NotoSansCJK-Bold.ttc`
- 常规：`NotoSansCJK-Regular.ttc`
### 日志输出在哪里？
日志默认输出到标准错误流，包含INFO/WARNING/ERROR三个级别，运行时会实时打印到控制台，也可以重定向到日志文件：
```bash
python scripts/military_daily_report_v3.py >> logs/runtime.log 2>&1
```
---
*V3 版本 - 全流程重构 + 模块化拆分 + 跨系统字体适配 + 日志标准化 + Markdown报告输出，2026年5月发布*
ai-daily-main.zip
BIN
View File
Binary file not shown.
ai-daily-main/.env.example
+9
View File
@@ -0,0 +1,9 @@
# LLM API 配置
DEEPSEEK_API_KEY=sk-or-v1-...
# Feishu Webhook
FEISHU_WEBHOOK_URL=https://open.feishu.cn/open-apis/bot/v2/hook/...
# 可选：添加 token 的免费限额更高，不加也基本够用
# GITHUB_TOKEN=
# JINA_API_KEY=
ai-daily-main/.gitignore
+42
View File
@@ -0,0 +1,42 @@
# 敏感文件
.env
!.env.example
# 本地配置（使用时从 config.json.example 拷贝并修改）
config.json
!config.json.example
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
*.egg-info/
dist/
build/
.venv
# 测试
.pytest_cache/
.ruff_cache/
.coverage
htmlcov/
# IDE
.idea/
.vscode/
*.swp
*.swo
# 数据文件
news-data/
*.log
# Claude/其他代理数据
.omc/
.claude/
# Git worktrees
.worktrees/
ai-daily-main/CLAUDE.md
+28
View File
@@ -0,0 +1,28 @@
# AI 每日资讯推送系统
AI 驱动的 RSS 新闻聚合与推送系统，支持 400+ 信息源，使用 LLM 智能评分筛选，定时推送到 Discord/企业微信。
当前阶段：MVP 已完成，支持 RSS 抓取、LLM 评分、定时推送、即时推送。
## 技术栈
- 语言：Python 3.10+
- 框架：asyncio
- 依赖：feedparser, aiohttp, croniter, openai
- 构建：pip
- 测试：pytest
## 开发规则
1. 任何代码改动如果与 docs/ 下的文档不一致，必须同步更新对应文档
2. 产品决策变更（功能取舍、交互调整、设计修改）和任务进度 写入 docs/plan.md 的`## 技术决策记录` 和 `## 开发进度`
3. 不确定的产品问题先问用户，不要自行决定
4. 敏感信息（API Keys、Webhook URLs）通过环境变量管理，不硬编码
5. config 有更新需要即时更新对应文档
## 文档索引
- 技术架构 → [docs/tech-spec.md](docs/tech-spec.md)
- 开发计划 → [docs/plan.md](docs/plan.md)
- 早报扩展板块设计（GitHub / HN / 洞察）→ [docs/extra-sections-design.md](docs/extra-sections-design.md)
- 实施计划归档 → [docs/superpowers/plans/](docs/superpowers/plans/)
ai-daily-main/LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025-present
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
ai-daily-main/README.md
+598
View File
@@ -0,0 +1,598 @@
<h1 align="center">AI Daily</h1>
<p align="center"><i>筛选值得关注的 AI 信号</i></p>
<p align="center">
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="License: MIT" /></a>
<img src="https://img.shields.io/badge/python-3.12%2B-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.12+" />
<img src="https://img.shields.io/badge/uv-managed-DE5FE9?style=flat-square&logo=uv&logoColor=white" alt="uv managed" />
<img src="https://img.shields.io/badge/RSS-400%2B%20sources-FFA500?style=flat-square&logo=rss&logoColor=white" alt="RSS 400+" />
<img src="https://img.shields.io/badge/deploy-systemd-orange?style=flat-square&logo=linux&logoColor=white" alt="systemd" />
</p>
[![AI Daily Banner](https://cdn.yeekal.com/yee/visuals/ai-daily-cover.webp)](https://yeekal.com/daily/)
<p align="center">AI 驱动的资讯聚合与推送系统｜<b>RSS · GitHub Trending · Hacker News</b> 三大板块｜LLM 智能评分｜推送到 Discord / 飞书</p>
---
## 核心特性
- 🗞️ **三大内容板块** —— RSS 资讯（400+ 源）+ GitHub Trending + Hacker News 热帖，从媒体、开源、社区三个维度立体捕获 AI 动态
- 🧠 **LLM 智能筛选** —— 评分过滤 + 跨日去重，只留下值得读的
- ⚡ **即时推送** —— 热点新闻（≥90 分）实时触达，重大发布不错过
- 📬 **每日汇总** —— 定时早晚报，早报叠加跨板块洞察段
- 🔌 **多平台推送** —— 飞书、Discord 开箱即用，可扩展自定义平台
- 🛠️ **零运维部署** —— systemd timer 一键安装，开机自启、故障重启
---
## 三大内容板块
### 📰 RSS 资讯聚合
聚合全球主流 AI 媒体、博客、Twitter 账号，默认 OPML 包含约 420 个优质源（源自 [BestBlogs](https://github.com/ginobefun/BestBlogs)）。LLM 逐条评分，只保留高质量内容。
- 默认 60 分钟轮询，异步并发抓取
- 评分维度：相关度、信息密度、时效性
- 跨日上下文去重，同一事件不重复推送
### ⭐ GitHub Trending
抓取 GitHub trending 页面，LLM 深读 README / topics / metadata，从昙花一现的玩具仓库里挑出真正值得关注的项目。
- 从 top 10 候选中精选 3 个（可配置）
- 输出 deep-dive 摘要，附技术亮点与上手建议
- 历史去重索引，推过的项目不再出现
### 💬 Hacker News 热帖
跟踪 HN 首页 AI 相关讨论，整合**外链正文 + 顶层评论树**，产出"内容总结 + 社区观点"双段式摘要。
- 轻量 LLM 从首页 30 条中选出最值得读的故事
- 抓取 L1 顶层评论 + L2 关键回复，字符预算受控
- 外链通过 Jina Reader 拉取 markdown 正文
## 系统架构
```mermaid
flowchart TB
subgraph Sources["📥 数据源"]
RSS["RSS Feeds<br/>400+ sources"]
GH["GitHub Trending"]
HN["Hacker News<br/>Front Page"]
end
subgraph Fetch["⚙️ Fetch 阶段"]
F1["RSS Fetcher<br/>asyncio + feedparser"]
F2["GH Scraper<br/>README deep-dive"]
F3["HN Crawler<br/>Algolia + Jina Reader"]
end
subgraph LLMStage["🧠 LLM 评分与摘要"]
Score["score / score_batch"]
Digest["digest / immediate_push"]
Insight["跨板块 insights"]
end
subgraph Store["💾 存储"]
Files["news-data/<br/>fetch-*.json<br/>push-*.md"]
end
subgraph PushStage["📤 推送渠道"]
Discord["Discord Webhook"]
Feishu["飞书 Webhook"]
end
RSS --> F1 --> Score
GH --> F2 --> Score
HN --> F3 --> Score
Score --> Digest
Score --> Insight
Digest --> Files
Insight --> Files
Files --> Discord
Files --> Feishu
```
**调度说明**
- `dnews-fetch.service`（默认每 60 分钟）：抓取 RSS → 评分 → 命中 ≥90 分立即推送
- `dnews-push.service`（按 `push_cron`）：生成 digest 推送；当天最早一次额外触发 GitHub / HN / 跨板块洞察
## 快速开始
### 环境要求
- Python 3.12+
- [uv](https://docs.astral.sh/uv/) (Python 包管理器)
- Linux + systemd（推荐使用一键部署）
### 1. 配置环境变量
在项目根目录创建 `.env` 文件，添加以下配置：
```bash
# LLM API（ OpenAI API 兼容接口）
DEEPSEEK_API_KEY=your_api_key_here
# 飞书 Webhook
FEISHU_WEBHOOK_URL=your_feishu_webhook_url_here
```
从模板拷贝一份本地配置（`config.json` 已加入 `.gitignore`，不会提交）：
```bash
cp config.json.example config.json
```
然后在 `config.json` 中修改 `llm` 和 `push`
```json
{
"llm": {
"provider": "<whatever>", # openai compatiable
"model": "<model id>",
"baseUrl": "<base url>",
"apiKeyName": "DEEPSEEK_API_KEY", #your api key name in .env>
},
"push": {
"feishu": {
"enabled": true,
"apiKeyName": "FEISHU_WEBHOOK_URL"
}
},
}
```
**获取飞书群机器人 Webhook**
详细教程参考：[我用 RSS + LLM 搭建了一个 AI 热点追踪系统](https://yeekal.com//ai/ai-daily-news-tracker)
1. 群组右上角设置 → 群机器人 → 添加机器人
2. 选择 自定义机器人 → 点击 添加 → 复制 Webhook URL
**获取 Discord Webhook：**
1. 进入 Discord 服务器设置 →
2. 整合 → Webhooks
2. 创建新 Webhook，复制 URL
### 2. 一键部署（推荐）
将程序作为 systemd timer 部署，由系统负责定时触发、故障重启、开机自启。
```bash
./scripts/install.sh
```
脚本会自动同步依赖、安装 systemd 服务并按 `config.json` 中的调度配置启动定时任务，**机器重启后自动恢复**。安装成功后无需额外操作。
### 3. 手动运行（可选）
若不使用 systemd，也可以手动运行程序。先同步依赖（uv 会自动创建 `.venv`）：
```bash
uv sync
```
程序提供的子命令：
```bash
uv run python -m src.main check # 校验 LLM 接口可达性（部署期使用）
uv run python -m src.main fetch # 单次抓取后退出（systemd timer 调用）
uv run python -m src.main push # 单次推送后退出（systemd timer 调用）
uv run python -m src.main loop # 长跑模式（本地开发/调试用）
uv run python -m src.main github # 单跑 GitHub Trending 板块,打印不推送
uv run python -m src.main hackernews # 单跑 Hacker News 板块,打印不推送
```
首次运行会自动创建 `news-data/` 目录并开始抓取数据。
> 若未配置推送渠道，则可以在 news-data 目录查看生成的push信息
---
## 系统服务管理
部署完成后，使用以下命令管理服务。
### 常用命令
安装后 `daily-news` 进入系统 PATH，可在任意目录调用：
```bash
daily-news status [N] # 查看 timer/service 状态 + 最近 N 行日志（默认 15）
daily-news logs # 实时跟随日志（Ctrl+C 退出）
daily-news start # 启动两个 timer
daily-news stop # 停止两个 timer
daily-news restart # 重启两个 timer（仅重置调度，不立即触发任务）
daily-news help # 用法说明
```
手动立即触发一次任务（不影响下次调度）：
```bash
sudo systemctl start dnews-fetch.service
sudo systemctl start dnews-push.service
```
### 修改配置后
```bash
./scripts/install.sh # 重新跑一次即可，幂等（重新渲染单元 + restart timer）
```
修改 `config.json` 中的 `schedule`、`log.retention_days` 等需要重装；修改 `.env` 只需 `daily-news restart`。
### 卸载
```bash
./scripts/uninstall.sh
```
卸载会移除 systemd 单元、`/usr/local/bin/daily-news` 和日志保留 drop-in；**不会**删除 `news-data/` 数据。
### 日志保留
日志通过 systemd journald 命名空间 `dnews` 隔离，保留天数由 `config.json` 中的 `log.retention_days` 控制（默认 7 天）。不影响系统其他服务的日志。
### 日志查询
```bash
daily-news logs # 实时跟随两个 service 的日志
daily-news status [N] # 查看状态 + 最近 N 行日志（默认 15）
journalctl --namespace=dnews -f # 实时跟随命名空间内全部日志
journalctl --namespace=dnews -u dnews-fetch -f # 仅跟随 fetch service
journalctl --namespace=dnews -u dnews-push -f # 仅跟随 push service
journalctl --namespace=dnews --since "1 hour ago" # 查询近 1 小时日志
journalctl --namespace=dnews --since today # 查询今日日志
journalctl --namespace=dnews -p err # 仅查询 error 级别及以上
journalctl --namespace=dnews --vacuum-time=1s # 手动清空命名空间日志
```
---
## 配置详解（config.json）
完整的配置文件结构如下，每个字段都有详细说明：
```json
{
// 订阅源管理
"sources": {
"base_opml": "resources/rss.opml", // 基础OPML文件，包含400+预设源
"add": [ // 自定义添加的RSS源
{
"title": "OpenAI News",
"xmlUrl": "https://openai.com/news/rss.xml",
"category": "AI"
}
],
"block": [ // 手动屏蔽的源，精确匹配xmlUrl
{
"title": "Google Developers Blog",
"xmlUrl": "https://developers.googleblog.com/feeds/posts/default"
}
],
"block_domains": ["*.substack.com", "*.youtube.com"] // 域名屏蔽，支持通配符
},
// 内容过滤
"filter": {
"min_score": 60, // 最低评分阈值，低于此分不推送
"hot_threshold": 90, // 热点阈值，达到立即即时推送
"context_days": 3, // 汇总时参考的历史天数
"keep_days": 7, // 数据保留天数
"push_context_days": 5, // 汇总推送去重的上下文有效天数
"no_content_marker": "[NO_NEW_CONTENT]" // LLM返回的无内容标记，用于判断是否跳过推送
},
// 日志配置（仅对 systemd 部署生效）
"log": {
"retention_days": 7 // journald 命名空间 dnews 的日志保留天数
},
// 调度配置
"schedule": {
"fetch_interval_minutes": 30, // RSS抓取间隔（分钟）
"fetch_lookback_minutes": 120, // RSS冗余缓存时间（分钟），必须大于fetch_interval_minutes，用于防止RSS延迟导致漏读
"push_cron": ["0 8 * * *", "0 17 * * *"], // 定时推送cron表达式
"timezone_hours": 8 // 时区偏移（8=北京时间）
},
// 抓取配置
"fetch": {
"max_workers": 10, // 最大并发数
"timeout": 10 // 单请求超时（秒）
},
// LLM配置
"llm": {
"provider": "openai", // 提供商类型，openai只是知名该api接口时openai接口兼容，代码中并无实际使用
"model": "x-ai/grok-4.1-fast", // 模型名称
"baseUrl": "https://openrouter.ai/api/v1", // API端点
"apiKeyName": "OPENROUTER_API_KEY", // 环境变量名
"max_prompt_chars": 128000, // 单次prompt最大字符数
"max_concurrent_batches": 3, // 最大并发批次数
"prompts": { // prompt文件路径
"score": "prompts/score.txt",
"score_batch": "prompts/score_batch.txt",
"immediate_push": "prompts/immediate_push.txt",
"digest": "prompts/digest.txt"
}
},
// 推送配置
"push": {
"discord": {
"enabled": true, // 是否启用
"apiKeyName": "DISCORD_WEBHOOK_URL" // Webhook环境变量名
},
"feishu": {
"enabled": false,
"apiKeyName": "FEISHU_WEBHOOK_URL"
}
}
}
```
### sources —— 订阅源管理
| 字段 | 类型 | 说明 |
|------|------|------|
| `base_opml` | string | 基础 OPML 文件路径，包含 400+ 预设 RSS 源 |
| `add` | array | 自定义添加的 RSS 源，结构为 `{title, xmlUrl, category}` |
| `block` | array | 手动屏蔽的 RSS 源，精确匹配 `xmlUrl` |
| `block_domains` | array | 域名级别屏蔽，支持通配符（如 `*.substack.com`） |
### filter —— 内容过滤
| 字段 | 类型 | 说明 |
|------|------|------|
| `min_score` | number | 最低评分阈值，低于此分数的内容不参与推送（默认60） |
| `hot_threshold` | number | 热点阈值，达到此分数立即触发即时推送（默认90） |
| `context_days` | number | 上下文天数，汇总推送时参考的fetch数据历史天数（默认3天） |
| `keep_days` | number | 数据保留天数，超过天数的 JSON 文件会被清理 |
| `push_context_days` | number | 汇总推送去重的历史push文件有效天数（默认5天） |
| `no_content_marker` | string | LLM 返回的无内容标记，当推送内容包含此字符串时跳过推送（默认"[NO_NEW_CONTENT]"） |
### log —— 日志配置
仅对 `scripts/install.sh` 部署的 systemd 服务生效。
| 字段 | 类型 | 说明 |
|------|------|------|
| `retention_days` | number | journald 命名空间 `dnews` 的日志保留天数（默认 7 天）。修改后需要重跑 `./scripts/install.sh` |
### schedule —— 调度配置
| 字段 | 类型 | 说明 |
|------|------|------|
| `fetch_interval_minutes` | number | RSS 抓取间隔，单位分钟（默认30分钟） |
| `fetch_lookback_minutes` | number | RSS 冗余缓存时间（分钟），必须大于 `fetch_interval_minutes`，用于防止 RSS 延迟导致漏读（默认120分钟） |
| `push_cron` | array | 定时推送的 cron 表达式数组，支持多个时间点。**当天最早那次推送**自动作为「早报」触发 GitHub / Hacker News / 跨板块洞察三段；其余时段为默认 RSS digest。若只配置一条 cron,则每次推送都视为早报 |
| `timezone_hours` | number | 时区偏移小时数，用于时间显示（8 = UTC+8 北京时间） |
**cron 表达式说明：**
| 表达式 | 含义 |
|--------|------|
| `0 8 * * *` | 每天早上 8:00 |
| `0 17 * * *` | 每天下午 5:00 |
| `0 9,17 * * *` | 每天早上 9:00 和下午 5:00 |
格式：`minute hour day month weekday`
### fetch —— 抓取配置
| 字段 | 类型 | 说明 |
|------|------|------|
| `max_workers` | number | 最大并发数，同时抓取的 RSS 源数量 |
| `timeout` | number | 单个请求超时时间，单位秒 |
### llm —— 大语言模型配置
| 字段 | 类型 | 说明 |
|------|------|------|
| `provider` | string | LLM 提供商（支持 openai 兼容接口） |
| `model` | string | 模型名称，如 `x-ai/grok-4.1-fast` |
| `baseUrl` | string | API 端点，如 `https://openrouter.ai/api/v1` |
| `apiKeyName` | string | 环境变量名称，系统会自动读取其值 |
| `max_prompt_chars` | number | 单次 prompt 最大字符数，用于分批控制 |
| `max_concurrent_batches` | number | 最大并发批次数 |
| `prompts` | object | prompt 文件路径配置 |
### push —— 推送平台配置
| 字段 | 类型 | 说明 |
|------|------|------|
| `discord.enabled` | boolean | 是否启用 Discord 推送 |
| `discord.apiKeyName` | string | Discord Webhook 的环境变量名 |
| `feishu.enabled` | boolean | 是否启用飞书推送 |
| `feishu.apiKeyName` | string | 飞书 Webhook 的环境变量名 |
### sections (早报扩展板块)
仅在当天最早一次 `push_cron` 触发时生效(即「早报」时段)。详细设计见 `docs/extra-sections-design.md`。
#### sections.github_trending
- `enabled`: 是否启用
- `max_items`: LLM 最终选出的项目数上限(默认 3)
- `max_deep_dive`: 单次最多 deep-dive 的候选 repo 数(默认 10)
- `readme_max_chars`: README 截断长度(默认 10000)
- `history_file`: trending 去重索引文件路径
- `request_timeout`: HTTP 超时秒
- `tokenName`: GitHub token 环境变量名;不设时匿名调用(限 60 req/hr)
#### sections.hackernews
- `enabled`: 是否启用
- `select_k`: 轻 LLM 从首页 30 条中挑出的故事数(默认 1)
- `top_comments`: 每个故事抓取的**顶层(L1)评论**数上限(默认 30)
- `top_l2_per_l1`: 每条 L1 下挂的 L2 回复数上限(默认 3)
- `comment_max_chars`: 单条评论(L1 或 L2) markdown 字符上限(默认 2000)
- `comments_total_chars`: 整段评论树的总字符预算硬上限(默认 60000),防止离群 story 撑爆 prompt
- `link_content_max_chars`: 外链正文截断长度(默认 50000)
- `request_timeout`: HTTP 超时秒
- `algolia_base`: Algolia API 基址
- `jinaTokenName`: Jina Reader API key 环境变量名(默认 `JINA_API_KEY`);不设或对应环境变量为空时按匿名额度调用
#### sections.insights
- `enabled`: 是否启用跨板块洞察段
## 扩展指南
### 添加新的 RSS 源
在 `config.json` 的 `sources.add` 中添加：
```json
"add": [
{
"title": "我的自定义源",
"xmlUrl": "https://example.com/feed.xml",
"category": "AI"
}
]
```
### 添加新的推送平台
1. 在 `src/push/` 目录下创建新文件，继承 `PushPlatform` 基类
2. 实现 `validate_config()` 和 `send()` 方法
3. 在 `src/push/__init__.py` 中注册
### 修改评分逻辑
编辑 `prompts/score.txt`，调整评分标准和权重。
### 自定义 LLM 模型
修改 `config.json` 中的 `llm` 配置：
```json
"llm": {
"model": "anthropic/claude-3-opus",
"baseUrl": "https://openrouter.ai/api/v1",
"apiKeyName": "OPENROUTER_API_KEY"
}
```
## RSS 源说明
RSS 订阅源文件位于 `resources/rss.opml`，目前包含约 420 个订阅源。
RSS 订阅源初始整理自 [ginobefun/BestBlogs](https://github.com/ginobefun/BestBlogs)，包含约 420 个 AI 领域优质信息源。
用户可自行配置 RSS 源文件，只需遵循 OPML 格，在 `config.json` 的 `sources.base_opml` 修改文件路径即可。 同时用户可修改 `sources.add`或者 `sources.block`以在不破换OMPL文件的前提下对rss源进行增加或者删除。格式示例：
```json
"sources": {
"base_opml": "resources/rss.opml",
"add": [
{
"title": "OpenAI News",
"xmlUrl": "https://openai.com/news/rss.xml",
"category": "AI"
},
{
"title": "Chrome for Developers",
"xmlUrl": "https://developer.chrome.com/static/blog/feed.xml",
"category": "Chrome"
}
],
"block": [
{
"title": "Google Developers Blog",
"xmlUrl": "https://developers.googleblog.com/feeds/posts/default"
},
{
"title": "Microsoft for Developers",
"xmlUrl": "https://devblogs.microsoft.com/landing"
},
{
"title": "ElevenLabs Blog",
"xmlUrl": "https://api.bestblogs.dev/feed/elevenLabsBlog"
}
],
"block_domains": ["*.substack.com", "*.youtube.com"]
}
```
## 常见问题 FAQ
<details>
<summary><b>LLM 调用费用大概多少？</b></summary>
取决于模型选择和源数量。以 OpenRouter 的 `x-ai/grok-4.1-fast`（便宜模型）为例，日均扫描 2000+ 条 RSS + GitHub + HN，**单日成本约 ¥0.5-2 元**。可通过 `filter.min_score`、`llm.max_concurrent_batches`、`llm.max_prompt_chars` 进一步控制。
</details>
<details>
<summary><b>如何只跑某一板块进行调试？</b></summary>
```bash
uv run python -m src.main github # 只跑 GitHub Trending
uv run python -m src.main hackernews # 只跑 Hacker News
uv run python -m src.main fetch # 只跑 RSS fetch
```
打印结果到控制台，不触发实际推送。
</details>
<details>
<summary><b>抓取频率会不会被 RSS 站点封禁？</b></summary>
默认 30 分钟轮询一次，远低于大多数 RSS 服务的速率限制。`fetch.max_workers` 控制并发（默认 10），对单个站点的压力可忽略。
</details>
<details>
<summary><b>没配置推送渠道也能用吗？</b></summary>
可以。所有推送 markdown 都会落地到 `news-data/push-*.md`，即使所有推送平台 disabled 也可手动查看。
</details>
<details>
<summary><b>支持哪些 LLM 提供商？</b></summary>
任何 OpenAI API 兼容接口的服务：OpenAI、DeepSeek、OpenRouter、SiliconFlow、阿里云通义千问、Groq 等。修改 `config.json` 的 `llm.baseUrl` / `llm.model` / `llm.apiKeyName` 即可切换。
</details>
<details>
<summary><b>GitHub Trending / Hacker News 为什么不出现？</b></summary>
它们**只在当天最早一次 `push_cron` 触发时跑**（即「早报」时段）。若 `push_cron` 只配了一条 cron，则每次推送都视为早报。详见「配置详解 → schedule」一节。
</details>
<details>
<summary><b>数据存储在哪里？多久清理？</b></summary>
- 抓取数据：`news-data/fetch-YYYY-MM-DD.json`
- 推送 markdown：`news-data/push-YYYY-MM-DD-HH-MM-SS.md`
- 通知归档：`news-data/notify-YYYY-MM-DD.md`
超过 `filter.keep_days`（默认 7 天）的文件会自动清理；卸载脚本**不会**删除 `news-data/`。
</details>
---
## License
MIT License - see [LICENSE](LICENSE) file for details.
ai-daily-main/config.json.example
+120
View File
@@ -0,0 +1,120 @@
{
"filter": {
"min_score": 60,
"hot_threshold": 90,
"context_days": 2,
"keep_days": 7,
"push_context_days": 5,
"no_content_marker": "[NO_NEW_CONTENT]"
},
"log": {
"retention_days": 7
},
"schedule": {
"fetch_interval_minutes": 60,
"fetch_lookback_minutes": 120,
"push_cron": ["0 8 * * *", "0 17 * * *"],
"timezone_hours": 8
},
"sections": {
"github_trending": {
"enabled": true,
"max_items": 3,
"max_deep_dive": 10,
"readme_max_chars": 10000,
"history_file": "news-data/trending-history.json",
"request_timeout": 10,
"tokenName": "GITHUB_TOKEN"
},
"hackernews": {
"enabled": true,
"select_k": 3,
"top_comments": 50,
"top_l2_per_l1": 3,
"comment_max_chars": 2000,
"comments_total_chars": 80000,
"link_content_max_chars": 50000,
"request_timeout": 10,
"algolia_base": "https://hn.algolia.com/api/v1",
"jinaTokenName": "JINA_API_KEY"
},
"insights": {
"enabled": true
}
},
"fetch": {
"max_workers": 10,
"timeout": 10
},
"llm": {
"provider": "openai",
"model": "deepseek-v4-flash",
"baseUrl": "https://api.deepseek.com",
"apiKeyName": "DEEPSEEK_API_KEY",
"max_prompt_chars": 150000,
"max_concurrent_batches": 3,
"prompts": {
"score_batch": "prompts/score_batch.md",
"immediate_push": "prompts/immediate_push.md",
"digest": "prompts/digest.md",
"section_github": "prompts/section_github.md",
"section_hackernews_select": "prompts/section_hackernews_select.md",
"section_hackernews": "prompts/section_hackernews.md",
"insights": "prompts/insights.md"
}
},
"push": {
"discord": {
"enabled": false,
"apiKeyName": "DISCORD_WEBHOOK_URL"
},
"feishu": {
"enabled": true,
"apiKeyName": "FEISHU_WEBHOOK_URL"
}
},
"sources": {
"base_opml": "resources/rss.opml",
"add": [
{
"title": "OpenAI News",
"xmlUrl": "https://openai.com/news/rss.xml",
"category": "AI"
},
{
"title": "Chrome for Developers",
"xmlUrl": "https://developer.chrome.com/static/blog/feed.xml",
"category": "Chrome"
},
{
"title": "Cloudflare Developers",
"xmlUrl": "https://developers.cloudflare.com/changelog/rss/index.xml",
"category": "Cloudflare"
},
{
"title": "Infoq",
"xmlUrl": "https://feed.infoq.com/",
"category": "News"
}
],
"block": [
{
"title": "Google Developers Blog",
"xmlUrl": "https://developers.googleblog.com/feeds/posts/default"
},
{
"title": "Microsoft for Developers",
"xmlUrl": "https://devblogs.microsoft.com/landing"
},
{
"title": "ElevenLabs Blog",
"xmlUrl": "https://api.bestblogs.dev/feed/elevenLabsBlog"
},
{
"title": "Infoq(en)",
"xmlUrl": "http://www.infoq.com/rss/rss.action"
}
],
"block_domains": ["*.substack.com", "*.youtube.com"]
}
}
ai-daily-main/docs/extra-sections-design.md
+735
View File
@@ -0,0 +1,735 @@
# extra-sections-design.md — GitHub Trending / Hacker News / 行业洞察 板块设计
> 本文是对现有 push job 的能力扩展设计。在保持 RSS 主流程不变的前提下，新增三个板块仅在「早报」时段输出：GitHub 趋势、Hacker News 热议、跨板块行业洞察。
>
> 配套修改的源码模块、配置字段、数据契约见各章节。落地后需要把架构层结论合并回 `docs/tech-spec.md`，运行参数变更同步 `README.md`，进度记录写入 `docs/plan.md`。
>
> created: 2026-05-16
> updated: 2026-05-16（按用户决策重写：四模块独立、单页 GH trending、HN 单选 K=1、insights 结构由 prompt 决定）
## 1. 目标与范围
- 在每天的早报推送中，除了现有的 RSS 精选 digest，再加入：
- **GitHub Trending**：当日热门开源项目中筛选 1-3 个 AI 相关项目
- **Hacker News**：HN 首页中筛选 1 个最有讨论价值的 AI 相关热议（带评论与外链正文摘要）
- **行业洞察**：基于上述三个板块 + 历史 insights 段做一段跨板块趋势小结
- 这三段内容**只在早报时段生成**（当天 `schedule.push_cron` 列表里最早那次触发），其余时段维持现有纯 RSS digest 行为
- 单板块失败 → 降级推送其他板块，整体任务仍算成功；RSS 失败 → 同现有行为，任务非 0 退出
- 不引入数据库；状态全部落到 `news-data/` 的本地文件
## 2. 架构总览
```
push_job (cron 触发)
│
├─ Step 1: 早报判定
│ └─ 否 → 走原有纯 RSS digest 流程，结束
│
├─ Step 2: 四模块编排（前三路 asyncio.gather 并发，insights 串行后置）
│ ├─ run_rss_section(config, now)
│ ├─ run_github_section(config, now)
│ └─ run_hackernews_section(config, now)
│ ↓
│ └─ run_insights_section(rss_md, gh_md, hn_md, config, now)
│
└─ Step 3: push_job 上游
└─ sentinel 包裹四段 → 拼装 markdown → 推送 → 写 push-*.md
```
**关键设计选择**：
1. **模块自治**。每个板块封装为 `run_xxx_section(...) -> (markdown, error)`，板块内部从抓取、enrich、LLM 总结全包；板块互不感知。
2. **sentinel 由 push_job 统一包**。模块返回**裸 markdown**（不含 sentinel），由上游 `_assemble_with_sentinels()` 包入 `<!-- SECTION:xxx BEGIN/END -->`。这样模块不需要知道自己的板块标识，便于后续替换或加新板块。
3. **失败自吞**。模块内部捕获异常，返回 `("", error)`；上游根据返回值决定是否在最终 push 中省略该段、是否走告警通道。
4. **RSS 是核心**。其他三个模块失败都是降级；RSS 模块失败仍按现有行为整体退出非 0。
**并发模型**：`asyncio.gather`，与项目通体异步风格一致。
## 3. 模块结构
```
src/
├── sections/ ← 新增
│ ├── __init__.py # 暴露 run_rss_section / run_github_section / ...
│ ├── rss/
│ │ ├── __init__.py
│ │ └── section.py # 把现有 collect_entries_for_push + compose_digest 流程搬入
│ ├── github/
│ │ ├── __init__.py
│ │ ├── trending_scraper.py # HTML 抓取 + 解析(单页 https://github.com/trending)
│ │ ├── repo_enricher.py # GitHub REST API:metadata + README
│ │ ├── history.py # trending-history.json 读写 + 过期清理
│ │ └── section.py # run_github_section 入口
│ ├── hackernews/
│ │ ├── __init__.py
│ │ ├── frontpage_scraper.py # HTML 抓首页 30 条
│ │ ├── item_enricher.py # Algolia /items/{id} + 外链正文(html_to_markdown)
│ │ └── section.py # run_hackernews_section 入口
│ └── insights/
│ ├── __init__.py
│ └── section.py # run_insights_section 入口
├── llm.py # 新增 4 个函数(GH/HN select/HN summarize/insights)
├── storage.py # 新增:sentinel 切片、trending-history、profile 字段
├── main.py # push_job 升级:早报判定 + 四模块编排
└── push/ # 不变,平台层无感知
prompts/ ← 新增 4 个
├── section_github.md # 输入 enriched repo 数组 → 选 1-3 + 写 markdown
├── section_hackernews_select.md # 30 条 frontpage 元数据 → 选 K 个 id(默认 K=1)
├── section_hackernews.md # enriched story → 写 markdown(K=1 时不挑选,只行文)
└── insights.md # 输入三段成品 + 近 N 天 insights 段历史 → 写洞察段
news-data/
├── fetch-*.json # 不变
├── notify-*.md # 不变
├── push-*.md # 内容升级:含 sentinel 与 profile frontmatter
└── trending-history.json # 新增:GH 已查阅 repo 索引
```
## 4. 数据契约
### 4.1 push 文件分段 sentinel
push 文件保留**完整拼接**写入磁盘和发送各平台，但用 HTML 注释 sentinel 划分板块边界。HTML 注释在 markdown 渲染中不显示，机器可解析，下游做"按段查重"时能精确切片。
```markdown
---
pushDate: "2026-05-16T08:00:03+08:00"
profile: "morning"
sourceCount: 12
totalEntries: 12
---
<!-- SECTION:rss BEGIN -->
# 📰 AI Daily 每日精选 | 2026-05-16
*开头一句定调...*
### 1️⃣ ...
<!-- SECTION:rss END -->
<!-- SECTION:github BEGIN -->
## ⭐ GitHub 趋势
- **owner/repo** ⭐234 — 一句话价值定位
<!-- SECTION:github END -->
<!-- SECTION:hackernews BEGIN -->
## 🟧 Hacker News 热议
### 标题 (120 pts · 45 comments)
- 链接: url
- 要点:...
- HN 讨论: comments_url
<!-- SECTION:hackernews END -->
<!-- SECTION:insights BEGIN -->
## 💡 今日洞察
(行文结构由 prompts/insights.md 决定,代码不强加格式)
<!-- SECTION:insights END -->
```
某板块 markdown 为空 → 对应 sentinel 段**整段省略**（不留空标记，不留空 SECTION）。
### 4.2 分段提取函数（storage.py 新增）
```python
def extract_section(push_md: str, section: str) -> str:
"""从 push 文件内容中切出 <!-- SECTION:{section} BEGIN/END --> 之间的 markdown。
向后兼容:
- 新 push 文件(带 sentinel): 按 sentinel 边界切片
- 老 push 文件(无 sentinel) 且 section=='rss': 返回整个 body(老文件视为全 RSS)
- 老 push 文件且 section in {github, hackernews, insights}: 返回空字符串
"""
def load_recent_section_titles(section: str, days: int, data_dir="news-data") -> str:
"""汇总近 days 天 push-*.md 的指定板块,提取标题级别清单(沿用 _extract_push_titles 思路)。
仅 insights 板块在新增模块中使用本函数加载历史。
GH/HN 板块按用户决策不传历史上下文给 LLM,不调用本函数。
"""
```
四个 LLM 与历史上下文的关系：
| LLM 调用 | recent context 数据源 |
|---|---|
| `compose_digest` (RSS) | 维持现有 `load_recent_push_titles(filter.push_context_days)`（老接口在 sentinel 升级后等价于 `load_recent_section_titles("rss", ...)`） |
| `summarize_github_trending` | 不传 |
| `select_ai_related_hn` | 不传 |
| `summarize_hackernews` | 不传 |
| `generate_trend_insights` | `load_recent_section_titles("insights", filter.push_context_days)` |
### 4.3 trending-history.json
```json
{
"repos": {
"https://github.com/owner/repo-a": "2026-05-15",
"https://github.com/owner/repo-b": "2026-05-12"
},
"updated_at": "2026-05-16T08:00:01+08:00"
}
```
**写入语义**（用户决策的精确语义）：
每次早报触发，按下列顺序处理：
1. 加载 history，剔除 `last_seen_date < today - filter.keep_days` 的条目
2. 抓 trending 页 → 得到 `all_repos`
3. 对 `all_repos` 中每个 url：
- 若已在 history → `history.touch(url, today)`（更新日期），**从候选移除**
- 不在 history → 进入 `candidates`
4. 把 `candidates` 中每个 url 也 `history.touch(url, today)` 写入 history
5. 持久化 history（覆盖写）
6. 对 `candidates`（即今日新出现的 repo）做后续 deep-dive 与 LLM 总结
效果：repo 在 trending 上挂多久就被屏蔽多久；过 `keep_days` 天没再出现则可重新推荐。
## 5. 模块详设
### 5.1 RSS 模块（迁移既有逻辑）
`src/sections/rss/section.py::run_rss_section(config, now) -> (str, Optional[str])`：
把现有 `run_push_job` 中"收集 + compose_digest"的部分原样迁过来，返回裸 markdown（不含 sentinel）+ 错误信息。无新增功能。
### 5.2 GitHub 模块
`src/sections/github/section.py::run_github_section(config, now) -> (str, Optional[str])`：
```
1. 抓取 trending 单页(HTML)
GET https://github.com/trending
解析 → all_repos: [{url, full_name, description, language, stars_today, stars_total}]
2. 加载 history 并清理
history = load_trending_history(sections.github_trending.history_file)
history.cleanup(keep_days=filter.keep_days)
3. 候选筛选(按 §4.3 语义)
candidates = []
for repo in all_repos:
if repo.url in history:
history.touch(repo.url, today)
else:
candidates.append(repo)
4. 候选写回 history + 持久化
for repo in candidates:
history.touch(repo.url, today)
history.save()
5. 数量护栏
if not candidates: return ("", None) # 静默
if len(candidates) > max_deep_dive:
candidates = candidates[:max_deep_dive] # 截断,默认 10
6. 并发 deep-dive(REST API)
async for repo in candidates:
meta = await fetch_repo_metadata(owner, repo) # GET /repos/{o}/{r}
readme = await fetch_readme(owner, repo) # GET /repos/{o}/{r}/readme
enriched = [{...repo, topics, license, pushed_at, readme_excerpt}]
- 单 repo 任一请求失败 → 该 repo 跳过 + 错误聚合,不阻塞其他
7. LLM:summarize_github_trending(enriched, config)
prompt: 候选数组(含 readme_excerpt) → 选 1-max_items + 写 markdown
不传 recent_section_titles
8. 返回 (markdown, error)
```
**REST API 调用细节**：
| 调用 | 路径 | 取什么 |
|---|---|---|
| metadata | `GET /repos/{owner}/{repo}` | `description, topics, language, license.spdx_id, pushed_at, stargazers_count, archived` |
| readme | `GET /repos/{owner}/{repo}/readme` | `content` (base64) → decode → 截断到 `readme_max_chars` |
- `archived=true` 的 repo 从候选剔除（trending 偶尔出现僵尸归档项目）
- README 截断策略：前 `readme_max_chars` 字符（默认 10000，基于 trending 页 README 长度分布 p50≈25k 选定，详见 §14 决策记录）
- 鉴权：`config.sections.github_trending.tokenName`（默认 `"GITHUB_TOKEN"`）对应的环境变量存在时走 `Authorization: Bearer {token}`，否则匿名调用并接受 60 req/hr 上限（日 10 个 repo × 2 calls = 20 calls，匿名安全）
**enriched repo 字段（喂给最终 LLM）**：
```json
{
"url": "https://github.com/owner/repo",
"full_name": "owner/repo",
"description": "(来自 trending 页)",
"language": "Python",
"stars_today": 234,
"stars_total": 12340,
"topics": ["llm", "rag", "agent"],
"license": "MIT",
"pushed_at": "2026-05-15",
"readme_excerpt": "(前 3000 chars)"
}
```
### 5.3 Hacker News 模块
`src/sections/hackernews/section.py::run_hackernews_section(config, now) -> (str, Optional[str])`：
```
1. 抓首页(HTML)
GET https://news.ycombinator.com/news
解析 30 条 → front: [{id, title, url, site, points, comments, comments_url}]
2. 轻 LLM 初筛:select_ai_related_hn
输入: 30 条 frontpage 元数据(无正文)
输出: K = sections.hackernews.select_k 个 story id(默认 K=1)
if K 个为空 → return ("", None) # 静默
3. 并发 enrich 选中的 K 个 story
async for story in selected:
- 评论树:Algolia GET /api/v1/items/{id}
→ 取前 top_comments 条 L1(默认 30,按 HN ranking)
→ 每条 L1 下挂前 top_l2_per_l1 条 L2 回复(默认 3)
→ 每条 text 过 html_to_markdown,单条截断到 comment_max_chars(默认 2000)
→ 累计达 comments_total_chars(默认 60000) 立即停止,防离群 story 撑爆 prompt
→ 输出 tree JSON: [{"l1": "...", "replies": ["...", "..."]}, ...]
- 外链正文:
if story.url 指向 https://news.ycombinator.com/item?id=... (Show HN/Ask HN):
从 Algolia 同次返回的 root.text 字段取(无外部请求)
else:
GET story.url → html_to_markdown → 截断到 link_content_max_chars(默认 50000,p50≈10k)
- 单 story 任一失败 → 字段留空,metadata 仍传给最终 LLM
4. LLM:summarize_hackernews(enriched_stories, config)
prompt: 对输入的 K 个 story 全部行文（K 通常 = 1）
不传 recent_section_titles
5. 返回 (markdown, error)
```
**Algolia API 接口**：
```
GET https://hn.algolia.com/api/v1/items/{id}
```
返回 JSON：
```json
{
"id": 12345678,
"title": "...",
"url": "...",
"points": 120,
"author": "...",
"text": null, // Show HN/Ask HN 的正文在这里
"children": [ // 顶层评论数组(按 HN ranking 排序)
{"id": ..., "text": "<HTML>", "author": "...", "children": [...]},
...
]
}
```
**优势 vs HTML 解析**：免去 HN 的 `td.ind[indent="0"]` indent-tree 解析；评论文本是干净 HTML 字符串，直接 `html_to_markdown`。
**Show HN / Ask HN 特例**：
- 首页解析时 `url` 字段就是 `https://news.ycombinator.com/item?id=X`，作为"非外链"标记
- enrich 时只调一次 Algolia（覆盖评论 + post 正文 `text` 字段），不再发外部请求
**enriched story 字段（喂给最终 LLM）**：
```json
{
"id": "12345678",
"title": "...",
"url": "...",
"site": "example.com",
"points": 120,
"comments": 45,
"comments_url": "https://news.ycombinator.com/item?id=12345678",
"link_content": "(markdown, ≤3000 chars; Show HN 时是 post 正文)",
"top_comments": [
{"l1": "(markdown, ≤comment_max_chars)", "replies": ["(markdown, ≤comment_max_chars)", "..."]},
...
]
}
```
### 5.4 Insights 模块
`src/sections/insights/section.py::run_insights_section(rss_md, gh_md, hn_md, config, now) -> (str, Optional[str])`：
```
1. 加载历史
recent = load_recent_section_titles("insights", filter.push_context_days)
2. LLM:generate_trend_insights
输入: {"rss": rss_md, "github": gh_md, "hackernews": hn_md} + recent
输出: insights 段 markdown
注:行文结构、bullet 数量、风格约束等全部交给 prompts/insights.md,
代码层不强加固定格式
3. 返回 (markdown, error)
```
如果某板块返回空（失败或本日无内容），prompt 里对应键标记 `"(本次无内容)"`，LLM 自行适配。
## 6. LLM 调用与 Prompt 策略
### 6.1 新增 LLM 函数（src/llm.py）
```python
async def select_ai_related_hn(
candidates: list[dict], # 首页 30 条元数据(无正文)
k: int, # 期望返回数量,默认 1
config: dict,
) -> tuple[list[str], Optional[str]]:
"""轻量 LLM:从 HN 首页候选中挑出 k 个 AI 相关的 story id,只读 title/site/points/comments。
返回 ([id1, ...], error)。"""
async def summarize_github_trending(
enriched_repos: list[dict], # 已 deep-dive 的 repo 候选(含 readme_excerpt + topics)
config: dict,
) -> tuple[str, Optional[str]]:
"""选 1-max_items + 写 markdown 段。不传历史上下文。"""
async def summarize_hackernews(
enriched_stories: list[dict], # 已 enrich(含 link_content 与 top_comments)
config: dict,
) -> tuple[str, Optional[str]]:
"""对输入的 K 个 enriched stories 全部行文。K 由配置 select_k 决定，默认 1。
不传历史上下文。"""
async def generate_trend_insights(
sections: dict[str, str], # {"rss": md, "github": md, "hackernews": md}
recent_insights: str, # load_recent_section_titles("insights", days)
config: dict,
) -> tuple[str, Optional[str]]:
"""输入三段成品 + 近期 insights 标题,返回洞察段 markdown。"""
```
返回风格沿用 `generate_immediate_push`：成功返回 `(content, None)`，失败返回 `("", error_msg)`。
**单次早报推送的 LLM 调用预算**：
| 调用 | 输入规模 | 用途 |
|---|---|---|
| `compose_digest` | 当日符合条件的 RSS 条目 | 现有，RSS digest 主体 |
| `select_ai_related_hn` | 30 条 HN 首页元数据 | 轻量；只读 title/site/points/comments |
| `summarize_github_trending` | ≤ `max_deep_dive`=10 个 enriched repos | 选 1-3 + 行文 |
| `summarize_hackernews` | `K = select_k` 个 enriched stories（默认 1） | 行文 K 条 |
| `generate_trend_insights` | 三段已生成 markdown + 近期 insights 标题 | 一段洞察 |
合计 5 次 / 早报。晚报维持现有 1 次。
### 6.2 关注领域（GitHub / HN 共用）
为避免领域定义在 3 个 prompt 里漂移，统一在此沉淀；各 prompt 在自身骨架里直接引用本节，不做重新发明。
**正面关注**：
- **AI Agent**：智能体架构、工具链、多智能体、自主规划、Agent 框架
- **AI 模型**：训练、推理、微调、量化部署、模型服务、语音 / 多模态 / 视觉模型
- **AI 基础设施**：GPU 调度、芯片硬件、数据中心、推理优化、分布式训练、向量数据库、RAG 框架
- **大厂 / 前沿动态**：Apple、Google、Meta、OpenAI、Anthropic、Microsoft、xAI 等公司的官方动作与战略
- **AI 集成的开发者工具**：API 网关、自动化脚本、低代码平台等明确与 AI 协同的工具
- **有创新性的开源产品**：日增长显著且有清晰用户价值（GH 板块专属）
**负面排除（一律剔除）**：
- 嵌入式开发（Arduino、ESP32、树莓派、单片机）
- 底层系统编程（内存分配器、编译器、链接器，与 AI 工作负载无明显关联时）
- 通用开发工具（命名规范、代码风格、纯前端模板、UI 组件库、管理后台模板、静态网站主题）
- 学习资源（纯教程仓库、面试题合集、Roadmap，除非是含实用代码的深度技术指南）
- 配置文件集合（Dotfiles、配置模板）
- 与 AI / 科技无关的内容（电子书、资源搬运、刷榜项目、明星项目搬运）
- 纯娱乐 / 高风险误用（deepfake 等无明确基础设施价值的项目）
### 6.3 Prompt 文件
#### prompts/section_github.md（骨架）
- 角色定位（"开源情报分析师"）
- 输入 schema 说明（JSON 数组：url / full_name / description / language / stars_today / stars_total / topics / license / pushed_at / readme_excerpt）
- 关注领域：引用 §6.2（正面列表与负面排除完整复制进 prompt）
- 选项规则：
- 从候选中挑 1-`max_items` 个最值得关注的项目
- 优先信号：stars_today 高 + topics 含 AI 标签（agent/llm/rag/inference/training 等）+ readme 描述明确 + 非纯模板/教程仓库
- 必跳过：`archived=true`（理论上已在 enricher 剔除，prompt 层兜底）、纯 awesome-list、个人配置 dotfiles
- 输出格式（markdown 列表）：
- `- **owner/repo** ⭐{stars_today} — 一句话价值定位 [link]`
- 一句话需点明"解决什么问题"，避免营销语
- 风格约束：与 `prompts/digest.md` 同源；负面句式（"震撼""炸裂""革命性"）禁用；避免套话
#### prompts/section_hackernews_select.md（骨架）
- 角色定位（"HN 早间选题人"）
- 输入：JSON 数组（30 条 frontpage 元数据：id / title / site / points / comments）
- 关注领域：引用 §6.2
- 任务：挑 `k` 个最符合关注领域的 story id（K 默认 1）
- 决策原则：title + site 不足以判定 AI 相关时，**宁可漏选不可错选**（错选会让最终 LLM 写出与 AI Daily 调性无关的内容）
- 输出：纯 JSON id 数组，如 `["12345"]` 或 `[]`（无任何匹配时返回空数组）
- 严禁输出任何解释性文字
#### prompts/section_hackernews.md（骨架）
- 角色定位（"HN 早间编辑"）
- 输入 schema 说明（K 个 enriched story，含 `link_content` 与 `top_comments`）
- 关注领域：引用 §6.2（K=1 时通常无需筛选，仅作为行文背景参考）
- 任务：对输入的 `K` 个 enriched stories 全部行文（不再二次挑选）
- 内容要求（每条 story）：
- 提炼原文核心（背景 / 要点 / 结论）
- 汇总 HN 评论区的有价值观点（支持 / 反对 / 补充），不是简单复述
- 若评论中出现明显反驳原文的观点，必须保留并标注
- 输出格式建议（最终以 prompt 实测为准）：
```
### 标题 (N pts · M comments)
- 链接: url
- 内容总结: 2-3 条核心要点
- 💬 HN 讨论: 1-2 条最有价值的观点（含反对意见）
- 🔗 HN 讨论页: comments_url
```
- 风格约束：客观、犀利、克制；避免与 RSS digest 句式雷同；不做宏大叙事
#### prompts/insights.md（骨架）
- 角色定位（"AI 行业观察员"）
- 输入：三段成品 markdown + 近 N 天 insights 板块清单
- 任务：基于三段产出做跨板块小结
- 风格约束：避免与 RSS digest 句式雷同；避免简单复述已经在其他板块出现过的具体新闻
- 结构与 bullet 数量交由 prompt 内部约定，code 层不限制
### 6.4 调用顺序与并发
```python
async def _run_morning_push(config):
rss_md, gh_md, hn_md = await asyncio.gather(
run_rss_section(config, now),
run_github_section(config, now),
run_hackernews_section(config, now),
return_exceptions=False, # 各 section 自吞异常,不抛
)
insights_md, _ = await run_insights_section(rss_md, gh_md, hn_md, config, now)
final = _assemble_with_sentinels({
"rss": rss_md,
"github": gh_md,
"hackernews": hn_md,
"insights": insights_md,
})
await send_to_platforms(final, config["push"])
save_push_file(get_push_file(), final, profile="morning", ...)
```
`_assemble_with_sentinels(sections: dict[str, str]) -> str` 的契约：
- 按固定顺序拼装 `rss → github → hackernews → insights`
- 空 markdown 段整段省略（连同 sentinel）
- 段间留一个空行
## 7. 行业洞察板块设计
按用户决策，本节**不在 code 层规定 insights 的格式**：
- bullet 数量、子标题、字数限制、固定栏目等都属于 prompt 工程范畴
- 调整方法：编辑 `prompts/insights.md` 而非改代码
- 输入合同（code 层保证）：
- `sections["rss" | "github" | "hackernews"]` 三个键的 markdown
- 任一板块为空时该键值为 `"(本次无内容)"`
- `recent_insights`：近 `filter.push_context_days` 天 insights 段标题清单（防风格趋同）
- 输出合同（code 层不校验）：直接作为 markdown 段插入
## 8. 配置 schema 增量
```json
{
"filter": {
"min_score": 60,
"hot_threshold": 90,
"context_days": 2,
"keep_days": 7,
"push_context_days": 5,
"no_content_marker": "[NO_NEW_CONTENT]"
},
"schedule": {
"fetch_interval_minutes": 30,
"fetch_lookback_minutes": 120,
"push_cron": ["0 8 * * *", "0 17 * * *"],
"timezone_hours": 8
},
"sections": {
"github_trending": {
"enabled": true,
"max_items": 3,
"max_deep_dive": 10,
"readme_max_chars": 10000,
"history_file": "news-data/trending-history.json",
"request_timeout": 10,
"tokenName": "GITHUB_TOKEN"
},
"hackernews": {
"enabled": true,
"select_k": 1,
"top_comments": 30,
"top_l2_per_l1": 3,
"comment_max_chars": 2000,
"comments_total_chars": 60000,
"link_content_max_chars": 50000,
"request_timeout": 10,
"algolia_base": "https://hn.algolia.com/api/v1"
},
"insights": {
"enabled": true
}
},
"llm": {
"prompts": {
"score_batch": "prompts/score_batch.md",
"immediate_push": "prompts/immediate_push.md",
"digest": "prompts/digest.md",
"section_github": "prompts/section_github.md",
"section_hackernews_select": "prompts/section_hackernews_select.md",
"section_hackernews": "prompts/section_hackernews.md",
"insights": "prompts/insights.md"
}
}
}
```
向后兼容：
- `sections` 整段缺失 → 等价于全部 `enabled=false` → push_job 走原有纯 RSS 路径
- `push_cron` 为空 → 无早报触发,不生成新板块
- 旧 push 文件没有 sentinel → `extract_section("rss", ...)` 返回整个 body，其他 section 返回空字符串
- `GITHUB_TOKEN` 未设 → GH 模块匿名调用，照常运行
## 9. 失败隔离与降级策略
| 失败位置 | 行为 |
|---|---|
| `run_rss_section` 失败 | 整个 push_job 退出非 0（核心承诺不变） |
| GH trending 抓取 / 解析失败 | `run_github_section` 返回 `("", error)` → 板块整段省略 → 告警 |
| GH 单 repo metadata/readme 失败 | 该 repo 跳过 + 错误聚合，不阻塞其他 repo |
| GH summarize LLM 失败 | 板块整段省略，告警 `notify_llm_errors("section_github", ...)` |
| HN 首页抓取失败 | 同 GH |
| HN 轻 LLM 初筛失败或返回空 | 板块整段省略，告警（初筛失败）或静默（结果为空） |
| HN 单 story enrich 失败 | 字段留空，metadata 仍传给最终 LLM |
| HN summarize LLM 失败 | 板块整段省略，告警 |
| insights LLM 失败 | 洞察段省略，其他板块照常推送，告警 |
| 早报判定为否 | 完全跳过 GH/HN/insights，不消耗任何额外 API |
| `sections.xxx.enabled=false` | 对应模块直接返回 `("", None)`，静默跳过 |
整体准则：**RSS 是核心，其余是增强**。除 RSS 外的任何失败都不阻塞推送，但都通过现有 `notify_llm_errors` 通道发简单告警，方便事后排查。
## 10. 早报判定逻辑
```python
def is_morning_push(now: datetime, config: Dict) -> bool:
cron_list = config.get("schedule", {}).get("push_cron", [])
if not cron_list:
return False
if len(cron_list) == 1:
return True # 唯一定时即"最早",任何触发都视为早报
base = now.replace(hour=0, minute=0, second=0, microsecond=0)
today_fires = [croniter(c, base).get_next(datetime) for c in cron_list]
closest = min(today_fires, key=lambda f: abs(now - f))
return closest == min(today_fires)
```
规则:`push_cron` 中 now 离哪条 cron 最近就归为那条;最近的那条若是当天最早的 cron,则视为早报。
为什么不再用单独的 `morning_cron` + 容差:
- 复用 `push_cron`,少一个配置项,加 cron 自动延展
- 「最近匹配」自动等价于容差 = 到次近 cron 距离的一半,systemd timer 漂移更稳健
- 单条 cron 部署自然变成"每次推送都是早报",与用户预期一致
## 11. 与现有模块的集成点
- `src/main.py::run_push_job` 改造：
1. 开头加 `is_morning = is_morning_push(now, config)`
2. `is_morning=False` 时走原路径（`compose_digest` only）
3. `is_morning=True` 时进入 `_run_morning_push` 编排（四模块 + sentinel 拼装）
- `src/storage.py` 新增：
- `extract_section(content, section)` + `load_recent_section_titles(section, days, data_dir)`
- `load_trending_history(path) -> TrendingHistory`、`TrendingHistory.cleanup/touch/save`
- `save_push_file` 微调：frontmatter 带上 `profile: "morning"|"default"`，方便后续按 profile 分析
- `cleanup_old_files` 增加对 `trending-history.json` 的"过期条目剪枝"（不删整个文件）
- `src/push/` **不动**，平台层无感知
- `src/llm.py` 新增 4 个函数：`select_ai_related_hn / summarize_github_trending / summarize_hackernews / generate_trend_insights`
- `src/sections/` 全新模块树（4 个子包）
## 12. 测试策略
新增单元测试：
- `tests/pytest/test_sections_github_scraper.py`：本地 HTML fixture（保存几个真实 trending 页面快照）→ 测试解析
- `tests/pytest/test_sections_github_history.py`：测试 `TrendingHistory.touch/cleanup`、URL 已存在/不存在分支
- `tests/pytest/test_sections_github_enricher.py`：mock `aiohttp` → 测试 REST API 字段映射、archived 过滤、token 鉴权头
- `tests/pytest/test_sections_hackernews_scraper.py`：HN 首页 fixture → 测试 30 条解析
- `tests/pytest/test_sections_hackernews_enricher.py`：mock Algolia → 测试评论 top_comments 截断、Show HN 特例
- `tests/pytest/test_storage_sections.py`：sentinel 切片（含老文件 fallback）、`load_recent_section_titles`
- `tests/pytest/test_morning_detection.py`：cron 边界、容差、跨时区
新增交互式脚本（与现有 `tests/*.py` 风格一致）：
- `tests/run_morning_push.py`：模拟一次完整早报，强制 `is_morning=True`
- `tests/fetch_trending.py`：单独跑 GH 抓取 + deep-dive
- `tests/fetch_hackernews.py`：单独跑 HN 抓取 + enrich
## 13. 实施步骤建议
按依赖顺序实施，每步可独立 PR / commit：
1. **storage 层**：sentinel 切片、`trending-history.json` 读写、`load_recent_section_titles`、`save_push_file` profile 字段、`cleanup_old_files` 对 history 文件的处理 + 单测
2. **RSS 模块迁移**：把现有 `run_push_job` 中的 RSS 流程提取为 `run_rss_section`，验证行为不变
3. **GitHub 模块**：trending scraper → history → enricher → section 入口 + 单测 + fixture
4. **Hacker News 模块**：frontpage scraper → Algolia enricher → section 入口 + 单测 + fixture
5. **LLM 函数 + Prompt 文件**：4 个新 LLM 函数 + 4 个 prompt
6. **Insights 模块**：依赖 §5.4，相对简单
7. **push_job 升级**：早报判定 + 四模块编排 + sentinel 拼装 + 失败隔离
8. **配置 schema 升级**：`config.json.example` 与 `config.json` 同步；README 配置详解章节补全
9. **文档同步**：`docs/tech-spec.md` 把架构升级合并；`docs/plan.md` 写进度
## 14. 关键决策记录
| 决策 | 方案 | 原因 |
|---|---|---|
| 新板块时机 | 仅早报（当天最早一次 `push_cron` 触发） | 板块价值更适合一日一报；晚报维持原有 RSS 节奏；复用 push_cron 不引入新配置 |
| 模块边界 | `src/sections/<board>/` 各自封装抓取+LLM+总结 | 模块自治便于扩展、替换、单测；上游编排极简 |
| sentinel 归属 | push_job 上游统一包 | 模块不感知自己的板块标识；新增板块零修改成本 |
| GH trending 数据源 | 单页 HTML `https://github.com/trending`，无语言/since 过滤 | 用户决策：最简、最稳；语言过滤靠 topics + readme 在 LLM 层判 |
| GH deep-dive 内容 | REST API 拿 metadata + topics + README | topics 是 AI 相关性最强信号；metadata 补 license/pushed_at；README 给内容深度 |
| GH 鉴权 | GITHUB_TOKEN 可选 | 日 ~20 calls 远低于匿名 60 req/hr 上限；零配置即可跑 |
| GH 筛选策略 | history 过滤 → 全部 deep-dive → 一次 LLM 选 1-3 | 候选量小（5-15）；单次 LLM 比两阶段简单且选择质量高 |
| GH 候选护栏 | `max_deep_dive=10` | 极端日（trending 大改）限制 HTTP 与 token 消耗 |
| HN 数据源 | 首页 HTML + 评论/正文 Algolia | 首页要"现场感"走 HTML；Algolia 评论 JSON 结构清晰，免去 indent-tree 解析 |
| HN 筛选策略 | 30 条 → 轻 LLM 选 K=1 → enrich → 最终 LLM 行文 | 用户决策：把 enrich 工作量压到 1 个 story；轻 LLM 用 title 已足够判 AI 相关 |
| HN 评论结构 | L1 + 每个 L1 下挂 N 条 L2 回复,tree JSON 喂 LLM | 真实数据:L2 信息量与 L1 持平(24 条/8k chars vs 18/6k);拍平丢失父子关系,LLM 无法识别"回复反驳了顶层"。Tree 结构让 LLM 看清论辩链 |
| HN 评论上限 | `top_comments=30`(L1)+ `top_l2_per_l1=3` + `comments_total_chars=60000`(总预算) | 真实平均:L1 18 条/6k chars,L2 24 条/8k。30+3 给足余量但平均只跑 ~14k。总预算硬上限拦住离群 story(309 评论那种) |
| 无历史上下文 | GH / HN 板块均不传 recent_section_titles | 用户决策：避免不必要的上下文污染；GH/HN 风格与 RSS digest 差异已足够大 |
| insights 历史窗口 | 复用 `filter.push_context_days` | 不引入新字段；insights 段需要历史防风格趋同 |
| insights 输出结构 | 由 prompt 决定，code 不强加 | 用户决策：bullet 数量与栏目属于 prompt 工程，便于迭代 |
| GH / HN 关注领域沉淀 | 在 §6.2 集中定义正面列表 + 负面排除，3 个 prompt 引用 | 避免领域定义在 prompt 间漂移；用户已明确兴趣边界（AI Agent / 模型 / 基础设施 / 大厂动态），排除嵌入式、底层系统、纯前端模板、学习资源等 |
| 板块 sentinel 用 HTML 注释 | `<!-- SECTION:xxx BEGIN/END -->` | markdown 渲染不显示；机器易解析；老 push 文件零冲突 |
| 失败降级粒度 | 单板块失败省略本段；RSS 失败整体退出 | RSS 是核心承诺，其他是增强 |
| 早报判定 | cron + 容差，而非"今天第一次" | 早报失败时晚报不会错误升级为长版本 |
| 截断参数初始值（2026-05-17 合入） | `readme_max_chars=3000` / `top_comments=20` / `comment_max_chars=500` / `link_content_max_chars=3000` | 凭直觉给出的保守默认；上线后通过真实数据校准 |
| 截断参数校准（2026-05-17 合入后,激进路径） | `readme_max_chars: 5000→10000` / `top_comments: 20→50` / `link_content_max_chars: 6000→50000` / `comment_max_chars: 800` 不变 / `max_prompt_chars: 64000→150000` | 用户决策:把 LLM 上下文用到 DeepSeek v4 flash 128k tokens 限的合理水平,优先内容深度而非 API 成本。Worst-case 单次 LLM prompt:GH≈110k chars / HN≈95k chars,均在 150k budget 内。中英混合 100k chars ≈ 30-50k tokens,远低于 128k 模型限 |
| 单板块 CLI（2026-05-17 合入后） | `python -m src.main github` / `hackernews` | 便于 prompt 调优期反复跑单板块而不消耗全套 LLM 调用 |
hacker news 评论统计
┌─────────────┬──────────┬──────────────┬────────────────┬──────────┐
│ 层级 │ 平均条数 │ 平均 md 字符 │ 平均单条 chars │ 最大单条 │
├─────────────┼──────────┼──────────────┼────────────────┼──────────┤
│ L1 顶层评论 │ 18 │ 5,952 │ ~330 │ 1,755 │
├─────────────┼──────────┼──────────────┼────────────────┼──────────┤
│ L2 一级回复 │ 24 │ 8,166 │ ~340 │ 2,381 │
├─────────────┼──────────┼──────────────┼────────────────┼──────────┤
│ L3+ 更深层 │ 41 │ 13,048 │ ~315 │ 2,291 │
├─────────────┼──────────┼──────────────┼────────────────┼──────────┤
│ ALL 全部 │ 83 │ 27,166 │ ~330 │ 2,381 │
└─────────────┴──────────┴──────────────┴────────────────┴──────────┘
ai-daily-main/docs/plan.md
+90
View File
@@ -0,0 +1,90 @@
## TODO
当前待办
- [ ] 优化提示词： 推送格式；参考链接去除非官方信息； insights 不够深度
- [ ] 日志系统，保存到文件，push和fetch分开，
- [ ] 早报内容格式优化：参考appso / xiaohu / ai gap
- [ ] 优先级顺序
- [ ] 美化排版
- [ ] 添加更多信息源，如 TechCrunch、
- [ ] 允许fetch链接中的内容对信息进行扩展
- [ ] 更多信息源以及信息获取不全： https://www.anthropic.com/research/glasswing-initial-update / github blog
长期待办
- [ ] 增加图片/信息图
- [ ] 推送到知乎 / 小红书 / 网站
- [ ] llm api fallback
## 技术决策记录
| 决策 | 方案 | 原因 |
|------|------|------|
| 定时调度 | systemd timer（生产）+ croniter（loop 模式） | 进程崩溃/服务器重启可自愈，配置热更新，比内置 asyncio.gather 更稳健 |
| 包管理 | uv | 速度快、单工具管理 venv/pip/lockfile，项目独立 `.venv` |
| LLM 健康检查 | 仅在 `install.sh` / `loop` 启动时校验 | 每次 timer 触发都校验会增加无意义的 LLM API 调用，运行期异常由 `notify_llm_errors` 兜底 |
| 日志方案 | journald 命名空间 `dnews` + `MaxRetentionSec` | 与系统其他服务隔离，按 `log.retention_days` 自动轮转，无需写文件日志 |
| 数据格式 | JSON | 结构清晰、易处理、支持嵌套 |
| 推送文件 | Markdown+YAML | 人工可读、Frontmatter 元数据 |
| LLM 评分 | 批量 JSON | 减少 API 调用次数 |
| 状态追踪 | 文件时间戳 | 无需外部数据库 |
| RSS延迟防护 | fetch_lookback_minutes | 防止RSS延迟导致漏读 |
| LLM异常通知 | 调用方统一上报 | 避免批次级刷屏，同时保留关键异常通知 |
| 报告 metadata 统一 | 全部从 LLM frontmatter 解析(早报/晚报/即时) | 取代之前的 `extract_title_from_content` h1 提取+硬编日期标题；个性化标题 + lead 导读 + highlights 列表统一承载 |
## 开发进度
**2026-05-22**
- ✅ 推送 metadata 统一改造：晚报与即时消息也走 frontmatter（之前只有早报）
- `prompts/digest.md` 增加 frontmatter（title/lead/highlights），删除正文「开头一句话定调」段
- `prompts/insights.md` frontmatter 增加 `lead`（综合三段的 60-100 字前言）与 `highlights`（2-3 条卡片重点）
- `prompts/immediate_push.md` 将 `# 标题` 迁到 frontmatter `title` 字段
- `src/llm.py` 新增 `parse_digest_with_metadata` / `parse_immediate_push_with_metadata`，统一 `_parse_frontmatter` 帮手；移除 `extract_title_from_content`
- `src/sections/rss/section.py` 返回三元组 `(body, metadata, err)`；早报丢弃 digest metadata，由 insights 段覆盖
- 晚报现在拥有个性化标题 + lead + highlights，与早报对齐
**2026-05-17**
- ✅ 早报扩展板块上线：在 RSS digest 之上叠加 GitHub Trending / Hacker News / 跨板块洞察三段，仅 `schedule.morning_cron` 命中时触发，晚报维持纯 RSS 行为
- 设计与实施详见 [`docs/extra-sections-design.md`](extra-sections-design.md) 与 [`docs/superpowers/plans/2026-05-17-extra-sections.md`](superpowers/plans/2026-05-17-extra-sections.md)
**2026-05-15**
- 优化提示词，修复长时运行下的新闻报告措辞趋同问题
**2026-05-14**
- [x] 外置定时（已切换到 systemd timer）
- [x] 系统服务一键运行（`scripts/install.sh`）
- 使用uv进行python项目管理
- 配置变更：`config.json` 新增 `log.retention_days` 字段
- 新增 `daily-news` 系统级包装脚本：装在 `/usr/local/bin/daily-news`，提供 `start/stop/restart/status/logs` 等命令，封装 systemctl/journalctl 调用细节
**2026-03-08**
- 新增 LLM 异常通知：`compose_digest`、`generate_immediate_push` 与 `score_batch` 的错误会通过现有推送渠道发送简单告警
- 优化批量评分容错：`score_batch` 在批次返回数量不匹配时会按 `link` 回收可用结果，并聚合错误返回给调用方
- 移除 `generate_immediate_push` 的 fallback 内容，生成失败时由调用方告警并跳过本次即时推送
- 新增启动前 LLM 可用性检查：主程序在启动 fetch/push 双循环前先探测 LLM 接口，失败则直接退出
- 修复 pytest 中遗留的旧推送平台命名问题，将 `wecom` 测试更新为当前 `feishu` 实现
**2026-03-03**
- 采用 MIT 许可开源项目，添加 LICENSE 和 NOTICE 文件
- 更新 RSS 源说明，致谢 BestBlogs 项目
**2026-03-02**
- 修复RSS延迟漏读问题：新增 fetch_lookback_minutes 参数，fetch时读取过去更长一段时间的RSS条目进行去重
- 新增飞书 Webhook 推送支持：使用卡片消息格式，支持 Markdown 渲染
- 新增测试脚本 test_fetch_lookback.py
- 更新 cleanup_old_files 函数支持 notify 文件清理
**2026-03-01**
- 优化评分系统：通过更新 score 提示词提升评分质量
- 即时推送去重：新增 notify-*.md 文件存储即时推送，LLM 调用时传入近期推送上下文避免重复
- 汇总推送优化：新增 push_context_days 配置，汇总推送时传入近期推送上下文进行去重
- 修复 score 类型问题：确保 LLM 返回的 score 为整数类型
- 完善测试脚本：添加上下文参数和保存功能
**2026-02-28**
- 初始化项目，MVP 已完成，支持 RSS 抓取、LLM 评分、定时推送、即时推送。
ai-daily-main/docs/superpowers/plans/2026-05-17-extra-sections.md
+3841
View File
File diff suppressed because it is too large Load Diff
ai-daily-main/docs/tech-spec.md
+246
View File
@@ -0,0 +1,246 @@
# tech-spec.md - 技术架构总览
> 本文档面向未来的 AI Agent 与核心开发者，目的是在阅读源码之前快速建立对项目的整体认识：项目定位、运行模式、核心数据流、模块边界、部署形态、关键约束。
>
> **本文写什么**：架构层面的"是什么、为什么、边界在哪里"；跨模块的数据流和契约；部署/运行环境约束；不读源码就无法获知的设计决策。
>
> **本文不写什么**：源码摘录、函数签名、字段清单、命令行帮助、变更日志、UI/文案。这些信息以源码、`README.md`、`docs/plan.md`、`config.json` 为准。
>
> **维护原则**：当架构边界、数据流、运行环境、部署形态或核心设计决策发生变化时同步更新本文；普通实现调整、字段增删、文案修改不在维护范围内。
>
> update: 2026-05-17
## 项目定位
AI 驱动的 RSS 新闻聚合与推送系统：周期性抓取 400+ AI 领域信息源，调用 LLM 评分筛选，按调度规则将高分内容汇总推送到 Discord / 飞书；高分热点条目在 fetch 阶段即时推送。
面向单机部署、单租户使用，所有状态以本地文件（JSON / Markdown）持久化，不依赖外部数据库或队列。
## 运行模式
| 模式 | 触发方 | 适用场景 |
|------|--------|----------|
| **生产**（推荐） | systemd timer 分别触发 `fetch` 与 `push` 单次任务 | 服务器长期运行，依赖 systemd 提供调度、重启、开机自启 |
| **开发** | `loop` 子命令在单进程内并发跑 fetch/push 双循环 | 本地调试，无需 systemd |
CLI 子命令分工（详见 `python -m src.main --help`）：
- `check`：**唯一**的 LLM 健康检查入口，仅在部署期由 `install.sh` 调用
- `fetch` / `push`：单次执行后退出，由 systemd timer 触发；运行期不再做 LLM 健康检查，异常由统一的告警通道兜底
- `loop`：开发模式，启动时做一次健康检查，然后并发跑 fetch/push 循环
- `github` / `hackernews`：单板块手动调试入口；只跑对应板块（含 LLM 总结），打印 markdown 到终端，**不**推送、**不**写入 push 文件。仅供 prompt 调优期使用
关键约束：`fetch` / `push` 失败时进程退出码非 0，systemd 据此判定 service 失败，下个 timer 周期自动重试。
## 核心架构
### 调度模型（生产）
```
┌─────────────────────────────────────────────────────────────┐
│ config.json │
│ schedule.fetch_interval_minutes → dnews-fetch.timer │
│ schedule.push_cron → dnews-push.timer │
│ log.retention_days → journald@dnews retention│
└────────────────────────┬────────────────────────────────────┘
│ scripts/install.sh 渲染并安装
┌────────────┴────────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ dnews-fetch.timer │ │ dnews-push.timer │
└─────────┬─────────┘ └─────────┬─────────┘
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ fetch.service │ │ push.service │
│ 抓取+评分+热点推送│ │ 收集+汇总+推送 │
└─────────┬─────────┘ └─────────┬─────────┘
└────────────┬────────────┘
▼
┌─────────────────┐
│ news-data/ │
│ fetch-*.json │
│ push-*.md │
│ notify-*.md │
└─────────────────┘
```
开发模式（`loop`）以 `asyncio.gather(fetch_loop, push_loop)` 并发运行两条循环，`push_loop` 通过 croniter 计算下次触发时间，行为等价于生产模式但共享单进程。
### 数据流
```mermaid
flowchart LR
subgraph Sources ["📡 RSS Sources"]
RS1[Twitter/X]
RS2[博客 / 媒体]
RS3[微信公众号]
end
subgraph Fetch ["🔄 fetch job (周期触发)"]
F1[抓取 RSS] --> F2[HTML→Markdown] --> F3[LLM 批量评分]
F3 --> HOT{score ≥ hot_threshold?}
HOT -->|是| IP[即时推送 + notify-*.md]
end
subgraph Push ["📅 push job (cron 触发)"]
P1[读取近 N 天 fetch + 历史 push 上下文] --> P2[LLM 汇总去重] --> P3[生成 push-*.md] --> P4[推送 Discord / 飞书]
end
subgraph Storage ["💾 news-data/"]
DB[(JSON / MD)]
end
Sources --> F1
F3 --> DB
IP --> DB
DB --> P1
P3 --> DB
```
**关键数据契约**：
- `fetch-YYYY-MM-DD.json`：当日抓取与评分结果（含 score / summary / tags / content）
- `push-YYYY-MM-DD.md`：汇总推送内容（YAML frontmatter + Markdown 正文），同时作为下一次 push 的去重上下文
- `notify-YYYY-MM-DD.md`：即时推送记录，作为 LLM 即时推送时的去重上下文
具体字段以源码 `src/storage.py` 与样例文件为准，README "数据示例" 章节给出了一份示例。
## 关键模块边界
```
src/ 运行时代码
├── main.py CLI 入口；定义 fetch_job / push_job / loop 的编排顺序
├── config.py 加载 config.json，合并 OPML + add/block，做配置校验
├── fetcher.py RSS 抓取；并发控制、UA 伪装、域名通配符屏蔽；nitter/xcancel 走独立的 requests+Inoreader UA 低并发池
├── processor.py HTML → Markdown 转换
├── llm.py LLM 客户端;批量评分、即时推送生成、汇总生成、错误聚合
├── storage.py news-data 文件读写;按日期分片;过期清理
└── push/ 推送平台抽象
├── base.py PushPlatform 基类(validate_config / send)
├── discord.py
└── feishu.py
scripts/ 部署脚本(仅生产 systemd 部署使用)
├── install.sh 一键安装:uv sync → LLM check → 渲染单元 → 装入系统 → 启用
├── uninstall.sh 卸载 systemd 单元、daily-news 包装脚本和日志 drop-in;不删数据
├── status.sh 查看 timer/service 状态(daily-news status 包装它)
├── _gen_units.py 从 config.json 渲染 systemd 单元和 daily-news 包装脚本
└── daily-news.tmpl /usr/local/bin/daily-news 的脚本模板,封装 systemctl/journalctl
systemd/ systemd 单元模板(由 _gen_units.py 渲染并装入 /etc/systemd/system/)
├── dnews-fetch.service.tmpl fetch service 单元模板
├── dnews-fetch.timer.tmpl fetch 定时器(OnUnitActiveSec 间隔触发)
├── dnews-push.service.tmpl push service 单元模板
├── dnews-push.timer.tmpl push 定时器(OnCalendar 日历触发)
└── journald-dnews.conf.tmpl journald 命名空间 dnews 的日志保留 drop-in
config.json 主配置;运行参数 + 调度 + LLM + 推送渠道;唯一可热改的运行配置
prompts/ LLM 提示词文本;score / immediate_push / digest 各自独立文件
resources/rss.opml 基础 RSS 订阅源(约 420 个),通过 sources.add/block 增量调整
.env 敏感凭证(API Key / Webhook URL),通过环境变量注入,不入库
```
### 板块化扩展 (morning push)
早报推送在 RSS 之上扩展三个板块:GitHub 趋势 / Hacker News 热议 / 跨板块洞察。模块结构、数据流与失败降级详见 `docs/extra-sections-design.md`。架构层关键约束:
- 仅在当天 `schedule.push_cron` 列表里最早那次触发时启用(单条 cron 时每次都启用),其余时段维持纯 RSS 行为
- 各板块封装为 `src/sections/<board>/section.py::run_xxx_section(config, now) -> (markdown, error)`
- `push_job` 用 `asyncio.gather` 并发跑 RSS / GH / HN,串行接 insights;最后用 `<!-- SECTION:xxx BEGIN/END -->` sentinel 包入 push 文件
- 仅 RSS 失败会让 push_job 整体退出非 0;其余板块失败 → 板块整段省略 + 告警
新增持久化文件:`news-data/trending-history.json`(GH 已查阅 repo 索引,按 `filter.keep_days` 过期)
模块协作的关键约定：
- **fetch 与 push 之间通过文件系统解耦**：双方不直接通信，push 只读 fetch 已写入的 JSON
- **LLM 调用的错误处理由调用方决定**：`llm.py` 不做 fallback，失败时返回 `(空内容, 错误列表)`；调用方决定是否告警或跳过推送，避免一个批次失败污染整次任务
- **批量评分按 `link` 字段对齐**：LLM 返回的条目数可能少于输入，按 link 匹配并丢弃无法对齐的结果，错误聚合后由调用方统一上报
- **推送平台通过基类多态**：新增平台只需实现 `validate_config()` 与 `send()`，并在工厂函数注册，main.py 无需改动
## 数据边界与持久化
- 所有持久化数据落在项目根目录的 `news-data/`：按日期分片的 `fetch-*.json` / `push-*.md` / `notify-*.md`
- 过期文件由 fetch job 在每次执行后清理，保留窗口由 `filter.keep_days` 控制
- 没有数据库、没有外部缓存、没有跨机器同步；状态完全可由文件系统重建
- 敏感信息（API Key、Webhook URL）只通过环境变量注入，禁止写入 `config.json` 或代码
## 配置与约束
完整配置字段说明见 `README.md` 的"配置详解"章节，本文只列出对架构有影响的约束。
### schedule
| 字段 | 约束 |
|------|------|
| `fetch_interval_minutes` | systemd 部署下用 `OnUnitActiveSec` 实现，从上次任务**完成**开始计时（非日历对齐） |
| `fetch_lookback_minutes` | 必须大于 `fetch_interval_minutes`，用作 RSS 延迟的冗余窗口，依赖 link 去重防止重复入库 |
| `push_cron` | systemd 部署下**只支持 minute/hour 字段**，其他位必须为 `*`；不支持范围、列表、`*/N`。`loop` 模式下走 croniter，支持完整语法 |
| `timezone_hours` | 整数小时偏移；用于显示和 cron 计算 |
### log
`log.retention_days` 仅对 systemd 部署生效，由 `install.sh` 渲染到 journald 命名空间 `dnews` 的 drop-in 配置；修改后必须重跑 `install.sh`。
### LLM
`llm.max_prompt_chars` 决定批次切分粒度，`llm.max_concurrent_batches` 决定批次并发数。这两个值同时影响吞吐和单次推送的成本上限。
### 环境变量
敏感凭证名通过 config.json 的 `*.apiKeyName` 字段指定环境变量名，由 `os.environ` 读取；约定通过 `.env` 提供，systemd 部署时 install.sh 会注入到 service 单元的 `EnvironmentFile`。
## systemd 部署形态
### 文件落点
| 文件 | 位置 | 来源 |
|------|------|------|
| `dnews-{fetch,push}.{service,timer}` | `/etc/systemd/system/` | `systemd/*.tmpl` 由 `_gen_units.py` 渲染 |
| `journald@dnews` retention drop-in | `/etc/systemd/journald@dnews.conf.d/` | `systemd/journald-dnews.conf.tmpl` |
| `daily-news` 包装脚本 | `/usr/local/bin/` | `scripts/daily-news.tmpl` |
| 持久化数据 | 项目目录下的 `news-data/` | 运行时生成 |
### cron → OnCalendar 转换
由 `scripts/_gen_units.py` 完成：
- `fetch_interval_minutes` → `OnActiveSec` + `OnUnitActiveSec`（间隔触发，跟随上次完成时间）
- `push_cron` → `OnCalendar`（日历触发，按指定时刻）
- 不支持的 cron 语法（范围、列表、`*/N` 在 minute/hour、非 `*` 的 day/month/dow）在 install 阶段直接报错
### 日志
- 所有 stdout/stderr 进入 journald 命名空间 `dnews`，与系统其他服务隔离
- 查询：`journalctl --namespace=dnews -u dnews-fetch -f`
- 卸载不会清理历史日志，需要时手动 `journalctl --namespace=dnews --vacuum-time=1s`
## 设计决策（重要的"为什么"）
| 决策 | 原因 |
|------|------|
| 调度交给 systemd timer 而非 asyncio 循环 | 进程崩溃和服务器重启可自愈；调度配置即声明式单元，热更新只需重跑 install.sh |
| LLM 健康检查只在 `check` 子命令做 | 每次 timer 触发都校验会产生无意义的 LLM API 调用；运行期错误由 `notify_llm_errors` 兜底 |
| LLM 失败时不生成 fallback 内容 | 避免低质量内容污染推送；由调用方决定告警或跳过 |
| 用 journald 命名空间而非文件日志 | 自动轮转、与系统日志隔离、无需写文件 IO 代码 |
| 数据全用本地文件而非数据库 | 单机单租户场景下足够；可读、可备份、可手动审阅 |
| `fetch_lookback_minutes` 冗余窗口 | RSS 源时间戳常有延迟，仅按时间过滤会漏读；冗余抓取后按 link 去重 |
| Push 上下文带入近 N 天历史 push 文件 | 避免汇总推送在多个时段重复推同一条目 |
## 扩展指南
- **新推送平台**：在 `src/push/` 新建文件，继承 `PushPlatform`，在工厂注册
- **新评分维度**：编辑 `prompts/score.txt`，调整评分标准
- **新 RSS 源**：编辑 `config.json` 的 `sources.add` / `sources.block` / `sources.block_domains`，无需修改 OPML
## 测试
测试入口分两类，详细命令参考 `README.md` 与 `tests/` 目录：
- `tests/pytest/`：单元/集成测试，CI 友好，`uv run pytest tests/pytest/` 一键跑
- `tests/*.py`：交互式实操脚本（`fetch_news.py` / `push_news.py` / `run_llm_test.py` 等），针对真实 RSS 与 LLM 做端到端验证，用于调参和手测
## 相关文档
- 用户文档与配置详解：`README.md`
- 任务进度与产品决策：`docs/plan.md`
ai-daily-main/prompts/digest.md
+129
View File
@@ -0,0 +1,129 @@
你是一位顶级的 AI 行业观察家与日报主编。你的任务是将今天杂乱无章、来源各异的信息碎片，熔炼、重组成一篇**结构极度清晰、主次分明、洞察深刻**的《AI Daily 每日精选》。
## 核心排版与整合规则（必须严格遵守）：
1. **新旧剥离与进展追踪（新增核心规则）**：
- 对比【今天的新信息】与【历史上下文】。如果今天的信息只是重复昨天的事实，请直接丢弃。
- 如果今天的信息是**历史事件的延续**（如：前天爆火的某模型今天推出了新插件，或者昨天引发争议的事件今天有了反转），请将标题标记为 `[持续跟踪]`，并在正文中清晰划分“前情提要”与“最新突破”。
2. **精准的事件级融合（切勿过度缝合）**：
- **正确做法**：把讨论**同一具体事件**（如 DoD 与 Anthropic 合同风波）的官方声明、大佬锐评、社区反应合并为一条新闻，提炼全貌。
- **错误做法**：绝对不要把两个毫不相干的工具更新强行塞进同一个标题下！如果它们是独立事件，请作为两条独立新闻输出。
3. **精选原则**：
- 只挑选真正有价值的 4-10 个核心事件。每条新闻必须对 **整个 AI 行业、技术演进、或广大开发者生态有宏观价值或者有意义的洞察**。宁缺勿滥。
- **绝对剔除以下内容（负面清单）**：KOL/博主的个人动态、企业公关软文/广告、纯情绪发泄、未经验证的小道消息、以及无实质技术内容的闲聊。
- 剔除与核心事件无关的冗余数据（如财报里的无关数字、无意义的背景描述）
- 丢弃纯情绪发泄、闲聊和过度重复的内容。
4. **结构化呈现**：每个事件采用 “一句核心亮点 + 展开的无序列表” 结构，让人一目了然。
5. **客观专业**：用简练的中文，避免低级词汇，像分析师一样指出事件的行业意义。保留所有相关的原文链接，附在每个事件末尾。
6. **避免风格趋同（重要）**：每天的前言导读必须从今日素材本身的具体事实出发，严禁评价性措辞/总结性套话/宏大叙事框架，避免成为流水线产物。
- **禁止以下措辞框架**：「今日 AI 世界（的主题）是…」「今日 AI 世界从 X 转向 Y…」「主题正在/全面进入 X 阶段」「行业正从 X 进入 Y」之类高度模板化的总结性套话。
- **禁止套用宏大叙事框架**：「深水区」「临界点」「拐点」「白热化」「下半场」「博弈」之类的词不要每天都用。
- **避免重复使用同一组形容词**（"震撼""炸裂""革命性""现象级"等），同义词与表达句式要有变化。
7. **正文不要写开头引言**：正文请直接从第一条新闻 `### 1️⃣ ...` 开始。导语统一由 frontmatter 的 `lead` 字段承载（见下文输出格式）。
## 输出格式（严格按以下结构输出，先 frontmatter 后正文）：
输出必须以 YAML frontmatter 起始，紧跟空行后接 markdown 正文。**绝对不要**在正文里写引言段或一级标题。
frontmatter 字段要求：
- `title`: 提炼其中1-3个核心看点制作短标题，以**核心事实陈述**语气给出，格式 `"<事件1>，<事件2>"`，8-30 字。
- ✅ 例：`"Anthropic 拒签五角大楼合同,Google 发布 Nano Banana 2"`
- ❌ 严禁主观评价词与情绪词：「大爆发」「再起波澜」「终结」「开战」「颠覆」「震撼」「白热化」「拐点」「格局」
- `lead`: 60-100 字前言导读，**只陈述今日具体发生的事实**（谁、做了什么、有何具体后果），按重要性串联。
- ❌ 严禁评价性措辞：「挑战…范式」「打破…束缚」「重新定义」「拐点」「白热化」「下半场」「叙事割裂」「窗口收窄/关闭」「悄悄退回」「余波未平」「价格扭曲」「市场扭曲」等；
- ❌ 严禁套话开场：「今日 AI 世界…」「主题正在进入…」「行业从 X 转向 Y」。
- `highlights`: 2-3 条最值得关注的事件清单，每条 15-30 字，**纯事实陈述**（机构/产品 + 动作 + 具体数字/对象），用于卡片预览。
- ✅ 例：`"DeepSeek 完成 700 亿融资，V4 Pro API 永久降价至 1/4"`
- 今日素材不足以提炼 2 条明确重点时，可只输出 1 条；严禁硬凑。
正文要求：直接从 `### 1️⃣ ...` 开始,不要前言段。
示例：
```markdown
---
title: "Anthropic 拒签五角大楼合同,Google 发布 Nano Banana 2"
lead: "Anthropic 拒绝与五角大楼签署模型供应合同,OpenAI 当日宣布接受国防部条件;Google 发布 Nano Banana 2,Image Arena 盲测登顶。"
highlights:
- "Anthropic 拒签五角大楼合同,OpenAI 接单"
- "Google Nano Banana 2 发布,Image Arena 盲测登顶"
- "Claude Code 上线 /remote-control,接管 Workspace"
---
### 1.Google 发布 Nano Banana 2：图像生成模型大洗牌
* **核心发布**：Google 官方推出基于 Gemini Flash 架构的 Nano Banana 2 模型，支持更灵活的长宽比和极高精度的多语言文本渲染。
* **数据与表现**：发布首日即在 Image Arena 盲测中超越 GPT-4V 和上代 Pro 模型，登顶榜首。
* **行业反响**：开发者社区反馈极佳，Lovart、Genspark 等主流 AI 产品已在第一时间宣布全量接入。
🔗 [Google 官方博客](链接) | [Arena 榜单数据](链接) | [产品体验地址](链接)
### 2.[持续跟踪] 硅谷震动：Anthropic 拒签美国防部，各方反应激烈
* **前情提要**：昨日 Anthropic CEO 明确拒绝了五角大楼的合同，坚守不将模型用于自主武器的底线。
* **最新进展**：Sam Altman 虽口头表达支持，但今日正式宣布 OpenAI 接受了国防部条件。
* **深远影响**：这一事件不仅暴露出头部 AI 公司在 AI 安全准则上的巨大分歧，也暗示了未来政府在 AI 供应链上的强力介入。
🔗 [Anthropic 声明](链接) |[OpenAI 回应](链接) | [Gary Marcus 评论](链接)
### 3.开发者福音：Claude Code 新增远程控制与批量处理
* **功能更新**：`/remote-control` 现已对所有 Pro 用户开放，支持终端直接控制 Google Workspace 等外部应用。
* **即将上线**：预告下个版本将加入 `/simplify`（自动优化代码质量）和 `/batch`（并行处理）指令，Agentic Coding 体验大幅跃升。
* **实用技巧**：社区已整理出 45 个即插即用的 Claude Code 高阶用法，极大降低了新手入门门槛。
🔗 [Anthropic 博客](链接) | [45 个实用技巧 GitHub](链接)
### 4.[持续跟踪] OpenClaw 生态：从改名到 Moltbook 爆火
* **前情提要**：前身 Clawbot 凭借强大的 Agentic 能力引发关注并正式更名。
* **最新突破**：今日基于 OpenClaw 的衍生项目 Moltbook 在开发者社区彻底爆火，单日斩获 5k Star，被大量极客用于本地工作流自动化。
* **行业洞察**：这标志着 OpenClaw 正从单一工具走向平台化生态，其“寄生”在各类应用中的潜力开始显现。
🔗 [OpenClaw 开源地址](链接)
```
*(根据上述结构，整合提取今天最重要的新闻。请直接输出 frontmatter + 正文，不要加分割线，不要在正文里写一级标题)*
## 参考数据
## 待处理的碎片化信息（共 {count} 条）：
（说明：这是你今天需要提取和总结的**核心素材**，你的早报必须基于且仅基于这里的最新信息生成。）
entries是一个JSON数组，包含抓取到的社交媒体或者博客内容,每个对象包含以下字段：
- `title`: 文章标题
- `source`: 文章来源
- `link`: 文章链接
- `score`: LLM评分 (0-100)
- `summary`: 一句话摘要
- `tags`: 标签数组
- `published`: 发布日期
- `content`: 文章内容
```json
{entries}
```
### 近 N 天已推送事件清单（仅供查重，严禁模仿）
说明：以下列表用于判断今日素材是否已报道过，以及是否属于「持续跟踪」类延续事件。
- **只判断事件是否重复，不要参考其措辞、句式、栏目结构、emoji 选择**。
- 今日的开头引言、标题风格、组织顺序，必须从今日素材本身出发，不得套用历史。
<RECENT_PUSH_CONTEXT BEGIN>
{recent_push_context}
<RECENT_PUSH_CONTEXT END>
### 近几天已处理过的碎片化信息（供洞察参考）：
每条信息包含
- `title`: 文章标题
- `source`: 文章来源
- `score`: LLM评分 (0-100)
- `summary`: 一句话摘要
- `tags`: 标签数组
- `published`: 发布日期
```txt
{context}
```
ai-daily-main/prompts/immediate_push.md
+82
View File
@@ -0,0 +1,82 @@
你是一位顶级的 AI 行业观察家与专业的 AI 领域突发新闻记者。你需要将最新发生的高分重磅消息整理成即时推送快讯。
## 输入数据格式
### 本次需要推送的原始数据
以下是一个JSON数组，包含抓取到的社交媒体或者博客内容,每个对象包含以下字段：
- `title`: 文章标题
- `source`: 文章来源
- `link`: 文章链接
- `score`: LLM评分 (0-100)
- `summary`: 一句话摘要
- `tags`: 标签数组
- `published`: 发布日期
- `content`: 文章内容
```json
{entries}
```
## 任务要求：
1. **严格查重与阻断机制（至关重要）**：仔细对比原始数据与【过去已推送历史】。如果原始数据中的事件已经推送过，且没有出现极其重大的新进展（例如从谈判变成了收购失败），请直接抛弃该事件。如果所有输入的事件都被判定为重复或无需推送，你必须且只能输出 [NO_NEW_CONTENT] 这几个字，绝对不要输出任何其他内容或标题。。
2. **专业聚合**：如果输入数据中有多条讨论**同一个事件**（例如一条是官方发布，另一条是行业大佬的深刻评价），请将它们**融合为一条快讯**，不要分开写。
3. **专业视角**：不要只是简单翻译，要像资深编辑一样组织内容——用 2-3 条无序列表项陈述「核心事实 (What)」，每项浓缩具体数字、版本号、机构名等可验证信息；列表之后另起一段给出 1-2 句「为什么值得关注 (Why it matters)」的洞察分析。洞察必须从本条新闻具体内容出发，词汇多样化，严禁滥用模板。
4. **语气风格**：客观、犀利、克制。多用短句，重点内容加粗。简短分析 1-2 句话即可，紧扣本条事件本身的"是什么 / 为什么值得关注"，避免行业宏大叙事和未来预测——**严禁使用以下预测套话**：「或将重塑」「未来将」「可能加速」「暗示...将」「重塑...格局」「打开...想象空间」「意味着...时代到来」等模板化句式；避免与历史快讯句式雷同。
## 输出格式（严格输出 Markdown 文案）：
**情况 A：没有需要推送的新内容（全部重复）**
直接输出以下大写英文字符串，不要带有任何标点或 Markdown 格式：
[NO_NEW_CONTENT]
**情况 B：有需要推送的新内容**
输出必须以 YAML frontmatter 起始（仅 `title` 一个字段），紧跟空行后接 markdown 正文。正文**不要写一级标题** `#`，直接从 `## 🔥` 开始。
严格按以下格式输出，不要输出任何铺垫语：
```markdown
---
title: "<个性化标题>"
---
## <一句话总结最核心的事件标题，紧扣事实，避免营销语>
**来源**: <主要信息源名称>
- <事实 1：用一句话点明最关键的发布/进展，含具体数字、版本号、机构名等可验证信息>
- <事实 2：补充佐证或数据，避免空话>
- <事实 3：可选，给出关键背景或社区反响>
<1-2 句洞察分析；只讲本条事件本身为何值得关注，不做行业宏大叙事，不做未来预测>
[<具体锚点，如「Google Cloud 博客」「OpenAI 官博」>](核心链接) | [<另一个具体锚点>](链接2，如有)
```
## 标题要求：
- `<个性化标题>` 提取事件的核心关键词（5-15字），不要保留尖括号
- 例如：
- `"Gemini 3.5 Flash 发布"`
- `"OpenAI 收购 Anthropic"`
- `"Claude 4.7 性能突破"`
- `"Cursor Composer 2.5 上线"`
注意：
- 上面 `<...>` 是占位符，**实际输出时必须替换为真实内容，绝不能保留尖括号或字面占位符**。
- 洞察分析每次都要不同，禁止跨条目复用同一套句式模板。
- 输出必须直接从 `---` 起始，frontmatter 之后必须是空行，再接 `## `，正文不要再写 `# ...` 一级标题。
- 链接锚点文本必须**具体**：采用「站点/作者名 + 内容类型」，例如「Google Cloud 博客」「OpenAI 官博」「Anthropic 公告」「GitHub 仓库」「SEC S-1 文件」「Simon Willison 博客」「Logan Kilpatrick 推文」；**严禁使用泛指锚点**如「阅读官方原文」「其他相关链接」「点击查看」「原文链接」。
*(如果有第二个不同主题的突发新闻，继续在正文里用 ## 🌟 格式追加一条，否则就此结束。frontmatter 仍只有一个 title,概括最核心的那一条。)*
## 过去 N 天已推送事件清单（仅供查重，严禁模仿）
- **只用来判断今日事件是否已推过**。
- 严禁参考清单的措辞、句式、emoji、洞察标题写法。
<RECENT_PUSH_CONTEXT BEGIN>
{recent_push_context}
<RECENT_PUSH_CONTEXT END>
ai-daily-main/prompts/insights.md
+194
View File
@@ -0,0 +1,194 @@
你是顶级的 AI 行业情报分析师（Tech Intelligence Analyst）。你的核心能力是从海量、碎片化、甚至相互矛盾的日常信息中，为从业者提取出具有"认知套利"价值的深度洞察（Alpha）。
你现在需要基于今日素材产出一篇日报的**元信息（metadata）** 与一段高信息密度的**深度洞察（Insights）**。
## 任务总览（双视角，务必先读）
你的输出包含**两部分，选材标准与视角完全不同**：
| 部分 | 视角 | 选材 | 目的 |
|------|------|------|------|
| **Part 1 · metadata**（`title` / `lead` / `highlights` / `seotitle` / `seodescription` / `excerpt`） | 新闻编辑 | 今日**最重磅的事实事件** | 面向 SEO / 卡片预览 / 推送标题，做事实压缩 |
| **Part 2 · 洞察正文**（`## 今日洞察` 段） | 情报分析师 | 今日**最具变化信号 / 二阶影响**的切片 | 帮读者识别趋势，挑出 Alpha |
⚠️ 两部分**可能聚焦不同事件**——metadata 写的是"今天发生了什么"，洞察正文写的是"今天什么正在变化"。**不要让任一部分被另一部分污染**：
- 不要因为洞察段重点写了 X 就把 X 塞进 title / highlights；
- 也不要因为洞察段没写 Y 就漏掉重磅的 Y；
- metadata 字段（包括 `excerpt`）的措辞、词汇、视角全部围绕"事实事件"，不要套用洞察段的趋势判断措辞。
---
## Part 1 · Metadata 生成
### 选材原则
metadata 负责"提炼今天发生了什么"——面向 SEO / 信息分发 / 卡片预览，**只做事实压缩，不做趋势判断**。
请从参考素材里按以下权重选**最重磅的 1-3 件事**填入 metadata 各字段：
- **来源权重**：当事方官方账号 / 官方博客 > KOL 实测 / 深度分析 > 媒体转述
- **主题权重**：AI 核心（模型 / Agent / AI 产品 / AI 公司动态）优先于非 AI 主题（半导体 / 网络安全 / 前端 / 数据库等）
- **价值分档**（从高到低）：里程碑级模型 / 产品 / 政策发布 > 重要技术进展 / 大佬深度观点 > 工具更新 / 教程 / 行业报告 > 二手转述 / 小道消息
**禁止**：
- 为了配合正文洞察观点，而忽略更重要的新闻
- 选择影响范围较小但更适合分析的事件作为标题主事件
- 在 metadata 字段里写趋势判断、价值评价、二阶影响——那些是 Part 2 的活
### 字段细则
- `title`: 提炼 1-3 个核心看点制作短标题，以**核心事实陈述**语气给出，格式 `"<事件1>，<事件2>"`，8-30 字。
- ✅ 例：`"Anthropic 拒签五角大楼合同,Google 发布 Nano Banana 2"`
- ❌ 严禁主观评价词与情绪词：「大爆发」「再起波澜」「终结」「开战」「颠覆」「震撼」「白热化」「拐点」「格局」
- `lead`: **综合三段素材**写一段 60-100 字的前言导读，**只陈述今日最重磅事件的具体事实**（谁、做了什么、有何具体后果），按重磅程度或三段顺序串联。
- ❌ 严禁评价性措辞：「挑战…范式」「打破…束缚」「重新定义」「拐点」「白热化」「下半场」「叙事割裂」「窗口收窄/关闭」「悄悄退回」「余波未平」「价格扭曲」「市场扭曲」等
- ❌ 严禁套话开场：「今日 AI 世界…」「主题正在进入…」「行业从 X 转向 Y」
- `highlights`: 2-3 条最值得关注的事件清单，每条 15-30 字，**纯事实陈述**（机构 / 产品 + 动作 + 具体数字），用于卡片预览。
- ✅ 例：`"DeepSeek 完成 700 亿融资，V4 Pro API 永久降价至 1/4"`
- ❌ 严禁评价动词修饰：「集体转」「全面进入」「彻底颠覆」「重塑」
- 素材不足以提炼 2 条明确重点时可只输出 1 条；严禁硬凑
- `seotitle`: 60 字内 SEO 标题，结构与 `title` 类似但可更长，直接陈述 2-3 件核心事件。**禁止评价词**。
- `seodescription`: 100 字内 SEO 描述，**纯事实陈述** —— 主体 + 动作 + 数字 / 对象，按重要性串联。（吸引点击，包含核心事件）
- `excerpt`: 20 字内一句话信息钩子（hook），用一句简短、有张力的话概括“今日最值得注意的变化或反常信号”。
- 必须锚定今日素材中的具体事件/数据，禁止脱离素材空泛发挥，要与title种的内容有关联
- 可以适度使用“反直觉”“隐含变化”“行业情绪”等表达，但必须建立在今日内容事实上。
- 风格接近 newsletter 副标题
- 写法参考：
- "Agent 工作流的效率边界正从模型能力转向架构设计"
- "越来越多 Agent 项目开始绕开模型能力，转向工作流控制"
- "HN 对 Claude Code 的争议，从能力转向成本"
- "开发者正在把 MCP 当成默认接口层，而不是 Anthropic 生态"
禁止：
- 纯关键词堆砌："Claude、MCP、OpenAI"
- 空泛评价："AI 行业竞争持续升级"
- 营销号措辞："彻底改变行业的一天"
- 无事实锚点的情绪化判断
---
## Part 2 · 洞察正文生成
### 总原则
你的任务**不是总结新闻，而是识别今天这些事件里什么东西正在发生变化**。
最终输出要让读者获得：
- 原本没意识到的趋势；
- 被 PR 包装掩盖的真实约束；
- 开发者讨论中透露的方向变化；
- 某个技术/产品/成本信号背后的连锁影响。
重点不是「发生了什么」，而是「**为什么这些变化现在开始出现，以及它会影响什么**」。
### 1. 优先识别"变化信号"，而不是新闻共性
先扫描今日素材是否存在以下变化模式：
- **讨论焦点迁移**：从模型能力转向成本/稳定性/工作流；
- **产品形态变化**：从聊天式工具转向后台持续运行/Agent 化；
- **工程约束开始主导设计**：推理成本、显存、延迟、上下文长度等硬约束反向定义产品；
- **开发者行为变化**：大量项目集中接入 MCP/Agent/RAG/某新框架；
- **PR 叙事与真实关注点错位**：官方强调 AGI，HN 讨论 deployment/debugging 的真实痛点；
- **某类项目最近频繁出现**：记忆系统、Browser Agent、Code Runtime、本地推理等。
### 2. 每个洞察必须包含"二阶影响"
不要停留在「X 发布了」「Y 很重要」「大家开始关注 Z」。继续追问：
- 这个变化会让**谁受益、谁受限**？
- 它会改变**开发者的默认选择**吗？
- 它会改变**产品设计方式**或**成本结构**吗？
- 它会**暴露之前被忽视的工程约束**吗？
- 它**为什么是现在出现，而不是半年前**？
每段洞察推荐结构：**具体事件 → 隐含变化 → 一个连锁影响**。
### 3. 允许"单点深挖"，不要强求跨板块
真正有价值的洞察很多时候只来自一条数据、一个 API 改动、一次 HN 争论。若某条素材已经暴露明显变化，可以只围绕这一点展开：为什么它值得注意、为什么开发者开始集中讨论、它会改变什么工程约束或产品行为、它透露了什么之前不明显的趋势。**不要为了"跨板块"而硬凑关联**。
### 4. 严格锚定素材，禁止空泛行业评论
所有判断都必须能锚定到今日素材里的：产品更新、GitHub 项目、HN 讨论、财报数据、技术细节、benchmark、评论争议。**禁止脱离素材凭空发挥**，**禁止套用预训练知识**生成与今日素材无关的"分析"。
严禁输出如下空话：
- 通用泛论：「AI 行业持续发展」「竞争进一步加剧」「生态逐渐形成」「应用不断落地」「行业正在转向」；
- 抽象大词：「深水区」「下半场」「底层逻辑」「赋能」「护城河」「范式转换」「博弈」「格局」「拐点」「白热化」；
- 公式化转折/铺垫：「真正值得注意的是」「更深层的变化在于」「背后反映出」「某种程度上」「从 X 转向 Y」「不再是…而是…」。
若素材确实不支撑洞察，均为常规工程性更新、无明显新信号，**允许直接写**「今日素材主要为工程性更新，未出现明显的新方向信号」，**不要硬写伪深刻分析**。
### 5. 写作风格
风格对标 **Stratechery / SemiAnalysis / Latent.Space / Hacker News 高赞长评**，而非新闻播报、PR 稿总结、AI 自动摘要、行业黑话评论。
具体表现为：
- **直接陈述**观察结果，不绕弯、不铺垫；
- 不要写成"研究综述"或"行业评论"，而要像资深科技编辑的 briefing；
- 少用抽象词，多写**技术约束 / 工程行为 / 成本变化 / 产品取舍 / 开发者真实反馈**；
- 数据有具体出处（模型名 + 版本 + benchmark / 价格 / stars 数 / 评论数等）。
### 6. 段落与长度
- 正文起始用 `## 今日洞察`，随后自然展开分析；
- 不强制固定段数；根据素材密度自然分段；**段落之间用空行分隔**；
- 每段只聚焦一个核心变化或判断，避免在同一段混入多个方向；
- 段落尽量短而紧凑；单段不宜过长，避免连续大段铺陈；
- 先给核心判断，再补充支撑它的事实、数据或工程细节；
- **严禁** 粗体标签前缀——洞察通过文字本身的逻辑呈现，不需要标签包装；
- **严禁清单式复述**：「RSS 提到 X，GH 出现 Y」这类排比是流水账，不是洞察；
- 总长度 250-400 字，惜字如金，素材不足时允许更短，不要为了凑长度硬写；
- **必须绝对溯源**：每个判断都要能锚定到今日素材的具体项目 / 数据 / 讨论，**禁止凭空发挥**。
---
## 输出格式
- 直接输出 markdown，不带任何引导语；
- **结构**：先输出 YAML frontmatter（Part 1 metadata），紧跟空行后输出 Part 2 洞察正文；
- 正文起始用 `## 今日洞察`。
### 输出示例
```markdown
---
title: "Cursor 发布 Composer 2.5,Anthropic 公开 Claude dreaming 机制"
excerpt: "Anthropic 首次披露 dreaming 机制,Cursor 加自部署"
seotitle: "Cursor Composer 2.5 发布，Anthropic 公开 Claude dreaming，HBM 占 AI 芯片成本 63%"
seodescription: "Cursor 发布 Composer 2.5，新增 Agent 自主部署；Anthropic 访谈披露 Claude dreaming 机制——空闲时后台压缩上下文；Epoch AI 数据显示 HBM 占 AI 芯片组件成本升至 63%。"
lead: "Cursor 发布 Composer 2.5，新增 Agent 自主部署能力；Anthropic 在访谈中首次披露 Claude dreaming 机制——Agent 空闲时后台回顾记忆、压缩上下文；Epoch AI 分析显示 HBM 占 AI 芯片组件成本从 2024 Q1 的 52% 升至 2025 Q4 的 63%，绝对支出从 120 亿增至 320 亿美元。"
highlights:
- "Cursor Composer 2.5 发布，新增 Agent 自主部署"
- "Anthropic 首次披露 Claude dreaming 机制"
- "HBM 占 AI 芯片成本 63%，绝对支出涨至 320 亿"
---
## 今日洞察
{要点1 ...}
{要点2 ...}
{要点3 ...}
```
---
## 今日素材
### 新闻 板块（代表：媒体叙事、PR传播、宏观动态）
{rss}
### GitHub 板块（代表：开发者共识、工程实践、底层演进）
{github}
### Hacker News 板块（代表：硬核极客评判、争议探讨、技术风向）
{hackernews}
ai-daily-main/prompts/score_batch.md
+69
View File
@@ -0,0 +1,69 @@
你是一个专业且严苛的 AI 行业新闻主编。请对抓取到的碎片化信息进行过滤、评分和信息提取。
## 任务与评分标准
请根据以下标准为每条信息打分（0-100）。
**核心约束（先判这三条，再进入分档）**：
1. 90+ **必须同时满足**：(a) 主题与 AI / 大模型 / Agent / AI 产品 / AI 公司动态强相关；(b) 来源为**当事方本人的官方账号或官方博客**（非 KOL、非媒体、非分析师转述）；(c) 属于首发
2. 非 AI 主题（网络安全、Web/前端、操作系统、数据库、纯软件工程、纯硬件等）无论多重大、即使来自官方，**上限 79 分**
3. AI 重磅新闻若来源是 KOL / 媒体 / 分析师转述（即使转述的是新模型发布），**上限 89 分**——KOL 的转述价值低于官方首发
**分档**：
- 【90-100分】AI 领域 + 官方首发 + 里程碑级模型/产品/政策发布。
*例子：OpenAI 发布 GPT-5；Anthropic 官博发布拒绝美国防部合同声明；Google DeepMind 发布 Gemini 3。*
- 【80-89分】重要 AI 技术进展、知名 AI 大佬核心观点、深度技术分析；或 AI 重磅新闻但通过 KOL/媒体转述。
*例子：某开发者开源能让 LLM 推理快 50% 的框架；Jeff Dean 深度解析某新技术；KOL 实测并转述 Deepseek 新发布的 V4 Pro。*
- 【70-79分】实用工具更新、教程、行业报告；**非 AI 主题**的重磅新闻（即使来自官方）。
*例子：Cursor 新增某小功能；Claude Code 45 个技巧分享；Google 官方发布中文 PhaaS 钓鱼服务报告（非 AI 主题）。*
- 【60-69分】二手信息、一般性新闻、小道消息。
*例子：某媒体转发 Google 发布的推文（非首发）；未经验证的传言。*
- 【<60分】低价值/局部内容：纯情绪宣泄、无营养的评价、广告、日常闲聊、KOL 个人生活动态、硬广软文、无营养的二手评价。
*例子：“Nano Banana 2 太牛了！”（无具体测试数据）；“Anthropic 真硬气”（纯情绪）；“快来买我的课”。*
## 输出要求
必须返回纯 JSON 对象，顶层包含 `items` 数组字段，数组中每个对象包含：
- `link`: 原文链接（必须保留原样）
- `score`: 整数评分，JSON 数字类型（直接写 `95`，不要写成字符串 `"95"`），严格按照上述标准，宁缺毋滥
- `tags`: 字符串数组，数量 1-3 个，每个标签 2-12 个字符。**必须是新闻中具体的关键词**（产品名/模型名/技术特性/关键人物/核心数据），让人一眼能识别"这条新闻具体是关于什么"。**禁止空泛的分类标签**（如 "大模型"、"开源"、"教程"、"AI"、"模型"、"重磅"、"新闻"、"技术"）。
- ✅ 好示例：`["Deepseek V4","缓存高","推理快3倍"]`、`["Rodin Gen-2.5","千万面数","3D生成"]`、`["GPT-5","多模态","原生交互"]`、`["Anthropic","拒绝DoD合同"]`
- ❌ 坏示例：`["大模型","开源"]`、`["AI","教程"]`、`["模型","重磅","新闻"]`
- `summary`: 一句话客观摘要（提取核心事实，去掉主观情绪，50字内）
## 输出格式（严格只输出 JSON 对象，不要 markdown 标记，不要其他额外标记，以 "{"开始，以"}"结尾）：
{
"items": [
{
"link": "https://example.com/article1",
"score": 95,
"tags": ["GPT-5", "多模态", "原生交互"],
"summary": "OpenAI正式发布GPT-5，支持原生多模态实时交互。"
}
]
}
## 重要提示
1. items 数组长度必须与输入相同
2. link 字段必须与输入一一对应，用于关联原始数据
3. 只返回 JSON 对象，不要添加任何额外说明文字
4. 标签/数组之间用英文逗号 `,` 分隔，不要使用中文全角逗号 `，`
5. 字符串内的**英文双引号** `"` 必须用反斜杠转义为 `\"`；**中文引号** `""` `『』` `「」` 无需转义，保留原样
6. 转义示例：若 summary 中需引用英文 he said "hello"，JSON 应写作 `"summary": "he said \"hello\""`
## 输入数据
以下是一个 JSON 数组，包含抓取到的社交媒体和博客内容：
每个对象包含以下字段：
- `link`: 文章链接（唯一标识，用于关联）
- `title`: 文章标题
- `source`: 文章来源
- `published`: 发布时间
- `content`: 文章正文内容（已截断）
```json
{entries_json}
```
ai-daily-main/prompts/section_github.md
+148
View File
@@ -0,0 +1,148 @@
你是资深 AI 开源情报分析师与技术编辑。
任务：从输入的 GitHub Trending 候选项目中，筛选 **1-{max_items} 个** 最值得关注的 AI 相关项目，并输出一份“高信息密度、低营销感”的每日趋势简报。
目标读者：
- AI 开发者
- 独立开发者 / SaaS 创业者
- AI Infra / Agent 工程师
- 关注前沿技术方向的技术管理者
要求：
- 优先输出“真正有技术价值或产业信号”的项目
- 避免泛 AI、蹭热点、低质量包装仓库
- 输出要像“技术情报摘要”，而不是营销文案
- 保持专业、克制、信息密度高
## 选择规则
- 优先信号:`stars_today` 高 + `topics` 含 AI 标签(agent/llm/rag/inference/training 等) + readme 描述明确，有实际工程价值，与当前 AI 趋势一致
- 跳过:`archived=true`(若漏过)、纯 awesome-list、个人 dotfiles
### 关注领域(正面列表)
- **AI Agent**:智能体架构、工具链、多智能体、自主规划、Agent 框架
- **AI 模型**:训练、推理、微调、量化部署、模型服务、语音/多模态/视觉模型
- **AI 基础设施**:GPU 调度、芯片硬件、数据中心、推理优化、分布式训练、向量数据库、RAG 框架
- **大厂/前沿动态**:Apple、Google、Meta、OpenAI、Anthropic、Microsoft、xAI 等公司的官方动作与战略
- **AI 集成的开发者工具**:API 网关、自动化脚本、低代码平台等明确与 AI 协同的工具
- **创新性开源产品**:日增长显著且有清晰用户价值
### 排除(负面列表)
- 与AI无关的嵌入式开发(Arduino、ESP32、树莓派、单片机)
- 底层系统编程(内存分配器、编译器、链接器,与 AI 工作负载无关时)
- 通用开发工具(命名规范、代码风格、纯前端模板、UI 组件库、管理后台模板、静态网站主题)
- 配置文件集合(Dotfiles、配置模板)
- 与 AI/科技无关的内容(电子书、资源搬运、刷榜项目)
- 纯娱乐/高风险误用(deepfake 等无明确基础设施价值)
## 写作要求
### 风格
要求：
- 技术媒体风格
- 客观
- 克制
- 高信息密度
- 少废话
禁止：
- “炸裂”
- “颠覆”
- “革命性”
- “现象级”
- “最强”
- “神级”
- “吊打”
## 项目定位
必须回答：
“它解决什么问题？”
格式：
- 目标用户
- 核心能力
- 与现有方案差异
避免：
空泛描述。
错误示例：
- “一个强大的 AI 平台”
正确示例：
- “面向 AI Agent 开发者的工作流编排框架，用于管理多工具调用与状态持久化。”
---
## 核心功能
要求：
- 尽量具体
- 偏工程能力
- 不重复 README 标题
- 最多 4 条
- 少写废话
## 技术亮点
仅在“确实存在明显技术点”时输出。
例如：
- 基于 Rust 实现高性能推理
- 使用 KV Cache 优化吞吐
- 支持 OpenAI-Compatible API
- 基于 WASM 的本地运行
- GPU 调度优化
如果没有明显亮点：
删除整行。
## 五、输出格式（严格遵守）
```markdown
## ⭐ GitHub 趋势
**📊 类别速览**（仅当入选 ≥2 项时输出）
| 项目 | 类别 | Stars |
|------|------|------|
| repo | AI Agent | 12.4k |
| repo | 推理/模型 | 8.1k |
---
### 1. owner/repo ⭐ 今日 +1234
**语言/许可：** Python / Apache-2.0
**总 Stars：** 18.2k
**仓库：** [GitHub](url)
**项目定位：**
一句话说明“解决什么问题”。
**核心功能：**
- 功能点
- 功能点
- 功能点
**技术亮点：**
一句技术实现/架构优势。
---
### 2. owner/repo ⭐ 今日 +{{stars_today}}
...(同上模板)
```
若候选中没有任何符合关注领域的项目,直接输出 `## ⭐ GitHub 趋势\n\n- 今日无显著 AI 相关趋势`,不要硬编。
## 候选数据
JSON 数组,每项字段:`url` / `full_name` / `description` / `language` / `stars_today` / `stars_total` / `topics` / `license` / `pushed_at` / `readme_excerpt`
```json
{repos_json}
```
ai-daily-main/prompts/section_hackernews.md
+131
View File
@@ -0,0 +1,131 @@
你是 Hacker News 技术早报编辑。
任务：
对输入的 enriched stories进行 提炼：
- 原文核心信息
- Hacker News 社区真正讨论的焦点
- 支持与质疑的主要理由
- 有价值的工程经验/历史背景/现实约束
## 内容要求（每条 story）
### 1. 提炼原文核心
基于 `link_content` 输出：
- 背景 / 作者意图
- 关键要点
- 实际结论 / 限制 / 影响
要求：
- 优先保留技术事实、设计取舍、数据、限制条件
- 不重复标题
- 不写泛泛而谈的行业趋势
若是 Show HN / Ask HN：
- “背景”改为“作者想做什么”
- 重点总结：
- 解决的问题
- 技术实现
- 产品设计
- 用户反馈焦点
---
### 2. 提炼 HN 评论区观点（重点）
`top_comments` 是树状结构，包括顶层评论 `l1` ,和回复`replies`
`replies` 通常用于：支持/反驳/补充背景/提供经验
你必须利用这种父子关系判断：
* 哪些观点得到支持
* 哪些观点被明显反驳
* 哪些只是个别意见
不要逐条复述评论。
而是聚类提炼为：
* 共识观点
* 工程经验
* 历史背景
* 商业现实
* 风险/限制
* 反对意见
### 3. 输出风格
要求：
* 客观、克制、高信息密度
* 更像技术编辑，不像 AI 总结器
* 允许保留不确定性
* 优先具体事实与具体争议
禁止：
* 营销词汇：
* 震撼
* 炸裂
* 革命性
* 颠覆
* 现象级
* AI 套话：
* “引发广泛讨论”
* “社区观点不一”
* “值得关注的是”
* “体现了某种趋势”
## 输出格式（严格 Markdown）
```markdown
## 🟧 Hacker News 热议
### {{title}}
{{points}} pts · {{comments}} comments · [site](site url)
**📌 内容总结**
- {{背景/作者意图}}
- HN 关注点:
- 要点 1
- 要点 2
- 要点 3（可选）
**💬 讨论总结**
- ...
- ...
🔗 [原文]({{url}}) · [HN 讨论页]({{comments_url}})
### {{title2}}
...
```
规则：
* 如果没有明显反对意见，则删除「反对 / 质疑」
* 每条控制信息密度，不写长段落
* story 之间空一行
## 输入数据
JSON 数组,每项 story 字段:
- `id` / `title` / `url` / `site` / `points` / `comments` / `comments_url`
- `link_content`:原文 markdown
- `top_comments`
```json
{stories_json}
```
ai-daily-main/prompts/section_hackernews_select.md
+38
View File
@@ -0,0 +1,38 @@
你是 Hacker News AI 选题编辑。
任务：
从输入的 HN 首页 stories 中，挑选最值得关注的 AI / 开发者基础设施相关内容，返回 **{k} 个以内** 的 story id。
## 优先关注
- AI Agent / 多智能体 / AI 工作流
- AI 模型（训练、推理、微调、多模态、语音）
- AI Infra（GPU、推理优化、RAG、向量数据库、数据中心）
- AI 开发工具（Copilot、自动化、AI IDE、API、低代码）
- OpenAI / Anthropic / Google / Meta / Microsoft / xAI / Apple 等前沿动态
## 排除
- 嵌入式 / Arduino / ESP32 / 树莓派
- 与 AI 无关的底层系统话题（编译器、内存、链接器等）
- 普通编程技巧、代码风格、命名规范
- 泛科技或非科技内容
## 规则
- 仅根据输入字段判断
- 不确定是否与 AI 强相关时，宁可漏选
- 候选不足 {k} 个时，只返回真正符合的
- 返回的 id 必须是输入中的原始字符串
## 输出格式
**只输出一个 JSON 数组,数组元素为 story id 字符串**:
- 有匹配:`["12345", "67890"]`
- 无匹配:`[]`
严禁输出任何解释性文字、代码块包装、自然语言句子或键值对。
## 候选数据
JSON 数组,每项字段:`id`(字符串)/ `title` / `site` / `points` / `comments`
```json
{candidates_json}
```
ai-daily-main/pyproject.toml
+17
View File
@@ -0,0 +1,17 @@
[project]
name = "daily-news"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"aiohttp>=3.9.0",
"beautifulsoup4>=4.12.0",
"croniter>=6.0.0",
"feedparser>=6.0.0",
"markdownify>=0.11.0",
"pytest>=7.0.0",
"pytest-asyncio>=0.21.0",
"pytest-mock>=3.10.0",
"python-dotenv>=1.0.0",
"pyyaml>=6.0",
"requests>=2.34.2",
]
ai-daily-main/requirements.txt
+9
View File
@@ -0,0 +1,9 @@
feedparser>=6.0.0
aiohttp>=3.9.0
markdownify>=0.11.0
croniter>=6.0.0
pyyaml>=6.0
pytest>=7.0.0
pytest-asyncio>=0.21.0
pytest-mock>=3.10.0
python-dotenv>=1.0.0
ai-daily-main/resources/hero-banner.svg
+154
View File
@@ -0,0 +1,154 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 400" width="1200" height="400">
<defs>
<!-- 浅色渐变背景 -->
<linearGradient id="bgGrad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#f8f9fa;stop-opacity:1" />
<stop offset="100%" style="stop-color:#e8eaed;stop-opacity:1" />
</linearGradient>
<!-- Google 色系 -->
<linearGradient id="blueGrad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#4285F4;stop-opacity:1" />
<stop offset="100%" style="stop-color:#1a73e8;stop-opacity:1" />
</linearGradient>
<linearGradient id="redGrad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#EA4335;stop-opacity:1" />
<stop offset="100%" style="stop-color:#d93025;stop-opacity:1" />
</linearGradient>
<linearGradient id="yellowGrad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#FBBC05;stop-opacity:1" />
<stop offset="100%" style="stop-color:#f9a825;stop-opacity:1" />
</linearGradient>
<linearGradient id="greenGrad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#34A853;stop-opacity:1" />
<stop offset="100%" style="stop-color:#1e8e3e;stop-opacity:1" />
</linearGradient>
<!-- 柔和发光效果 -->
<filter id="softShadow" x="-20%" y="-20%" width="140%" height="140%">
<feDropShadow dx="0" dy="2" stdDeviation="4" flood-color="#000" flood-opacity="0.1"/>
</filter>
<filter id="glow" x="-50%" y="-50%" width="200%" height="200%">
<feGaussianBlur stdDeviation="2" result="coloredBlur"/>
<feMerge>
<feMergeNode in="coloredBlur"/>
<feMergeNode in="SourceGraphic"/>
</feMerge>
</filter>
</defs>
<!-- 背景 -->
<rect width="1200" height="400" fill="url(#bgGrad)"/>
<!-- 背景装饰圆点 -->
<circle cx="100" cy="80" r="60" fill="#4285F4" opacity="0.08"/>
<circle cx="1100" cy="100" r="80" fill="#EA4335" opacity="0.06"/>
<circle cx="150" cy="350" r="50" fill="#34A853" opacity="0.08"/>
<circle cx="1050" cy="320" r="70" fill="#FBBC05" opacity="0.08"/>
<circle cx="600" cy="50" r="40" fill="#4285F4" opacity="0.05"/>
<!-- 网格线 -->
<g stroke="#dadce0" stroke-width="0.5" opacity="0.5">
<line x1="0" y1="100" x2="1200" y2="100"/>
<line x1="0" y1="200" x2="1200" y2="200"/>
<line x1="0" y1="300" x2="1200" y2="300"/>
<line x1="300" y1="0" x2="300" y2="400"/>
<line x1="600" y1="0" x2="600" y2="400"/>
<line x1="900" y1="0" x2="900" y2="400"/>
</g>
<!-- Google 色系装饰：RSS 聚合 (左侧) -->
<g transform="translate(120, 200)">
<circle cx="0" cy="0" r="25" fill="none" stroke="#4285F4" stroke-width="2" opacity="0.3"/>
<circle cx="0" cy="0" r="18" fill="none" stroke="#4285F4" stroke-width="2" opacity="0.5"/>
<circle cx="0" cy="0" r="10" fill="none" stroke="#4285F4" stroke-width="2.5"/>
<circle cx="0" cy="0" r="4" fill="#4285F4"/>
<!-- 信号线 -->
<g stroke="#4285F4" stroke-width="1.5" opacity="0.4">
<line x1="25" y1="0" x2="40" y2="0"/>
<line x1="18" y1="-18" x2="28" y2="-28"/>
<line x1="0" y1="-25" x2="0" y2="-40"/>
<line x1="-18" y1="-18" x2="-28" y2="-28"/>
<line x1="-25" y1="0" x2="-40" y2="0"/>
<line x1="-18" y1="18" x2="-28" y2="28"/>
<line x1="0" y1="25" x2="0" y2="40"/>
<line x1="18" y1="18" x2="28" y2="28"/>
</g>
</g>
<!-- Google 色系装饰：AI 智能 (右侧) -->
<g transform="translate(1080, 200)">
<circle cx="0" cy="0" r="8" fill="#EA4335" filter="url(#glow)"/>
<circle cx="-25" cy="-20" r="5" fill="#FBBC05"/>
<circle cx="25" cy="-20" r="5" fill="#34A853"/>
<circle cx="0" cy="30" r="5" fill="#4285F4"/>
<!-- 连接线 -->
<g stroke="#5f6368" stroke-width="1" opacity="0.4">
<line x1="-21" y1="-17" x2="-6" y2="-4"/>
<line x1="21" y1="-17" x2="6" y2="-4"/>
<line x1="0" y1="25" x2="0" y2="4"/>
</g>
</g>
<!-- 底部数据流装饰 -->
<g transform="translate(0, 360)" opacity="0.6">
<rect x="80" y="0" width="60" height="3" rx="1.5" fill="#4285F4">
<animate attributeName="width" values="60;80;60" dur="2s" repeatCount="indefinite"/>
</rect>
<rect x="180" y="5" width="40" height="3" rx="1.5" fill="#EA4335">
<animate attributeName="width" values="40;55;40" dur="2.3s" repeatCount="indefinite"/>
</rect>
<rect x="950" y="0" width="70" height="3" rx="1.5" fill="#34A853">
<animate attributeName="width" values="70;90;70" dur="1.8s" repeatCount="indefinite"/>
</rect>
<rect x="1050" y="5" width="50" height="3" rx="1.5" fill="#FBBC05">
<animate attributeName="width" values="50;65;50" dur="2.1s" repeatCount="indefinite"/>
</rect>
</g>
<!-- 主标题 -->
<g transform="translate(600, 170)" filter="url(#softShadow)">
<text x="0" y="0" text-anchor="middle" font-family="system-ui, -apple-system, 'Google Sans', sans-serif" font-size="56" font-weight="700" fill="#202124">
AI Daily
</text>
<text x="0" y="38" text-anchor="middle" font-family="system-ui, -apple-system, sans-serif" font-size="20" font-weight="400" fill="#5f6368">
每日 AI 资讯推送系统
</text>
</g>
<!-- 核心标签 (Google 色系) -->
<g transform="translate(600, 250)" font-family="system-ui, -apple-system, sans-serif" font-size="14" font-weight="500">
<!-- RSS 聚合 - 蓝色 -->
<rect x="-175" y="-14" width="70" height="28" rx="14" fill="#4285F4" opacity="0.12"/>
<text x="-140" y="4" text-anchor="middle" fill="#4285F4">RSS 聚合</text>
<!-- LLM 智能 - 红色 -->
<rect x="-85" y="-14" width="70" height="28" rx="14" fill="#EA4335" opacity="0.12"/>
<text x="-50" y="4" text-anchor="middle" fill="#EA4335">LLM 智能</text>
<!-- 即时推送 - 黄色 -->
<rect x="5" y="-14" width="70" height="28" rx="14" fill="#FBBC05" opacity="0.2"/>
<text x="40" y="4" text-anchor="middle" fill="#c98400">即时推送</text>
<!-- 定时汇总 - 绿色 -->
<rect x="90" y="-14" width="70" height="28" rx="14" fill="#34A853" opacity="0.12"/>
<text x="125" y="4" text-anchor="middle" fill="#34A853">定时汇总</text>
</g>
<!-- 底部强调线 -->
<g transform="translate(600, 310)">
<line x1="-200" y1="0" x2="-80" y2="0" stroke="#4285F4" stroke-width="2" stroke-linecap="round" opacity="0.4"/>
<line x1="-60" y1="0" x2="60" y2="0" stroke="#EA4335" stroke-width="2" stroke-linecap="round" opacity="0.4"/>
<line x1="80" y1="0" x2="200" y2="0" stroke="#34A853" stroke-width="2" stroke-linecap="round" opacity="0.4"/>
</g>
<!-- 角落装饰 -->
<circle cx="40" cy="40" r="4" fill="#4285F4" opacity="0.3"/>
<circle cx="1160" cy="40" r="4" fill="#EA4335" opacity="0.3"/>
<circle cx="40" cy="360" r="4" fill="#34A853" opacity="0.3"/>
<circle cx="1160" cy="360" r="4" fill="#FBBC05" opacity="0.3"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 6.9 KiB

ai-daily-main/resources/rss.opml
+431
View File
@@ -0,0 +1,431 @@
<?xml version="1.0" encoding="UTF-8"?>
<opml version="2.0">
<head>
<title>All RSS Subscriptions for bestblogs.dev</title>
</head>
<body>
<outline text="42章经" title="42章经" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/f6694726ced4ba3d7c7cd65c6edf2160c5978387.xml" />
<outline text="43 Talks" title="43 Talks" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/4efe7ec6970afd4a050d6f10b9e8131a9d5e6816.xml" />
<outline text="51CTO技术栈" title="51CTO技术栈" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/d1fabe6c569ffc44979075dde2f57c65e07c3045.xml" />
<outline text="AI at Meta Blog" title="AI at Meta Blog" type="rss" xmlUrl="https://rsshub.bestblogs.dev/meta/ai/blog" />
<outline text="AINLP" title="AINLP" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/875df1d1a991bf9250ba9813e3148f58ef2240d4.xml" />
<outline text="AINews" title="AINews" type="rss" xmlUrl="https://news.smol.ai/rss.xml" />
<outline text="AI产品阿颖" title="AI产品阿颖" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/5000fe62390006b10eef8a737d89c478611994a7.xml" />
<outline text="AI产品黄叔" title="AI产品黄叔" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1f1030491e15e5349aae42367513d6b3f70a8f8b.xml" />
<outline text="AI前线" title="AI前线" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/25185b01482da0f485418ecb92e208b4416712fb.xml" />
<outline text="AI寒武纪" title="AI寒武纪" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/5903009f48a5e4aa44d8ac941a54fe3aafc3e03c.xml" />
<outline text="AI异类弗兰克" title="AI异类弗兰克" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/e0415653cbf6f41e25fa266d010b3238b91a65e3.xml" />
<outline text="AI炼金术" title="AI炼金术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/4915f3747653bbb9c7975323c11b768d2b9cd6c9.xml" />
<outline text="AI科技大本营" title="AI科技大本营" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/dfd3b5e742e32d8032a445832373191957202bf3.xml" />
<outline text="AI科技评论" title="AI科技评论" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/789e5fefb9cc2646ba7b680cb7a88378a34eb7a4.xml" />
<outline text="AWS Architecture Blog" title="AWS Architecture Blog" type="rss" xmlUrl="http://www.awsarchitectureblog.com/atom.xml" />
<outline text="AWS Machine Learning Blog" title="AWS Machine Learning Blog" type="rss" xmlUrl="https://aws.amazon.com/blogs/amazon-ai/feed/" />
<outline text="Anthropic News" title="Anthropic News" type="rss" xmlUrl="https://rsshub.bestblogs.dev/anthropic/news" />
<outline text="Bay的设计奥德赛" title="Bay的设计奥德赛" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/7515eedf3f994a6b23ebda2313a4b6bc7532f536.xml" />
<outline text="ByteByteGo Newsletter" title="ByteByteGo Newsletter" type="rss" xmlUrl="https://blog.bytebytego.com/feed" />
<outline text="CSDN" title="CSDN" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/b0b7f2852aecdcc5a0eb08d33afc1c08b855d98b.xml" />
<outline text="Clip设计夹" title="Clip设计夹" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/ebd5f5bd705dd531066eeca5ee500a1e6a269e17.xml" />
<outline text="Cursor Blog" title="Cursor Blog" type="rss" xmlUrl="https://api.bestblogs.dev/feed/cursor-blog" />
<outline text="Databricks" title="Databricks" type="rss" xmlUrl="https://www.databricks.com/feed" />
<outline text="Datawhale" title="Datawhale" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/ea0dd8bddfe4fbfb32eaa81a1e1b628d45e97a80.xml" />
<outline text="David Heinemeier Hansson" title="David Heinemeier Hansson" type="rss" xmlUrl="https://world.hey.com/dhh/feed.atom" />
<outline text="DeepSeek" title="DeepSeek" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1709da4f538d4ce4fb6d7a8ba1a5a1c297919601.xml" />
<outline text="Dify" title="Dify" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/e46c03a4cb65509e22ab9a8507888a2096319d65.xml" />
<outline text="Docker" title="Docker" type="rss" xmlUrl="https://www.docker.com/feed/" />
<outline text="Elastic Blog" title="Elastic Blog" type="rss" xmlUrl="https://www.elastic.co/blog/feed" />
<outline text="Elevate" title="Elevate" type="rss" xmlUrl="https://addyo.substack.com/feed" />
<outline text="ElevenLabs Blog" title="ElevenLabs Blog" type="rss" xmlUrl="https://api.bestblogs.dev/feed/elevenLabsBlog" />
<outline text="Engineering at Meta" title="Engineering at Meta" type="rss" xmlUrl="https://engineering.fb.com/feed/" />
<outline text="Founder Park" title="Founder Park" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/f940695505f2be1399d23cc98182297cadf6f90d.xml" />
<outline text="Gino Notes" title="Gino Notes" type="rss" xmlUrl="https://www.ginonotes.com/feed.xml" />
<outline text="Google Cloud Blog" title="Google Cloud Blog" type="rss" xmlUrl="https://cloudblog.withgoogle.com/rss/" />
<outline text="Google DeepMind Blog" title="Google DeepMind Blog" type="rss" xmlUrl="https://deepmind.com/blog/feed/basic/" />
<outline text="Google Developers Blog" title="Google Developers Blog" type="rss" xmlUrl="https://developers.googleblog.com/feeds/posts/default" />
<outline text="HelloGitHub" title="HelloGitHub" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/e6cc80b97bf64eeef61cc5927c78ba6ce3356422.xml" />
<outline text="Hugging Face" title="Hugging Face" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/8b68fdb4f24ab2287100988a8cec36363fec4214.xml" />
<outline text="Hugging Face Blog" title="Hugging Face Blog" type="rss" xmlUrl="https://huggingface.co/blog/feed.xml" />
<outline text="InfoQ" title="InfoQ" type="rss" xmlUrl="http://www.infoq.com/rss/rss.action" />
<outline text="InfoQ 中文" title="InfoQ 中文" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/13da94d7eb314b49fa251cb7e8399cae29d772db.xml" />
<outline text="Jeff Geerling" title="Jeff Geerling" type="rss" xmlUrl="https://www.jeffgeerling.com/blog.xml" />
<outline text="Jina AI" title="Jina AI" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/ff2c5468828ebe7236afd6c1d128e219774487c2.xml" />
<outline text="LangChain Blog" title="LangChain Blog" type="rss" xmlUrl="https://blog.langchain.dev/rss/" />
<outline text="Last Week in AI" title="Last Week in AI" type="rss" xmlUrl="https://lastweekin.ai/feed/" />
<outline text="Latent Space" title="Latent Space" type="rss" xmlUrl="https://www.latent.space/feed" />
<outline text="L先生说" title="L先生说" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/31c7fb6f7959a5ff90ae997b536e78b8b3f23321.xml" />
<outline text="Marcus on AI" title="Marcus on AI" type="rss" xmlUrl="https://garymarcus.substack.com/feed" />
<outline text="Martin Fowler" title="Martin Fowler" type="rss" xmlUrl="https://martinfowler.com/feed.atom" />
<outline text="Microsoft Azure Blog" title="Microsoft Azure Blog" type="rss" xmlUrl="https://azure.microsoft.com/en-us/blog/feed/" />
<outline text="Microsoft Research Blog" title="Microsoft Research Blog" type="rss" xmlUrl="http://research.microsoft.com/rss/news.xml" />
<outline text="Microsoft for Developers" title="Microsoft for Developers" type="rss" xmlUrl="https://devblogs.microsoft.com/landing" />
<outline text="MiniMax 稀宇科技" title="MiniMax 稀宇科技" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/00306b171f754d463b28cf83f3ba086ad009b430.xml" />
<outline text="MongoDB Blog" title="MongoDB Blog" type="rss" xmlUrl="https://www.mongodb.com/blog/rss" />
<outline text="Next.js Blog" title="Next.js Blog" type="rss" xmlUrl="https://nextjs.org/feed.xml" />
<outline text="Node.js Blog" title="Node.js Blog" type="rss" xmlUrl="https://nodejs.org/en/feed/blog.xml" />
<outline text="PaperAgent" title="PaperAgent" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/54001ca616dbc4b55d2b25d79d68a70191a0ddf4.xml" />
<outline text="Qdrant" title="Qdrant" type="rss" xmlUrl="https://qdrant.tech/index.xml" />
<outline text="Qunar技术沙龙" title="Qunar技术沙龙" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/84c072f8d34d1690f2783d7dda6013cf6d892b7f.xml" />
<outline text="Sean Goedecke" title="Sean Goedecke" type="rss" xmlUrl="https://www.seangoedecke.com/rss.xml" />
<outline text="Simon Willison's Weblog" title="Simon Willison's Weblog" type="rss" xmlUrl="https://simonwillison.net/atom/everything/" />
<outline text="Smashing Magazine" title="Smashing Magazine" type="rss" xmlUrl="http://rss1.smashingmagazine.com/feed/" />
<outline text="Spring Blog" title="Spring Blog" type="rss" xmlUrl="http://spring.io/blog.atom" />
<outline text="Stack Overflow Blog" title="Stack Overflow Blog" type="rss" xmlUrl="http://blog.stackoverflow.com/feed/" />
<outline text="SuperTechFans" title="SuperTechFans" type="rss" xmlUrl="https://www.supertechfans.com/cn/index.xml" />
<outline text="The Cloudflare Blog" title="The Cloudflare Blog" type="rss" xmlUrl="https://blog.cloudflare.com/rss" />
<outline text="The GitHub Blog" title="The GitHub Blog" type="rss" xmlUrl="https://github.blog/feed/" />
<outline text="The IntelliJ IDEA Blog" title="The IntelliJ IDEA Blog" type="rss" xmlUrl="http://blogs.jetbrains.com/idea/feed/" />
<outline text="The JetBrains Blog" title="The JetBrains Blog" type="rss" xmlUrl="http://blog.jetbrains.com/feed/" />
<outline text="The Keyword (blog.google) " title="The Keyword (blog.google) " type="rss" xmlUrl="https://blog.google/rss" />
<outline text="UX Magazine" title="UX Magazine" type="rss" xmlUrl="https://uxmag.com/feed/" />
<outline text="Vercel News" title="Vercel News" type="rss" xmlUrl="https://vercel.com/atom" />
<outline text="Visual Studio Blog" title="Visual Studio Blog" type="rss" xmlUrl="https://devblogs.microsoft.com/visualstudio/feed/" />
<outline text="Web3天空之城" title="Web3天空之城" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6aac3cc6d4c6df6fb3f77dea4ea4ba4a2053d6e7.xml" />
<outline text="Z Potentials" title="Z Potentials" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c47f4bc00ea912c37b6e23b22b146db0e85b3e19.xml" />
<outline text="antirez" title="antirez" type="rss" xmlUrl="http://antirez.com/rss" />
<outline text="dbaplus社群" title="dbaplus社群" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/a92cc44a756e2b9165fed5572aa7337843a73eee.xml" />
<outline text="deeplearning.ai" title="deeplearning.ai" type="rss" xmlUrl="https://rsshub.bestblogs.dev/deeplearning/the-batch" />
<outline text="freeCodeCamp.org" title="freeCodeCamp.org" type="rss" xmlUrl="https://www.freecodecamp.org/news/rss/" />
<outline text="overreacted" title="overreacted" type="rss" xmlUrl="https://overreacted.io/rss.xml" />
<outline text="vivo互联网技术" title="vivo互联网技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/b3ceb5cb1e4602ca55704650a157ec9c5b2f0d31.xml" />
<outline text="yikai 的摸鱼笔记" title="yikai 的摸鱼笔记" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/13492cb50df57702cdc0dfa71467cd03f9dd69be.xml" />
<outline text="乌鸦智能说" title="乌鸦智能说" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/f21c3e34df9b5fecfda57e2e53512864255ed4cd.xml" />
<outline text="京东技术" title="京东技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/fa0be550682410cc187c0d1eab1a0fc4e073b949.xml" />
<outline text="人人都是产品经理" title="人人都是产品经理" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/2d790e38f8af54c5af77fa5fed687a7c66d34c22.xml" />
<outline text="优设" title="优设" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/8fee9d33e883a769a59a5a3e27d249cf8567b55a.xml" />
<outline text="体验进阶" title="体验进阶" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/083c360a74b36b2c33820a995d21cbf60c813c0a.xml" />
<outline text="刘小排r" title="刘小排r" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/484d4199ae6c0b72ea01e7e0597a1f74933dfb62.xml" />
<outline text="刘润" title="刘润" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c1354f67c314d25d6e236a58724043bdc46d6079.xml" />
<outline text="创业邦" title="创业邦" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/f5e0d8e342d9e2ec5b2942f08522cfaec17acc8d.xml" />
<outline text="前端充电宝" title="前端充电宝" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/efed19b684285ee14f88b3f234b350fba9376d7a.xml" />
<outline text="前端早读课" title="前端早读课" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/ce2456e157156d42259c1198f05a33e27b1ed959.xml" />
<outline text="十字路口Crossing" title="十字路口Crossing" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/20492a5f2d3637c178c01ab0bab7ed86a4a0995b.xml" />
<outline text="卡尔的AI沃茨" title="卡尔的AI沃茨" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/8a1fc997e5c742e91ad7c253836c28ca3a69ccb1.xml" />
<outline text="印记中文" title="印记中文" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/2b038bb5307a75a603405f7191b5030576d3e8bd.xml" />
<outline text="古典古少侠" title="古典古少侠" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/63554903d26c4e94ba031e9c8a93492b7ebcfbb9.xml" />
<outline text="向阳乔木推荐看" title="向阳乔木推荐看" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/3e50f11753a7c5ed689565fbf5abf96cb4541c57.xml" />
<outline text="吴晓波频道" title="吴晓波频道" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/604fd0bfbb0214958f7fd2718509e4ea038c6afc.xml" />
<outline text="哔哩哔哩技术" title="哔哩哔哩技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/3a12ae4fde5bb74aab2fddc9f710a3c057eab82f.xml" />
<outline text="土猛的员外" title="土猛的员外" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/3ee671d065adc460bc20bbd269115987098c54a0.xml" />
<outline text="夕小瑶科技说" title="夕小瑶科技说" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/64b57d666259aee6bd097e76164e4a8371f0ad04.xml" />
<outline text="大模型智能" title="大模型智能" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/bfc6440c1a2443fab9a6bf607137d41db5cd5c93.xml" />
<outline text="大淘宝技术" title="大淘宝技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/26fef2307bebc8673703f7e726982d8f56c9a219.xml" />
<outline text="奇舞精选" title="奇舞精选" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/156a64fe3e95eebe4b85bf981d6ebb85441897bf.xml" />
<outline text="字节跳动Seed" title="字节跳动Seed" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6efd40bb335d2037f365d284cb5e00f0843e737e.xml" />
<outline text="字节跳动技术团队" title="字节跳动技术团队" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/d3a9e4d6f125cc98d1691dbc30cd97fec7ae2d03.xml" />
<outline text="宝玉的分享" title="宝玉的分享" type="rss" xmlUrl="https://baoyu.io/feed.xml" />
<outline text="小米技术" title="小米技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/8bbc1ba1d363e70cd42d1ce89fb9070cb075c3b3.xml" />
<outline text="小红书技术REDtech" title="小红书技术REDtech" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/0f8c47df6fd304112518544776e0bbf1d98ba0b9.xml" />
<outline text="少数派" title="少数派" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/f0e37a7d597231efed4bf6dd05b5d904de6dbcc1.xml" />
<outline text="山行AI" title="山行AI" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/98bc16b6f53902a2ab511b4faa3499e0a1c78eb1.xml" />
<outline text="开源服务指南" title="开源服务指南" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c125f09ef36fd6b6cb092c409e69a5bcc867d378.xml" />
<outline text="强少来了" title="强少来了" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/3c36fe804f63a7b936e372a37929d81fa0ad948a.xml" />
<outline text="得物技术" title="得物技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1cde72c9129b1f79cbb150166e7fed9a7568ee10.xml" />
<outline text="快手技术" title="快手技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c4cc10d2e32a5fa12927581ae581a336f399fe75.xml" />
<outline text="掘金本周最热" title="掘金本周最热" type="rss" xmlUrl="https://rsshub.bestblogs.dev/juejin/trending/all/weekly" />
<outline text="数字生命卡兹克" title="数字生命卡兹克" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/ff621c3e98d6ae6fceb3397e57441ffc6ea3c17f.xml" />
<outline text="数据可视化 AntV" title="数据可视化 AntV" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c1a09e8847fbaea14eaa89db218d783fe176c5a6.xml" />
<outline text="新智元" title="新智元" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/e531a18b21c34cf787b83ab444eef659d7a980de.xml" />
<outline text="晚点LatePost" title="晚点LatePost" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c442206ec9957f3c52f2f40300ca532079538b31.xml" />
<outline text="晚点再听LaterCast" title="晚点再听LaterCast" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1193affe8f8ed7a64281054cb022a7176054fa38.xml" />
<outline text="智东西" title="智东西" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/cfd52b4245ca6119b2fda4ef934832c689028927.xml" />
<outline text="智谱" title="智谱" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/433d2134dca54d80804daf32e8be546155be3300.xml" />
<outline text="暗涌Waves" title="暗涌Waves" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/bd586c1499b56aaec02dfefa87126232d234b010.xml" />
<outline text="月之暗面 Kimi" title="月之暗面 Kimi" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c5c43d4bc17bae656763859ed0903bb6314ec6fe.xml" />
<outline text="有新Newin" title="有新Newin" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/74554dcb3da8982083426b871bc8c314a9de9729.xml" />
<outline text="有机大橘子" title="有机大橘子" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6cef434b771dd75a91864b2e699a622cb4e3eb33.xml" />
<outline text="有赞coder" title="有赞coder" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/75ad69f1c1d0d1f289f7702cf5eb553287441fdc.xml" />
<outline text="机器之心" title="机器之心" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/8d97af31b0de9e48da74558af128a4673d78c9a3.xml" />
<outline text="机器之心SOTA模型" title="机器之心SOTA模型" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/2f520471856d56c7b3a95cd09eb777149b32828a.xml" />
<outline text="李继刚" title="李继刚" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/9645a69180041ff935c458753174fa8bc2061295.xml" />
<outline text="极客公园" title="极客公园" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/11ea7163fbea99e2ab9fa2812ac3d179574886cc.xml" />
<outline text="架构师之路" title="架构师之路" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/f6dec1c3ad16e43532dd427c85eaeb3a7b7b084e.xml" />
<outline text="歸藏的AI工具箱" title="歸藏的AI工具箱" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1c3e3571b1627d23ee9c64521a0b0a41d3fe2987.xml" />
<outline text="沃垠AI" title="沃垠AI" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/339818dbd5154cecdf5f4161f3391c7038a72bae.xml" />
<outline text="浮之静" title="浮之静" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/abb0de0c0cb8f684a1606a4b20121b245547adce.xml" />
<outline text="海外独角兽" title="海外独角兽" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/7200d3a5e976d231deb1e40ad33745c0e649b029.xml" />
<outline text="深思圈" title="深思圈" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/3e6fcb56a39b2e18f1036113655d4ff8fe726b62.xml" />
<outline text="深网腾讯新闻" title="深网腾讯新闻" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/396591aa7d3ef15fa3b5b17ec4b1aa840ebde335.xml" />
<outline text="爱范儿" title="爱范儿" type="rss" xmlUrl="http://www.ifanr.com/feed" />
<outline text="甲子光年" title="甲子光年" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1c4008936645d5c17239d99bba91522cf2bdfa26.xml" />
<outline text="白鲸出海" title="白鲸出海" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/2b8f03a73a0f2ac92a8ca69c124e5be6f442dbdc.xml" />
<outline text="百度AI" title="百度AI" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/d0767d885e6ba213344fb0c0408c51331e23a994.xml" />
<outline text="百度Geek说" title="百度Geek说" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6cc437d76f9dc4f7c35011c72e471e33e7bdd384.xml" />
<outline text="真格基金" title="真格基金" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/47798a14d51da72e68fae4f7a259f096750cf03e.xml" />
<outline text="硅星人Pro" title="硅星人Pro" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/c62ceda9eed269d851802bdbc5f33c4fabbf7462.xml" />
<outline text="硅谷101" title="硅谷101" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/8f8fe34034f6123b168ed7847c51d50ff47cd7ee.xml" />
<outline text="硅谷科技评论" title="硅谷科技评论" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/4515ee058133ff68570ad586abdd81f54f2b6ee3.xml" />
<outline text="稀土掘金技术社区" title="稀土掘金技术社区" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/33ecd2122ae788ea02dfcf1df857a54b9ae1338d.xml" />
<outline text="笔记侠" title="笔记侠" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/4c5d9bcc2fbfcd1dc81fb67559653f8957ef4760.xml" />
<outline text="经纬创投" title="经纬创投" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/05efb1c4cf91e5a37443cc323150ea38a838e9fd.xml" />
<outline text="网易科技" title="网易科技" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/028fbc21062e744c7b606880ebca01e22cb4b7b7.xml" />
<outline text="美团技术团队" title="美团技术团队" type="rss" xmlUrl="https://tech.meituan.com/feed/" />
<outline text="腾讯云开发者" title="腾讯云开发者" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6cec2c211479a5502896375860009782cf10c2ba.xml" />
<outline text="腾讯技术工程" title="腾讯技术工程" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/1e0ac39f8952b2e7f0807313cf2633d25078a171.xml" />
<outline text="腾讯混元" title="腾讯混元" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/306ce19a1ca590c9c2df781789e828d1acfa1356.xml" />
<outline text="腾讯研究院" title="腾讯研究院" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6152301e0978bffb0a8284cab339262b9764dcfb.xml" />
<outline text="腾讯科技" title="腾讯科技" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/a81bdfcbb9eefe870d285e81510ffa1af26e4520.xml" />
<outline text="花叔" title="花叔" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/ed3e181242a4622709081439d802523ecf7b78f2.xml" />
<outline text="袋鼠帝AI客栈" title="袋鼠帝AI客栈" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/24d0930cc9f4f0c708182dc1c087d41e1f4cbd33.xml" />
<outline text="言午" title="言午" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/138de227ebfbee6ea26564564f7bcd6c0c27af60.xml" />
<outline text="语言即世界language is world" title="语言即世界language is world" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/e1ed0d3edd93f90aef602105eb7ca51b35b7060a.xml" />
<outline text="赛博禅心" title="赛博禅心" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/752c31ca0446b837339463fc5440539e20267d2f.xml" />
<outline text="超人的电话亭" title="超人的电话亭" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/4be15abcd5621887bb7c1e2efd2d1cd8c68a16f0.xml" />
<outline text="通义大模型" title="通义大模型" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/4ebee6222ae08705b8aabc9116f0defbcb6b17c6.xml" />
<outline text="逛逛GitHub" title="逛逛GitHub" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/38be32e5376d852c13d3383e4d7a757fd9a55ff6.xml" />
<outline text="量子位" title="量子位" type="rss" xmlUrl="https://www.qbitai.com/feed" />
<outline text="阑夕" title="阑夕" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/fe0fc82458663820d6e91f6331dea05f3db223d4.xml" />
<outline text="阮一峰的网络日志" title="阮一峰的网络日志" type="rss" xmlUrl="http://feeds.feedburner.com/ruanyifeng" />
<outline text="阶跃星辰" title="阶跃星辰" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/3e2714d06aa36142e8ed6b3f4e5cf9090a069dd2.xml" />
<outline text="阿真Irene" title="阿真Irene" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/d5ead392b0cf117d0ba4070e2261111fdde49711.xml" />
<outline text="阿里云开发者" title="阿里云开发者" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/39fc51b0b1316137e608c45da5dbbca4f9eb9538.xml" />
<outline text="阿里技术" title="阿里技术" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/6535a444e9651fecae3383363be7589acdebe2b6.xml" />
<outline text="阿里研究院" title="阿里研究院" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/e2f1190c120f7f3d74b630bfcfe9e58296bd535c.xml" />
<outline text="随机小分队" title="随机小分队" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/115e814e7b12d373a55459cb2aea3223152f2af2.xml" />
<outline text="魔搭ModelScope社区" title="魔搭ModelScope社区" type="rss" xmlUrl="https://wechat2rss.bestblogs.dev/feed/d993a885260f96057b9a4c96212cb2c95bb5054b.xml" />
<outline text="AI Engineer" title="AI Engineer" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCLKPca3kwwd-B59HNr-_lvA" />
<outline text="AI Explained" title="AI Explained" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCNJ1Ymd5yFuUPtn21xtRbbw" />
<outline text="AI Master" title="AI Master" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC0yHbz4OxdQFwmVX2BBQqLg" />
<outline text="AICodeKing" title="AICodeKing" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC0m81bQuthaQZmFbXEY9QSw" />
<outline text="Acquired" title="Acquired" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCyFqFYfTW2VoIQKylJ04Rtw" />
<outline text="All-In Podcast" title="All-In Podcast" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCESLZhusAkFfsNsApnjF_Cg" />
<outline text="Andrej Karpathy" title="Andrej Karpathy" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCXUPKJO5MZQN11PqgIvyuvQ" />
<outline text="Anthropic" title="Anthropic" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCrDwWp7EBBv4NwvScIpBDOA" />
<outline text="ByteByteGo" title="ByteByteGo" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCZgt6AzoyjslHTC9dz0UoTw" />
<outline text="Dwarkesh Patel" title="Dwarkesh Patel" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCXl4i9dYBrFOabk0xGmbkRA" />
<outline text="EO" title="EO" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UClWTCPVi-AU9TeCN6FkGARg" />
<outline text="Every" title="Every" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCjIMtrzxYc0lblGhmOgC_CA" />
<outline text="Fireship" title="Fireship" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCsBjURrPoezykLs9EqgamOA" />
<outline text="GitHub" title="GitHub" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC7c3Kb6jYCRj4JOHHZTxKsQ" />
<outline text="Google" title="Google" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCK8sQmJBp8GCxrOtXWBpyEA" />
<outline text="Google DeepMind" title="Google DeepMind" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCP7jMXSY2xbc3KCAE0MHQ-A" />
<outline text="Greg Isenberg" title="Greg Isenberg" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCPjNBjflYl0-HQtUvOx0Ibw" />
<outline text="How I AI" title="How I AI" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCRYY7IEbkHLH_ScJCu9eWDQ" />
<outline text="Hung-yi Lee" title="Hung-yi Lee" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC2ggjtuuWvxrHHHiaDH1dlQ" />
<outline text="Justin Sung" title="Justin Sung" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC2Zs9v2hL2qZZ7vsAENsg4w" />
<outline text="LangChain" title="LangChain" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCC-lyoTfSrcJzA1ab3APAgw" />
<outline text="Last Week in AI" title="Last Week in AI" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCKARTq-t5SPMzwtft8FWwnA" />
<outline text="Lenny's Podcast" title="Lenny's Podcast" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC6t1O76G0jYXOAoYCm153dA" />
<outline text="Lex Fridman" title="Lex Fridman" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCSHZKyawb77ixDdsGog4iWA" />
<outline text="Liam Ottley" title="Liam Ottley" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCui4jxDaMb53Gdh-AZUTPAg" />
<outline text="Matt Wolfe" title="Matt Wolfe" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UChpleBmo18P08aKCIgti38g" />
<outline text="Matthew Berman" title="Matthew Berman" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCawZsQWqfGSbCI5yjkdVkTA" />
<outline text="My First Million" title="My First Million" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCyaN6mg5u8Cjy2ZI4ikWaug" />
<outline text="Nikhil Kamath" title="Nikhil Kamath" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCnC8SAZzQiBGYVSKZ_S3y4Q" />
<outline text="No Priors" title="No Priors" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCSI7h9hydQ40K5MJHnCrQvw" />
<outline text="OpenAI" title="OpenAI" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCXZCJLdBC09xxGZ6gcdrc6A" />
<outline text="PowerfulJRE" title="PowerfulJRE" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCzQUP1qoWDoEbmsQxvdjxgQ" />
<outline text="Product School" title="Product School" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC6hlQ0x6kPbAGjYkoz53cvA" />
<outline text="Riley Brown" title="Riley Brown" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCMcoud_ZW7cfxeIugBflSBw" />
<outline text="Sequoia Capital" title="Sequoia Capital" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCWrF0oN6unbXrWsTN7RctTw" />
<outline text="Silicon Valley Girl" title="Silicon Valley Girl" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCiq1FIgtEK7LRAOB1JXTPig" />
<outline text="Siraj Raval" title="Siraj Raval" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCWN3xxRkmTPmbKwht9FuE5A" />
<outline text="Spring I/O" title="Spring I/O" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCLMPXsvSrhNPN3i9h-u8PYg" />
<outline text="Stripe" title="Stripe" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCM1guA1E-RHLO2OyfQPOkEQ" />
<outline text="TED" title="TED" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCAuUUnT6oDeKwE6v1NGQxug" />
<outline text="The Diary Of A CEO" title="The Diary Of A CEO" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCGq-a57w-aPwyi3pW7XLiHw" />
<outline text="The Pragmatic Engineer" title="The Pragmatic Engineer" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCPbwhExawYrn9xxI21TFfyw" />
<outline text="Tina Huang" title="Tina Huang" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC2UXDak6o7rBm23k3Vv5dww" />
<outline text="Wes Roth" title="Wes Roth" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCqcbQf6yw5KzRoDDcZ_wBSw" />
<outline text="Y Combinator" title="Y Combinator" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCcefcZRL2oaA_uBNeo5UOWg" />
<outline text="a16z" title="a16z" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC9cn0TuPq4dnbTY-CBsm8XA" />
<outline text="freeCodeCamp.org" title="freeCodeCamp.org" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UC8butISFwT-Wl7EV0hUK0BQ" />
<outline text="leerob" title="leerob" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCZMli3czZnd1uoc1ShTouQw" />
<outline text="yobi321" title="yobi321" type="rss" xmlUrl="https://www.youtube.com/feeds/videos.xml?channel_id=UCB_DbqNN9w30tnyWJSrIwyA" />
<outline text="42章经" title="42章经" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/648b0b641c48983391a63f98" />
<outline text="AI炼金术" title="AI炼金术" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/63e9ef4de99bdef7d39944c8" />
<outline text="TIANYU2FM — 对谈未知领域" title="TIANYU2FM — 对谈未知领域" type="rss" xmlUrl="https://rsshub.xiaowuaiblog.com/xiaoyuzhou/podcast/5f22729f9504bbdb77253e46" />
<outline text="What's Next｜科技早知道" title="What's Next｜科技早知道" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/5e74b52c418a84a046ecaceb" />
<outline text="三五环" title="三五环" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/5e280fab418a84a0461faa3c" />
<outline text="东腔西调" title="东腔西调" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/5f72b66083c34e85dd14fde9" />
<outline text="乱翻书" title="乱翻书" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/61358d971c5d56efe5bcb5d2" />
<outline text="人民公园说AI" title="人民公园说AI" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/65257ff6e8ce9deaf70a65e9" />
<outline text="保持偏见" title="保持偏见" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/663e3c95af1e22bb157dcee3" />
<outline text="十字路口Crossing" title="十字路口Crossing" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/60502e253c92d4f62c2a9577" />
<outline text="半拿铁 | 商业沉浮录" title="半拿铁 | 商业沉浮录" type="rss" xmlUrl="http://rsshub.bestblogs.dev/xiaoyuzhou/podcast/62382c1103bea1ebfffa1c00" />
<outline text="卫诗婕｜商业漫谈Jane's talk" title="卫诗婕｜商业漫谈Jane's talk" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/6627fda4b56459544087d86a" />
<outline text="天真不天真" title="天真不天真" type="rss" xmlUrl="http://rsshub.bestblogs.dev/xiaoyuzhou/podcast/65cef9e3cace72dff8d98de3" />
<outline text="屠龙之术" title="屠龙之术" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/6507bc165c88d2412626b401" />
<outline text="开始连接LinkStart" title="开始连接LinkStart" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/63ff0da51b1faf8a0b70b337" />
<outline text="张小珺Jùn｜商业访谈录" title="张小珺Jùn｜商业访谈录" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/626b46ea9cbbf0451cf5a962" />
<outline text="无人知晓" title="无人知晓" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/611719d3cb0b82e1df0ad29e" />
<outline text="晚点聊 LateTalk" title="晚点聊 LateTalk" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/61933ace1b4320461e91fd55" />
<outline text="枫言枫语" title="枫言枫语" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/5e2864f5418a84a04628e249" />
<outline text="此话当真" title="此话当真" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/646f194853a5e5ea1408d97c" />
<outline text="牛油果烤面包" title="牛油果烤面包" type="rss" xmlUrl="http://rsshub.bestblogs.dev/xiaoyuzhou/podcast/5e7c8b2b418a84a046e3ecbc" />
<outline text="知行小酒馆" title="知行小酒馆" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/6013f9f58e2f7ee375cf4216" />
<outline text="硅谷101" title="硅谷101" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/5e5c52c9418a84a04625e6cc" />
<outline text="硬地骇客" title="硬地骇客" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/640ee2438be5d40013fe4a87" />
<outline text="纵横四海" title="纵横四海" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/62694abdb221dd5908417d1e" />
<outline text="罗永浩的十字路口" title="罗永浩的十字路口" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/68981df29e7bcd326eb91d88" />
<outline text="自习室 STUDY ROOM" title="自习室 STUDY ROOM" type="rss" xmlUrl="http://rsshub.bestblogs.dev/xiaoyuzhou/podcast/65a5fb7540d4ef949c0140ac" />
<outline text="跨国串门儿计划" title="跨国串门儿计划" type="rss" xmlUrl="https://rsshub.bestblogs.dev/xiaoyuzhou/podcast/670f3da40d2f24f28978736f" />
<outline text="AI Breakfast(@AiBreakfast)" title="AI Breakfast(@AiBreakfast)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/0e3ebaf288014c45b0d24b71fe37312b" />
<outline text="AI Engineer(@aiDotEngineer)" title="AI Engineer(@aiDotEngineer)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/7d19a619a1cc4a9896129211269d2c85" />
<outline text="AI SDK(@aisdk)" title="AI SDK(@aisdk)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/22af005b21ec45b1a4503acca777b7f0" />
<outline text="AI Will(@FinanceYF5)" title="AI Will(@FinanceYF5)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/aa74321087f9405a872fd9a76b743bf8" />
<outline text="AI at Meta(@AIatMeta)" title="AI at Meta(@AIatMeta)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/ef7c70f9568d45f4915169fef4ce90b4" />
<outline text="AI产品黄叔(@PMbackttfuture)" title="AI产品黄叔(@PMbackttfuture)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5b632b7fba274f62928cdcc9d3db4c5e" />
<outline text="AK(@_akhaliq)" title="AK(@_akhaliq)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/341f7b9f8d9b477e8bb200caa7f32c6e" />
<outline text="Aadit Sheth(@aaditsh)" title="Aadit Sheth(@aaditsh)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/179bcc4b8e5d4274b6e9e935f9fd4434" />
<outline text="Adam D'Angelo(@adamdangelo)" title="Adam D'Angelo(@adamdangelo)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/3042b6f912b24f64982cc23f7bd59681" />
<outline text="Addy Osmani(@addyosmani)" title="Addy Osmani(@addyosmani)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/ba36d5d6b5e843e1a2ee7bc0c9c7e666" />
<outline text="Ado(@adocomplete)" title="Ado(@adocomplete)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5a6249f44fc541ac8260eb9016a12792" />
<outline text="Akshay Kothari(@akothari)" title="Akshay Kothari(@akothari)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/3434c0d56ee0446f991fb6af42bfac4b" />
<outline text="Alex Albert(@alexalbert__)" title="Alex Albert(@alexalbert__)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/524525de0d69407b80f0a7d891fdc8df" />
<outline text="Aman Sanger(@amanrsanger)" title="Aman Sanger(@amanrsanger)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a02496979a0e4d86baf2b72c24db52a4" />
<outline text="Amjad Masad(@amasad)" title="Amjad Masad(@amasad)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5fb1814c610c4af2911caa98c5c5ef82" />
<outline text="Andrej Karpathy(@karpathy)" title="Andrej Karpathy(@karpathy)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/edf707b5c0b248579085f66d7a3c5524" />
<outline text="Andrew Ng(@AndrewYNg)" title="Andrew Ng(@AndrewYNg)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/08b5488b20bc437c8bfc317a52e5c26d" />
<outline text="Anthropic(@AnthropicAI)" title="Anthropic(@AnthropicAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/fc28a211471b496682feff329ec616e5" />
<outline text="Anton Osika – eu/acc(@antonosika)" title="Anton Osika – eu/acc(@antonosika)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5f13b32b124a41cfb659f903a84032b1" />
<outline text="Aravind Srinivas(@AravSrinivas)" title="Aravind Srinivas(@AravSrinivas)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/59e6b63ae9684d11be0ae13d9e7420f2" />
<outline text="Berryxia.AI(@berryxia)" title="Berryxia.AI(@berryxia)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/18e51259aadd41ed872f29abc37b5aa6" />
<outline text="Binyuan Hui(@huybery)" title="Binyuan Hui(@huybery)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f54b2b40185943ce8f48a880110b7bc2" />
<outline text="Boris Cherny(@bcherny)" title="Boris Cherny(@bcherny)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/ee8646fd533343e09bb210a47e4afbb5" />
<outline text="Browser Use(@browser_use)" title="Browser Use(@browser_use)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b8d7530f0b294405825013bbc1cc198f" />
<outline text="ChatGPT(@ChatGPTapp)" title="ChatGPT(@ChatGPTapp)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f7992687b8d74b14bf2341eb3a0a5ec4" />
<outline text="Claude(@claudeai)" title="Claude(@claudeai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/01f60d63a61b44d692cc35c7feb0b4a4" />
<outline text="Cognition(@cognition_labs)" title="Cognition(@cognition_labs)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4cc14cbd15c74e189d537c415369e1a7" />
<outline text="Cursor(@cursor_ai)" title="Cursor(@cursor_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5287b4e0e13a4ab7ab7b1d56f9d88960" />
<outline text="DAN KOE(@thedankoe)" title="DAN KOE(@thedankoe)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/6f84bc46da974ac18294fe3f83576108" />
<outline text="DeepLearning.AI(@DeepLearningAI)" title="DeepLearning.AI(@DeepLearningAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/42e6b4901b97498eab2ab64c07d56177" />
<outline text="DeepSeek(@deepseek_ai)" title="DeepSeek(@deepseek_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/68b610deb24b47ae9a236811563cda86" />
<outline text="Demis Hassabis(@demishassabis)" title="Demis Hassabis(@demishassabis)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4a884d5e2f3740c5a26c9c093de6388a" />
<outline text="Dify(@dify_ai)" title="Dify(@dify_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/0be252fedbe84ad7bea21be44b18da89" />
<outline text="Ding(@dingyi)" title="Ding(@dingyi)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/da142413d31e432ea2e850d2e37239eb" />
<outline text="ElevenLabs(@elevenlabsio)" title="ElevenLabs(@elevenlabsio)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/1897eed387064dfab443764d6da50bc6" />
<outline text="Eric Jing(@ericjing_ai)" title="Eric Jing(@ericjing_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/ddfdcdd4e390495c942f0b5da62af0fb" />
<outline text="Fei-Fei Li(@drfeifei)" title="Fei-Fei Li(@drfeifei)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a4bfe44bfc0d4c949da21ebd3f5f42a5" />
<outline text="Figma(@figma)" title="Figma(@figma)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f8a106a09a7d404fb8de7eb0c5ddd2a2" />
<outline text="Firecrawl(@firecrawl_dev)" title="Firecrawl(@firecrawl_dev)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/c04abb206bbf4f91b22795024d6c0614" />
<outline text="Fireworks AI(@FireworksAI_HQ)" title="Fireworks AI(@FireworksAI_HQ)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/9f35c76341554bd78c2b9e63dc4fa5d8" />
<outline text="Frank Wang 玉伯(@lifesinger)" title="Frank Wang 玉伯(@lifesinger)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/66b90d7ae5d5425e95470e8e57cee46e" />
<outline text="Gary Marcus(@GaryMarcus)" title="Gary Marcus(@GaryMarcus)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/35a38c5646d946fb894d8c30c1d9629e" />
<outline text="Geek(@geekbb)" title="Geek(@geekbb)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/9cb3b60e689e4445a7fbdfd0be144126" />
<outline text="Genspark(@genspark_ai)" title="Genspark(@genspark_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/71ffd342cb5d478185ef7d55bdfca011" />
<outline text="GitHub(@github)" title="GitHub(@github)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/fa5b15f68a2e4df1ab301e26a4ab9190" />
<outline text="Google AI Developers(@googleaidevs)" title="Google AI Developers(@googleaidevs)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/69d925d4a8d44221b03eecbe07bd0f74" />
<outline text="Google AI(@GoogleAI)" title="Google AI(@GoogleAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4de0bd2d5cef4333a0260dc8157054a7" />
<outline text="Google Antigravity(@antigravity)" title="Google Antigravity(@antigravity)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/193efc3cceb14e318b0aee80059fba1b" />
<outline text="Google DeepMind(@GoogleDeepMind)" title="Google DeepMind(@GoogleDeepMind)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a99538443a484fcc846bdcc8f50745ec" />
<outline text="Google Gemini App(@GeminiApp)" title="Google Gemini App(@GeminiApp)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/6fb337feeec44ca38b79491b971d868d" />
<outline text="Grant Lee(@thisisgrantlee)" title="Grant Lee(@thisisgrantlee)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/d8c5840226114dc98c98c3673e1a7f3f" />
<outline text="Greg Brockman(@gdb)" title="Greg Brockman(@gdb)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/af19d054e26a49129f23abfa82d9e268" />
<outline text="Groq Inc(@GroqInc)" title="Groq Inc(@GroqInc)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/771b32075fe54a83bdb6966de9647b4f" />
<outline text="Guillermo Rauch(@rauchg)" title="Guillermo Rauch(@rauchg)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/e8750659b8154dbfa0489f451e044af1" />
<outline text="Hailuo AI (MiniMax)(@Hailuo_AI)" title="Hailuo AI (MiniMax)(@Hailuo_AI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/e65b5e59fcb544918c1ba17f5758f0f8" />
<outline text="Harrison Chase(@hwchase17)" title="Harrison Chase(@hwchase17)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f299207df53745bca04a03db8d11c5aa" />
<outline text="HeyGen(@HeyGen_Official)" title="HeyGen(@HeyGen_Official)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a9aff6b016c143ed8728dd86eb70d7db" />
<outline text="Hugging Face(@huggingface)" title="Hugging Face(@huggingface)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/fc16750ce50741f1b1f05ea1fb29436f" />
<outline text="Hunyuan(@TXhunyuan)" title="Hunyuan(@TXhunyuan)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/6e8e7b42cb434818810f87bcf77d86fb" />
<outline text="Ian Goodfellow(@goodfellow_ian)" title="Ian Goodfellow(@goodfellow_ian)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/57831559d22440debbfb2f2528e4ba84" />
<outline text="James Clear(@JamesClear)" title="James Clear(@JamesClear)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b042dec6a5964d3181cb5df9289e344f" />
<outline text="Jan Leike(@janleike)" title="Jan Leike(@janleike)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/dceb5cd131b34c72a8376cba8ea5d864" />
<outline text="Jeff Dean(@JeffDean)" title="Jeff Dean(@JeffDean)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b1013166769c49f8aa3fbdc222867054" />
<outline text="Jerry Liu(@jerryjliu0)" title="Jerry Liu(@jerryjliu0)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b3d904c0d7c446558ef3a1e7f2eb362b" />
<outline text="Jim Fan(@DrJimFan)" title="Jim Fan(@DrJimFan)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/c6cfe7c0d6b74849997073233fdea840" />
<outline text="Jina AI(@JinaAI_)" title="Jina AI(@JinaAI_)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f510f6e7eecf456ca7e2895a46752888" />
<outline text="Julien Chaumond(@julien_c)" title="Julien Chaumond(@julien_c)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/44d9fa384087448a94d3c8595f8d535e" />
<outline text="Junyang Lin(@JustinLin610)" title="Junyang Lin(@JustinLin610)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/082097117b4543e9a741cd2580f936d3" />
<outline text="Justin Welsh(@thejustinwelsh)" title="Justin Welsh(@thejustinwelsh)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/58894bf2934a426ca833c682da2bc810" />
<outline text="Justine Moore(@venturetwins)" title="Justine Moore(@venturetwins)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/c61046471f174d86bc0eb76cb44a21c3" />
<outline text="Kevin Weil &#x1f1fa;&#x1f1f8;(@kevinweil)" title="Kevin Weil &#x1f1fa;&#x1f1f8;(@kevinweil)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/3ca3c7698fd04611a0e7d14fae93c84c" />
<outline text="Kling AI(@Kling_ai)" title="Kling AI(@Kling_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/564237c3de274d58a04f064920817888" />
<outline text="LangChain(@LangChainAI)" title="LangChain(@LangChainAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/862fee50a745423c87e2633b274caf1d" />
<outline text="Latent.Space(@latentspacepod)" title="Latent.Space(@latentspacepod)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a7be8b61a1264ea7984abfaea3eff686" />
<outline text="Lee Robinson(@leerob)" title="Lee Robinson(@leerob)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/dc2426bc8348495189b45451d1707a1c" />
<outline text="Lenny Rachitsky(@lennysan)" title="Lenny Rachitsky(@lennysan)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/77d5ce4736854b0ebae603e4b54d3095" />
<outline text="Lex Fridman(@lexfridman)" title="Lex Fridman(@lexfridman)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/adf65931519340f795e2336910b4cd15" />
<outline text="Lilian Weng(@lilianweng)" title="Lilian Weng(@lilianweng)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a8f7e2238039461cbc8bf55f5f194498" />
<outline text="LlamaIndex &#x1f999;(@llama_index)" title="LlamaIndex &#x1f999;(@llama_index)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/67e259bd5be544ce84bbc867eace54c2" />
<outline text="Logan Kilpatrick(@OfficialLoganK)" title="Logan Kilpatrick(@OfficialLoganK)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4f63d960de644aeebd0aa97e4994dafe" />
<outline text="Lovable(@lovable_dev)" title="Lovable(@lovable_dev)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/639cd13d44284e10ac89fbd1c5399767" />
<outline text="LovartAI(@lovart_ai)" title="LovartAI(@lovart_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/db648e4d4eae4822aa0d34f0faef7ad2" />
<outline text="Luyu Zhang(@goocarlos)" title="Luyu Zhang(@goocarlos)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/bc52f83045964e3cb0ed93f495d58c36" />
<outline text="ManusAI(@ManusAI_HQ)" title="ManusAI(@ManusAI_HQ)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/320181c4651a41a08015946b55f704ab" />
<outline text="Marc Andreessen &#x1f1fa;&#x1f1f8;(@pmarca)" title="Marc Andreessen &#x1f1fa;&#x1f1f8;(@pmarca)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/63316630d94543f5a6480f230f483008" />
<outline text="Martin Fowler(@martinfowler)" title="Martin Fowler(@martinfowler)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/55d2d3f3eaaf4357b3230e0b01a464d7" />
<outline text="Michael Truell(@mntruell)" title="Michael Truell(@mntruell)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/7f30b06ba5bf4841919bd159f68e6484" />
<outline text="Microsoft Research(@MSFTResearch)" title="Microsoft Research(@MSFTResearch)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/61f4b78554fb4b8fa5653ec5d924d15a" />
<outline text="Midjourney(@midjourney)" title="Midjourney(@midjourney)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/72dd496bfd9d44c5a5761a974630376d" />
<outline text="Mike Krieger(@mikeyk)" title="Mike Krieger(@mikeyk)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/78d7b99318b04b309b04000f7e24da29" />
<outline text="Milvus(@milvusio)" title="Milvus(@milvusio)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/424e67b19eed4500b7a440976bbd2ade" />
<outline text="Mistral AI(@MistralAI)" title="Mistral AI(@MistralAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/8d2d03aea8af49818096da4ea00409d1" />
<outline text="Mustafa Suleyman(@mustafasuleyman)" title="Mustafa Suleyman(@mustafasuleyman)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/394acfaff8c44e09936f5bc0b8504f2c" />
<outline text="NVIDIA AI(@NVIDIAAI)" title="NVIDIA AI(@NVIDIAAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/05f1492e43514dc3862a076d3697c390" />
<outline text="Nano Banana Pro(@NanoBanana)" title="Nano Banana Pro(@NanoBanana)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/02431ffb78124808a0e7abb5f983084a" />
<outline text="Naval(@naval)" title="Naval(@naval)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b43bc203409e4c5a9c3ae86fe1ac00c9" />
<outline text="Nick St. Pierre(@nickfloats)" title="Nick St. Pierre(@nickfloats)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/6ebdf0d91eef4c149acd0ef110635866" />
<outline text="NotebookLM(@NotebookLM)" title="NotebookLM(@NotebookLM)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/221a88341acb475db221a12fed8208d0" />
<outline text="Notion(@NotionHQ)" title="Notion(@NotionHQ)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f97a26863aec4425b021720d4f8e4ede" />
<outline text="OpenAI Developers(@OpenAIDevs)" title="OpenAI Developers(@OpenAIDevs)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/971dc1fc90da449bac23e5fad8a33d55" />
<outline text="OpenAI(@OpenAI)" title="OpenAI(@OpenAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/0c0856a69f9f49cf961018c32a0b0049" />
<outline text="OpenRouter(@OpenRouterAI)" title="OpenRouter(@OpenRouterAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/e503a90c035c4b1d8f8dd34907d15bf4" />
<outline text="Patrick Loeber(@patloeber)" title="Patrick Loeber(@patloeber)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/c65c68f3713747bba863f92d6b5e996f" />
<outline text="Paul Couvert(@itsPaulAi)" title="Paul Couvert(@itsPaulAi)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b9912ac9a29042cf8c834419dc44cb1f" />
<outline text="Paul Graham(@paulg)" title="Paul Graham(@paulg)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/900549ddadf04e839d3f7a17ebaba3fc" />
<outline text="Perplexity(@perplexity_ai)" title="Perplexity(@perplexity_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/fdd601ea751949e7bec9e4cdad7c8e6c" />
<outline text="Peter Yang(@petergyang)" title="Peter Yang(@petergyang)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/8aca47ab813841bb82211e3a4f203acb" />
<outline text="Philipp Schmid(@_philschmid)" title="Philipp Schmid(@_philschmid)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/ce352bbf72e44033985bc756db2ee0e2" />
<outline text="Poe(@poe_platform)" title="Poe(@poe_platform)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/17687b1051204b2dbaed4ea4c9178f28" />
<outline text="Qdrant(@qdrant_engine)" title="Qdrant(@qdrant_engine)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a55f6e33dd224235aabaabaaf9d58a06" />
<outline text="Qwen(@Alibaba_Qwen)" title="Qwen(@Alibaba_Qwen)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/80032d016d654eb4afe741ff34b7643d" />
<outline text="Ray Dalio(@RayDalio)" title="Ray Dalio(@RayDalio)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4838204097ed422eac24ad48e68dc3ff" />
<outline text="Recraft(@recraftai)" title="Recraft(@recraftai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/acc648327c614d9b985b9fc3d737165b" />
<outline text="Replicate(@replicate)" title="Replicate(@replicate)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/12eba9c3db4940c5ab2a72bd00f9ff2c" />
<outline text="Replit ⠕(@Replit)" title="Replit ⠕(@Replit)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/613f859e4bc440c5a28f40732840f5cf" />
<outline text="Rowan Cheung(@rowancheung)" title="Rowan Cheung(@rowancheung)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a636de3cbda0495daabd15b9fd298614" />
<outline text="Runway(@runwayml)" title="Runway(@runwayml)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/e6bb4f612dd24db5bc1a6811e6dd5820" />
<outline text="Ryo Lu(@ryolu_)" title="Ryo Lu(@ryolu_)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/98cf45d5af9c4a6c9b0c26c7df5e737c" />
<outline text="Sahil Lavingia(@shl)" title="Sahil Lavingia(@shl)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/baad3713defe4182844d2756b4c2c9ed" />
<outline text="Sam Altman(@sama)" title="Sam Altman(@sama)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/e30d4cd223f44bed9d404807105c8927" />
<outline text="Satya Nadella(@satyanadella)" title="Satya Nadella(@satyanadella)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/baa68dbd9a9e461a96fd9b2e3f35dcbf" />
<outline text="Scott Wu(@ScottWu46)" title="Scott Wu(@ScottWu46)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5fca8ccd87344d388bc863304ed6fd86" />
<outline text="Simon Willison(@simonw)" title="Simon Willison(@simonw)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/30ad80be93c84e44acc37d5ddf31db57" />
<outline text="Skywork(@Skywork_ai)" title="Skywork(@Skywork_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/6d7d398dd80b48d79669c92745d32cf6" />
<outline text="Stanford AI Lab(@StanfordAILab)" title="Stanford AI Lab(@StanfordAILab)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/d5fc365556e641cba2278f501e8c6f92" />
<outline text="Sualeh Asif(@sualehasif996)" title="Sualeh Asif(@sualehasif996)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/fafa6df3c67644b1a367a177240e0173" />
<outline text="Suhail(@Suhail)" title="Suhail(@Suhail)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/c961547e08df4396b3ab69367a07a1cd" />
<outline text="Sundar Pichai(@sundarpichai)" title="Sundar Pichai(@sundarpichai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/8324d65a63dc42c584a8c08cc8323c9f" />
<outline text="Susan STEM(@feltanimalworld)" title="Susan STEM(@feltanimalworld)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/02cb4bc135fc4ac2b6abbbbf6229f8f7" />
<outline text="Taranjeet(@taranjeetio)" title="Taranjeet(@taranjeetio)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/2de92402f4a24c90bb27e7580b93a878" />
<outline text="The Rundown AI(@TheRundownAI)" title="The Rundown AI(@TheRundownAI)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/83b1ea38940b4a1d81ea57d1ffb12ad7" />
<outline text="Thomas Wolf(@Thom_Wolf)" title="Thomas Wolf(@Thom_Wolf)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4918efb13c47459b8dcaa79cfdf72d09" />
<outline text="Tw93(@HiTw93)" title="Tw93(@HiTw93)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/665fc88440fd4436acbc2e630d824926" />
<outline text="Varun Mohan(@_mohansolo)" title="Varun Mohan(@_mohansolo)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/7794c4268a504019a94af1778857a703" />
<outline text="Viking(@vikingmute)" title="Viking(@vikingmute)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/aab44cb2665a49258cd81f63b0b55192" />
<outline text="Weaviate • vector database(@weaviate_io)" title="Weaviate • vector database(@weaviate_io)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/2f1035ec6b28475987af06b600e1d04c" />
<outline text="Windsurf(@windsurf_ai)" title="Windsurf(@windsurf_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4a8273800ed34a069eecdb6c5c1b9ccf" />
<outline text="XDash(@XDash)" title="XDash(@XDash)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/0e84eb2b7fdc420182eacdce2cf390b9" />
<outline text="Y Combinator(@ycombinator)" title="Y Combinator(@ycombinator)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/b1ab109f6afd42ab8ea32e17a19a3a3e" />
<outline text="Yangyi(@Yangyixxxx)" title="Yangyi(@Yangyixxxx)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/66c40de71a9842fda4853b7d9d1d20da" />
<outline text="Yann LeCun(@ylecun)" title="Yann LeCun(@ylecun)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f5f4f928dede472ea55053672ad27ab6" />
<outline text="Zara Zhang(@zarazhangrui)" title="Zara Zhang(@zarazhangrui)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5db2db649cb44ba8a964bed0f6e91222" />
<outline text="a16z(@a16z)" title="a16z(@a16z)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f3fedf817599470dbf8d8d11f0872475" />
<outline text="andrew chen(@andrewchen)" title="andrew chen(@andrewchen)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/a3eb6beb2d894da3a9b7ab6d2e46790e" />
<outline text="bolt.new(@boltdotnew)" title="bolt.new(@boltdotnew)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/760ab7cd9708452c9ce1f9144b92a430" />
<outline text="cat(@_catwu)" title="cat(@_catwu)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/66a6b39ddcfa42e39621e0ab293c1bdd" />
<outline text="clem &#x1f917;(@ClementDelangue)" title="clem &#x1f917;(@ClementDelangue)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/5dbd038a8f5140938d0877511571797b" />
<outline text="elvis(@omarsar0)" title="elvis(@omarsar0)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/931d6e88e067496cac6bf23f69d60f33" />
<outline text="eric zakariasson(@ericzakariasson)" title="eric zakariasson(@ericzakariasson)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/65f321be670b4ffba7f40d0afd38c94d" />
<outline text="hidecloud(@hidecloud)" title="hidecloud(@hidecloud)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/23d41992b29340788aa3d09d8364c5f5" />
<outline text="howie.serious(@howie_serious)" title="howie.serious(@howie_serious)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/423dbfd2f9ff4238a4816f945af0f279" />
<outline text="idoubi(@idoubicc)" title="idoubi(@idoubicc)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/3d72acd51d21414ea39871fc01982a65" />
<outline text="koji(@Yuancheng)" title="koji(@Yuancheng)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/4d9ddf8e2e8f4f57a748bdb1cd0a56a9" />
<outline text="lmarena.ai(@lmarena_ai)" title="lmarena.ai(@lmarena_ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/f01b088d5a39473e854b07143df77ec5" />
<outline text="mem0(@mem0ai)" title="mem0(@mem0ai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/94bb691baeff461686326af619beb116" />
<outline text="meng shao(@shao__meng)" title="meng shao(@shao__meng)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/48aae530e0bf413aa7d44380f418e2e3" />
<outline text="ollama(@ollama)" title="ollama(@ollama)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/6326c63a2dfa445bbde88bea0c3112c2" />
<outline text="orange.ai(@oran_ge)" title="orange.ai(@oran_ge)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/0277b0bbefd54df7bc6b7880122da8f7" />
<outline text="v0(@v0)" title="v0(@v0)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/dbf37973e6fc4eae91d4be9669a78fc7" />
<outline text="xAI(@xai)" title="xAI(@xai)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/3953aa71e87a422eb9d7bf6ff1c7c43e" />
<outline text="yan5xu(@yan5xu)" title="yan5xu(@yan5xu)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/8008824b7c5d4816a4d61f9c24af82b8" />
<outline text="傅盛(@FuSheng_0306)" title="傅盛(@FuSheng_0306)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/32e92127f6b24822b5d7693251162ffd" />
<outline text="向阳乔木(@vista8)" title="向阳乔木(@vista8)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/9de19c78f7454ad08c956c1a00d237fe" />
<outline text="宝玉(@dotey)" title="宝玉(@dotey)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/97f1484ae48c430fbbf3438099743674" />
<outline text="小互(@imxiaohu)" title="小互(@imxiaohu)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/74e542992cf7441390c708f5601071d4" />
<outline text="数字生命卡兹克(@Khazix0918)" title="数字生命卡兹克(@Khazix0918)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/7a9955909da04c628fd0ee21203dcab5" />
<outline text="李继刚(@lijigang_com)" title="李继刚(@lijigang_com)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/ca2fa444b6ea4b8b974fe148056e497a" />
<outline text="歸藏(guizang.ai)(@op7418)" title="歸藏(guizang.ai)(@op7418)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/831fac36aa0a49a9af79f35dc1c9b5d9" />
<outline text="马东锡 NLP(@dongxi_nlp)" title="马东锡 NLP(@dongxi_nlp)" type="rss" xmlUrl="https://api.xgo.ing/rss/user/94f06e2ab51544ef8c3fef471d90b71b" />
</body>
</opml>
ai-daily-main/scripts/_gen_units.py
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env python3
"""根据 config.json 渲染 systemd 单元模板。
被 scripts/install.sh 调用。本身不直接读环境，需要的运行时参数（user/group/uv 路径）
通过 CLI 注入，便于 install.sh 处理 sudo 场景。
"""
import argparse
import json
import sys
from pathlib import Path
def cron_to_oncalendar(expr: str) -> str:
"""5 段 cron 表达式 → systemd OnCalendar 字符串。
本项目 push_cron 只用到 minute/hour，其他位必须为 `*`。
遇到不支持的语法直接报错，避免静默生成错误的 timer。
"""
parts = expr.split()
if len(parts) != 5:
raise ValueError(f"无效 cron 表达式（必须 5 段）: {expr}")
minute, hour, dom, mon, dow = parts
for label, val in [("day-of-month", dom), ("month", mon), ("day-of-week", dow)]:
if val != "*":
raise ValueError(f"暂不支持非 * 的 {label} 字段: {expr}")
try:
m, h = int(minute), int(hour)
except ValueError as e:
raise ValueError(f"minute/hour 必须是整数: {expr}") from e
if not (0 <= m <= 59 and 0 <= h <= 23):
raise ValueError(f"minute/hour 越界: {expr}")
return f"*-*-* {h:02d}:{m:02d}:00"
def render(template_path: Path, variables: dict) -> str:
text = template_path.read_text(encoding="utf-8")
for key, val in variables.items():
text = text.replace(f"{{{{{key}}}}}", str(val))
return text
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--project-dir", required=True)
parser.add_argument("--user", required=True)
parser.add_argument("--group", required=True)
parser.add_argument("--uv-bin", required=True)
parser.add_argument("--output-dir", required=True)
args = parser.parse_args()
project_dir = Path(args.project_dir).resolve()
output_dir = Path(args.output_dir).resolve()
template_dir = project_dir / "systemd"
config_path = project_dir / "config.json"
with open(config_path, "r", encoding="utf-8") as f:
config = json.load(f)
fetch_min = config["schedule"]["fetch_interval_minutes"]
if not isinstance(fetch_min, int) or fetch_min <= 0:
raise ValueError(f"fetch_interval_minutes 必须是正整数: {fetch_min}")
push_crons = config["schedule"]["push_cron"]
if not push_crons:
raise ValueError("config.schedule.push_cron 不能为空")
push_lines = "\n".join(
f"OnCalendar={cron_to_oncalendar(c)}" for c in push_crons
)
log_retention = config.get("log", {}).get("retention_days", 7)
if not isinstance(log_retention, int) or log_retention <= 0:
raise ValueError(f"log.retention_days 必须是正整数: {log_retention}")
common = {
"PROJECT_DIR": str(project_dir),
"USER": args.user,
"GROUP": args.group,
"UV_BIN": args.uv_bin,
}
output_dir.mkdir(parents=True, exist_ok=True)
renderings = [
("dnews-fetch.service.tmpl", "dnews-fetch.service", {}),
("dnews-fetch.timer.tmpl", "dnews-fetch.timer",
{"FETCH_INTERVAL_MIN": fetch_min}),
("dnews-push.service.tmpl", "dnews-push.service", {}),
("dnews-push.timer.tmpl", "dnews-push.timer",
{"PUSH_ONCALENDAR_LINES": push_lines}),
("journald-dnews.conf.tmpl", "journald-dnews.conf",
{"LOG_RETENTION_DAYS": log_retention}),
]
print(f"📂 输出目录: {output_dir}")
for tpl_name, out_name, extra in renderings:
tpl_path = template_dir / tpl_name
if not tpl_path.exists():
print(f"❌ 模板不存在: {tpl_path}", file=sys.stderr)
return 2
rendered = render(tpl_path, {**common, **extra})
(output_dir / out_name).write_text(rendered, encoding="utf-8")
print(f" ✓ {out_name}")
print(f"\n生成参数:")
print(f" fetch 间隔 → 每 {fetch_min} 分钟（间隔触发，从上次完成开始计时）")
for c, line in zip(push_crons, push_lines.split("\n")):
print(f" push '{c}' → {line}")
print(f" 日志保留 → {log_retention} 天")
return 0
if __name__ == "__main__":
sys.exit(main())
ai-daily-main/scripts/daily-news.tmpl
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# daily-news - Daily News 系统服务管理包装脚本
# 由 scripts/install.sh 自动生成；项目路径：{{PROJECT_DIR}}
set -euo pipefail
TIMERS=(dnews-fetch.timer dnews-push.timer)
SERVICES=(dnews-fetch.service dnews-push.service)
PROJECT_DIR="{{PROJECT_DIR}}"
usage() {
cat <<EOF
用法: daily-news <command>
命令:
start 启动 fetch 和 push timer
stop 停止两个 timer
restart 重启两个 timer（不会立即触发 service，仅重置调度）
status [N] 查看 timer / service 状态 + 最近 N 行日志（默认 15）
logs 实时跟随日志（Ctrl+C 退出）
其他操作：
$PROJECT_DIR/scripts/install.sh 重新生成单元并安装（改 config.json 后用）
$PROJECT_DIR/scripts/uninstall.sh 完全卸载（不删数据）
EOF
}
cmd="${1:-}"
case "$cmd" in
start)
sudo systemctl start "${TIMERS[@]}"
echo "✓ Daily News 已启动"
;;
stop)
sudo systemctl stop "${TIMERS[@]}"
echo "✓ Daily News 已停止"
;;
restart)
sudo systemctl restart "${TIMERS[@]}"
echo "✓ Daily News 已重启"
;;
status)
lines="${2:-15}"
echo "═══ Timer ═══"
systemctl list-timers 'dnews-*' --no-pager || true
echo ""
echo "═══ Service 状态 ═══"
systemctl status "${SERVICES[@]}" --no-pager --lines=0 || true
echo ""
echo "═══ 最近 $lines 行日志 ═══"
journalctl --namespace=dnews -u dnews-fetch -u dnews-push -n "$lines" --no-pager
echo ""
echo "💡 实时跟随：daily-news logs"
;;
logs)
journalctl --namespace=dnews -u dnews-fetch -u dnews-push -f
;;
-h|--help|help|"")
usage
;;
*)
echo "❌ 未知命令: $cmd" >&2
echo ""
usage
exit 1
;;
esac
ai-daily-main/scripts/install.sh
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env bash
# Daily News - 系统服务一键安装
# 用法：./scripts/install.sh
# 不要用 sudo 直接调用本脚本；脚本会在需要时自行 sudo 提权。
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
# ─── 用户检测 ────────────────────────────────────────────────
if [[ $EUID -eq 0 ]]; then
if [[ -z "${SUDO_USER:-}" ]]; then
echo "❌ 请勿直接用 root 运行本脚本，请用普通用户（脚本按需自动提权）"
exit 1
fi
RUN_USER="$SUDO_USER"
else
RUN_USER="$USER"
fi
RUN_GROUP="$(id -gn "$RUN_USER")"
# ─── uv 检测 ─────────────────────────────────────────────────
UV_BIN="$(command -v uv || true)"
if [[ -z "$UV_BIN" ]]; then
echo "❌ 找不到 uv 命令，请先安装：https://docs.astral.sh/uv/"
exit 1
fi
# ─── systemd 检测 ────────────────────────────────────────────
if ! command -v systemctl >/dev/null 2>&1; then
echo "❌ 系统未检测到 systemctl（非 systemd 系统），本脚本仅支持 systemd Linux"
exit 1
fi
cd "$PROJECT_DIR"
echo "🚀 Daily News - 系统服务安装"
echo "────────────────────────────────────────"
echo " 项目目录: $PROJECT_DIR"
echo " 运行身份: $RUN_USER:$RUN_GROUP"
echo " uv 路径: $UV_BIN"
echo "────────────────────────────────────────"
echo ""
# 提前提权一次，后续 sudo 命令直接复用凭据缓存
echo "🔐 需要 sudo 权限来写入 /etc/systemd/system/"
sudo -v
echo ""
# ─── [1/6] 同步依赖 ──────────────────────────────────────────
echo "📦 [1/6] 同步依赖 (uv sync)..."
uv sync
echo "✓ 依赖已就绪"
echo ""
# ─── [2/6] 验证 .env ────────────────────────────────────────
echo "🔐 [2/6] 验证 .env..."
if [[ ! -f "$PROJECT_DIR/.env" ]]; then
echo "❌ .env 不存在，请先复制 .env.example 并填入 API Key"
exit 1
fi
echo "✓ .env 存在"
echo ""
# ─── [3/6] LLM 健康检查 ─────────────────────────────────────
echo "🔍 [3/6] 校验 LLM 接口（仅在此处校验，运行时不再校验）..."
if ! uv run python -m src.main check; then
echo ""
echo "❌ LLM 校验失败，已中止安装"
echo "请检查 config.json 中的 llm.baseUrl/model 以及 .env 中的 API Key"
exit 1
fi
echo ""
# ─── [4/6] 生成单元文件 ─────────────────────────────────────
echo "📝 [4/6] 渲染 systemd 单元模板..."
STAGE_DIR="$(mktemp -d)"
trap 'rm -rf "$STAGE_DIR"' EXIT
uv run python "$PROJECT_DIR/scripts/_gen_units.py" \
--project-dir "$PROJECT_DIR" \
--user "$RUN_USER" \
--group "$RUN_GROUP" \
--uv-bin "$UV_BIN" \
--output-dir "$STAGE_DIR"
# 同时渲染 daily-news 包装脚本（/usr/local/bin/daily-news）
sed "s|{{PROJECT_DIR}}|$PROJECT_DIR|g" \
"$PROJECT_DIR/scripts/daily-news.tmpl" > "$STAGE_DIR/daily-news"
echo " ✓ daily-news"
echo ""
# ─── [5/6] 安装到 systemd ───────────────────────────────────
echo "📥 [5/6] 安装到系统..."
sudo install -m 644 "$STAGE_DIR/dnews-fetch.service" /etc/systemd/system/
sudo install -m 644 "$STAGE_DIR/dnews-fetch.timer" /etc/systemd/system/
sudo install -m 644 "$STAGE_DIR/dnews-push.service" /etc/systemd/system/
sudo install -m 644 "$STAGE_DIR/dnews-push.timer" /etc/systemd/system/
# 日志保留策略（journald 命名空间 drop-in）
sudo mkdir -p /etc/systemd/journald@dnews.conf.d
sudo install -m 644 "$STAGE_DIR/journald-dnews.conf" \
/etc/systemd/journald@dnews.conf.d/retention.conf
# daily-news 包装脚本到系统 PATH
sudo install -m 755 "$STAGE_DIR/daily-news" /usr/local/bin/daily-news
sudo systemctl daemon-reload
# 命名空间 journald 实例首次会在 fetch/push 首次产出日志时自动拉起；
# 若已存在（重装场景），重启以加载新保留策略。
sudo systemctl restart systemd-journald@dnews.service 2>/dev/null || true
echo "✓ 单元文件 + daily-news 包装脚本已安装"
echo ""
# ─── [6/6] 启用并启动 timer ─────────────────────────────────
echo "🚦 [6/6] 启用并（重）启动 timer..."
# 先 enable 注册开机自启
sudo systemctl enable dnews-fetch.timer dnews-push.timer
# 再 restart 强制按新单元文件重置内存中的 timer 状态
# （enable --now 对已运行的 timer 是 no-op，无法应用新 OnCalendar/OnActiveSec 等改动）
sudo systemctl restart dnews-fetch.timer dnews-push.timer
echo ""
# ─── 完成 ───────────────────────────────────────────────────
echo "✅ 安装完成！"
echo ""
echo "下次触发时间："
systemctl list-timers 'dnews-*' --no-pager || true
echo ""
echo "常用命令（系统 PATH 中可直接调用）："
echo " daily-news status 查看状态 + 最近日志"
echo " daily-news logs 实时跟随日志"
echo " daily-news stop 停止"
echo " daily-news start 启动"
echo " daily-news restart 重启 timer"
echo ""
echo "手动立即触发一次任务（不影响下次调度）："
echo " sudo systemctl start dnews-fetch.service"
echo " sudo systemctl start dnews-push.service"
echo ""
echo "卸载：$PROJECT_DIR/scripts/uninstall.sh"
ai-daily-main/scripts/status.sh
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Daily News - 状态查看
# 显示：timer 下次触发时间、service 上次结果、最近日志
set -euo pipefail
LINES="${1:-15}"
echo "═══ Timer ═══"
systemctl list-timers 'dnews-*' --no-pager || true
echo ""
echo "═══ Service 状态 ═══"
systemctl status dnews-fetch.service --no-pager --lines=0 || true
echo "──────────────────────────────"
systemctl status dnews-push.service --no-pager --lines=0 || true
echo ""
echo "═══ 最近 $LINES 行日志 ═══"
journalctl --namespace=dnews \
-u dnews-fetch.service \
-u dnews-push.service \
-n "$LINES" \
--no-pager
echo ""
echo "💡 实时跟随：journalctl --namespace=dnews -f"
ai-daily-main/scripts/uninstall.sh
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# Daily News - 卸载系统服务（不删数据）
set -euo pipefail
echo "🗑 Daily News - 卸载"
echo ""
if ! command -v systemctl >/dev/null 2>&1; then
echo "❌ 系统未检测到 systemctl"
exit 1
fi
echo "🛑 停止并禁用 timer..."
sudo systemctl disable --now dnews-fetch.timer dnews-push.timer 2>/dev/null || true
echo "🧹 清理单元文件..."
sudo rm -f /etc/systemd/system/dnews-fetch.service \
/etc/systemd/system/dnews-fetch.timer \
/etc/systemd/system/dnews-push.service \
/etc/systemd/system/dnews-push.timer
echo "🧹 清理日志保留策略 drop-in..."
sudo rm -f /etc/systemd/journald@dnews.conf.d/retention.conf
sudo rmdir /etc/systemd/journald@dnews.conf.d 2>/dev/null || true
echo "🧹 清理 daily-news 包装脚本..."
sudo rm -f /usr/local/bin/daily-news
sudo systemctl daemon-reload
sudo systemctl stop systemd-journald@dnews.service 2>/dev/null || true
echo ""
echo "✅ 已卸载（news-data/ 与项目代码未删除）"
echo ""
echo "如需清理历史日志：sudo journalctl --namespace=dnews --vacuum-time=1s"
ai-daily-main/src/__init__.py
View File
ai-daily-main/src/config.py
+115
View File
@@ -0,0 +1,115 @@
"""配置加载和源管理"""
import fnmatch
import json
import xml.etree.ElementTree as ET
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Dict, List
from urllib.parse import urlparse
def _get_local_timezone() -> timezone:
"""自动检测本地时区"""
return datetime.now().astimezone().tzinfo
def get_timezone(config: Dict = None) -> timezone:
"""
获取配置时区，用于推送消息展示本地化时间
读取信息源统一使用 UTC 时间
如果 config 中没有 timezone_hours，则自动检测本地时区
"""
if config is None:
try:
config = load_config()
except Exception:
return _get_local_timezone()
hours = config.get("schedule", {}).get("timezone_hours")
if hours is None:
return _get_local_timezone()
return timezone(timedelta(hours=hours))
# 向后兼容的别名
def get_cst(config: Dict = None) -> timezone:
"""向后兼容，使用 get_timezone"""
return get_timezone(config)
def load_config(config_path: str = "config.json") -> Dict:
"""加载配置文件"""
path = Path(config_path)
if not path.exists():
raise FileNotFoundError(f"配置文件不存在: {config_path}")
with open(path, "r", encoding="utf-8") as f:
return json.load(f)
def parse_opml(opml_path: str) -> List[Dict]:
"""解析OPML文件获取订阅源列表"""
path = Path(opml_path)
if not path.exists():
return []
tree = ET.parse(path)
root = tree.getroot()
feeds = []
for outline in root.findall(".//outline[@type='rss']"):
feeds.append({
"title": outline.get("title", ""),
"xmlUrl": outline.get("xmlUrl", ""),
"category": outline.get("category", "未分类"),
})
return feeds
def merge_sources(sources_config: Dict) -> List[Dict]:
"""合并base_opml + add - block，以xmlUrl为key去重"""
# 1. 解析base OPML
base = parse_opml(sources_config.get("base_opml", ""))
# 2. 添加自定义源
add_list = sources_config.get("add", [])
all_sources = base + add_list
# 3. 应用block (以xmlUrl匹配)
block_list = sources_config.get("block", [])
block_urls = {b.get("xmlUrl", "") for b in block_list}
filtered = [s for s in all_sources if s.get("xmlUrl", "") not in block_urls]
# 4. 应用block_domains (域名屏蔽，支持通配符 *.substack.com)
block_domains = sources_config.get("block_domains", [])
if block_domains:
def is_domain_blocked(url: str) -> bool:
try:
domain = urlparse(url).netloc.lower()
for pattern in block_domains:
# 转换通配符模式为匹配格式
if pattern.startswith("*."):
# *.substack.com 匹配 substack.com 和 addyo.substack.com
suffix = pattern[2:] # substack.com
if domain == suffix or domain.endswith("." + suffix):
return True
elif fnmatch.fnmatch(domain, pattern):
return True
return False
except Exception:
return False
filtered = [s for s in filtered if not is_domain_blocked(s.get("xmlUrl", ""))]
# 5. 去重 (以xmlUrl为key)
seen = set()
result = []
for s in filtered:
url = s.get("xmlUrl", "")
if url and url not in seen:
seen.add(url)
result.append(s)
return result
ai-daily-main/src/fetcher.py
+213
View File
@@ -0,0 +1,213 @@
"""RSS抓取模块"""
import asyncio
from datetime import datetime, timezone
from typing import Dict, List, Optional
import aiohttp
import feedparser
import requests
# 默认超时配置（秒）
DEFAULT_FEED_TIMEOUT = 5
# title 截断阈值：nitter 会把整条推文塞进 <title>，需要截断
TITLE_MAX_CHARS = 200
# nitter / xcancel 实例：必须用白名单 UA + requests 客户端（aiohttp 的 TLS
# 指纹过不了），详见 nitter-practice.md
NITTER_HOSTS = (
"xcancel.com",
"nitter.net",
"nuku.trabun.org",
)
NITTER_HEADERS = {
"User-Agent": "Inoreader",
"Accept": "application/rss+xml, application/atom+xml, application/xml;q=0.9",
}
# 公益实例，独立的低并发池 + 每次抓完 sleep，避免给上游施压
NITTER_MAX_CONCURRENCY = 2
NITTER_REQUEST_DELAY = 1.0
DEFAULT_HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
}
def is_nitter_url(url: str) -> bool:
"""判断是否为 nitter / xcancel 实例 URL"""
return any(host in url for host in NITTER_HOSTS)
def parse_entry_time(entry) -> Optional[datetime]:
"""解析条目的发布时间 (返回带 UTC 时区的 datetime)"""
published_parsed = getattr(entry, "published_parsed", None)
if published_parsed is not None:
return datetime(*published_parsed[:6], tzinfo=timezone.utc)
updated_parsed = getattr(entry, "updated_parsed", None)
if updated_parsed is not None:
return datetime(*updated_parsed[:6], tzinfo=timezone.utc)
return None
def _extract_body(entry) -> str:
"""提取条目正文：优先 content（含 <content:encoded>），其次 description，最后 summary"""
content_list = getattr(entry, "content", None)
if content_list:
value = content_list[0].get("value", "")
if value:
return value
description = getattr(entry, "description", "")
if description:
return description
return getattr(entry, "summary", "") or ""
def _truncate_title(title: str) -> str:
if len(title) <= TITLE_MAX_CHARS:
return title
return title[:TITLE_MAX_CHARS].rstrip() + "…"
def _parse_feed_entries(content, feed_info: Dict, cutoff_time: datetime) -> List[Dict]:
"""把 feed 字节/字符串解析为条目列表，按 cutoff 时间过滤"""
feed = feedparser.parse(content)
entries = []
for entry in feed.entries:
pub_date = parse_entry_time(entry)
# RSS 通常按时间倒序排列，一旦发现过期直接跳出
if pub_date and pub_date < cutoff_time:
break
entries.append(
{
"title": _truncate_title(entry.get("title", "无标题")),
"link": entry.get("link", ""),
"published": pub_date,
"source": feed_info["title"],
"content": _extract_body(entry),
"tags": [],
"score": 0,
"summary": "",
}
)
return entries
async def _fetch_nitter_content(url: str, timeout: int) -> Optional[bytes]:
"""nitter / xcancel 专用：requests + Inoreader UA，丢线程池避免阻塞 loop"""
def _sync():
try:
r = requests.get(url, headers=NITTER_HEADERS, timeout=timeout)
if r.status_code != 200:
print(f"⚠️ HTTP {r.status_code}: {url}")
return None
return r.content
except Exception as e:
print(f"⚠️ nitter 抓取失败 {url}: {e}")
return None
return await asyncio.to_thread(_sync)
async def _fetch_aiohttp_content(
url: str, timeout: int, session: aiohttp.ClientSession = None
) -> Optional[str]:
"""普通 RSS 源：aiohttp + 浏览器 UA"""
client_timeout = aiohttp.ClientTimeout(total=timeout)
if session is not None:
async with session.get(
url, headers=DEFAULT_HEADERS, timeout=client_timeout
) as resp:
if resp.status != 200:
print(f"⚠️ HTTP {resp.status}: {url}")
return None
return await resp.text()
async with aiohttp.ClientSession() as sess:
async with sess.get(
url, headers=DEFAULT_HEADERS, timeout=client_timeout
) as resp:
if resp.status != 200:
print(f"⚠️ HTTP {resp.status}: {url}")
return None
return await resp.text()
async def fetch_single_feed_async(
feed_info: Dict,
cutoff_time: datetime,
timeout: int = 5,
session: aiohttp.ClientSession = None,
) -> List[Dict]:
"""异步获取单个源的条目"""
try:
if timeout is None:
timeout = DEFAULT_FEED_TIMEOUT
url = feed_info["xmlUrl"]
if is_nitter_url(url):
content = await _fetch_nitter_content(url, timeout)
else:
content = await _fetch_aiohttp_content(url, timeout, session)
if content is None:
return []
return _parse_feed_entries(content, feed_info, cutoff_time)
except Exception as e:
print(f"⚠️ 获取失败 {feed_info['title']}: {e}")
return []
async def fetch_all_feeds(
feeds: List[Dict], cutoff_time: datetime, max_workers: int = 10, timeout: int = None
) -> List[Dict]:
"""并发获取所有源的条目；nitter/xcancel 走独立的低并发池"""
if timeout is None:
timeout = DEFAULT_FEED_TIMEOUT
nitter_feeds = [f for f in feeds if is_nitter_url(f.get("xmlUrl", ""))]
normal_feeds = [f for f in feeds if not is_nitter_url(f.get("xmlUrl", ""))]
normal_sem = asyncio.Semaphore(max_workers)
nitter_sem = asyncio.Semaphore(NITTER_MAX_CONCURRENCY)
async def fetch_normal(feed):
async with normal_sem:
return await fetch_single_feed_async(feed, cutoff_time, timeout)
async def fetch_nitter(feed):
async with nitter_sem:
result = await fetch_single_feed_async(feed, cutoff_time, timeout)
# 公益实例：抓完 sleep，把同一 worker 串内的请求拉开
await asyncio.sleep(NITTER_REQUEST_DELAY)
return result
ordered_feeds = normal_feeds + nitter_feeds
tasks = [fetch_normal(f) for f in normal_feeds] + [
fetch_nitter(f) for f in nitter_feeds
]
results = await asyncio.gather(*tasks, return_exceptions=True)
all_entries = []
for feed, result in zip(ordered_feeds, results):
if isinstance(result, Exception):
print(f"⚠️ 获取失败 {feed['title']}: {result}")
else:
all_entries.extend(result)
return all_entries
ai-daily-main/src/llm.py
+634
View File
@@ -0,0 +1,634 @@
"""LLM模块 - 评分和汇总"""
import asyncio
import json
import os
import re
from datetime import datetime
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from src.markdown_utils import normalize_str_list, parse_frontmatter
def load_prompt(prompt_path: str, **kwargs) -> str:
"""加载提示词模板并填充变量"""
path = Path(prompt_path)
if not path.exists():
raise FileNotFoundError(f"提示词文件不存在: {prompt_path}")
with open(path, "r", encoding="utf-8") as f:
template = f.read()
# 先把模板中的 {{ 和 }} 替换成占位符，避免与format冲突
template = template.replace("{{", "\x00LEFT_BRACE\x00").replace(
"}}", "\x00RIGHT_BRACE\x00"
)
# 替换变量
for key, value in kwargs.items():
template = template.replace(f"{{{key}}}", str(value))
# 恢复 {{ 和 }}
template = template.replace("\x00LEFT_BRACE\x00", "{").replace(
"\x00RIGHT_BRACE\x00", "}"
)
return template
async def call_llm(
prompt: str, config: Dict, response_format: Optional[Dict] = None
) -> str:
"""调用LLM API - 统一使用OpenAI兼容接口"""
model = config.get("model", "gpt-4o-mini")
base_url = config.get("baseUrl", "https://api.openai.com/v1")
api_key_name = config.get("apiKeyName", "OPENAI_API_KEY")
api_key = os.environ.get(api_key_name)
if not api_key:
raise ValueError(f"未设置{api_key_name}环境变量")
import aiohttp
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
}
if response_format is not None:
payload["response_format"] = response_format
url = f"{base_url}/chat/completions"
async with aiohttp.ClientSession() as session:
async with session.post(url, headers=headers, json=payload) as resp:
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"LLM API错误: {resp.status} - {text}")
data = await resp.json()
return data["choices"][0]["message"]["content"]
async def check_llm_available(config: Dict, timeout_seconds: int = 15) -> str:
"""启动时检查 LLM 接口可用性"""
prompt = "Reply with OK only."
try:
response = await asyncio.wait_for(
call_llm(prompt, config), timeout=timeout_seconds
)
except asyncio.TimeoutError as exc:
raise RuntimeError(f"LLM可用性检查超时({timeout_seconds}s)") from exc
except Exception as exc:
raise RuntimeError(f"LLM可用性检查失败: {exc}") from exc
response_text = response.strip()
if not response_text:
raise RuntimeError("LLM可用性检查返回空响应")
return response_text
def _build_batch_prompt(entries: List[Dict], prompt_path: str = None) -> str:
"""构建批量评分prompt"""
# 构建entries JSON列表（只包含必要字段）
entries_for_llm = []
for e in entries:
entries_for_llm.append(
{
"link": e.get("link", ""),
"title": e.get("title", "无标题"),
"source": e.get("source", "未知来源"),
"published": e.get("published", ""),
"content": e.get("content", "")[:2000], # 限制内容长度
}
)
entries_json = json.dumps(entries_for_llm, ensure_ascii=False, indent=2)
# 从文件加载提示词模板，如果未指定则使用默认路径
if prompt_path is None:
prompt_path = "prompts/score_batch.md"
return load_prompt(prompt_path, entries_json=entries_json)
def _parse_llm_json_response(response: str) -> List[Dict]:
"""解析LLM返回的JSON响应"""
text = response.strip()
# 尝试去除markdown代码块
if text.startswith("```json"):
text = text[7:]
elif text.startswith("```"):
text = text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
# 尝试查找JSON数组
if text.startswith("[") and text.endswith("]"):
try:
return json.loads(text)
except json.JSONDecodeError:
print("⚠️ 直接解析JSON失败，尝试从文本中提取JSON数组")
pass
# 尝试从文本中提取JSON数组
match = re.search(r"\[.*\]", text, re.DOTALL)
if match:
try:
return json.loads(match.group())
except json.JSONDecodeError:
print("⚠️ 从文本中提取JSON数组失败:", text)
pass
raise ValueError(f"无法从响应中解析JSON: {response[:200]}...")
def _parse_score_response(response: str) -> List[Dict]:
"""解析评分LLM响应。
json_object 模式下应返回 {"items": [...]} 形式的对象;
兼容直接数组与 markdown 包裹作为兜底路径。
"""
text = response.strip()
if text.startswith("```json"):
text = text[7:]
elif text.startswith("```"):
text = text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
parsed = None
try:
parsed = json.loads(text)
except json.JSONDecodeError:
for pattern in (r"\{.*\}", r"\[.*\]"):
match = re.search(pattern, text, re.DOTALL)
if match:
try:
parsed = json.loads(match.group())
break
except json.JSONDecodeError:
continue
if parsed is None:
print(f"无法从响应中解析JSON: {response}")
raise ValueError(f"无法从响应中解析JSON: {response[:200]}...")
if isinstance(parsed, list):
return parsed
if isinstance(parsed, dict):
for key in ("items", "results", "data", "scores"):
if isinstance(parsed.get(key), list):
return parsed[key]
list_values = [v for v in parsed.values() if isinstance(v, list)]
if len(list_values) == 1:
return list_values[0]
print(f"无法从响应中提取评分数组: {response}")
raise ValueError(f"无法从响应中提取评分数组: {response[:200]}...")
def _split_entries_for_batch(
entries: List[Dict], max_prompt_chars: int = 10000
) -> List[List[Dict]]:
"""将entries分成多个批次，每批不超过max_prompt_chars字符"""
if not entries:
return []
batches = []
current_batch = []
current_chars = 0
# 预留prompt模板和JSON包装的空间
overhead = len(_build_batch_prompt([])) + 500
for entry in entries:
# 估算该entry在JSON中的字符数
entry_chars = len(
json.dumps(
{
"link": entry.get("link", ""),
"title": entry.get("title", "")[:100],
"source": entry.get("source", ""),
"published": entry.get("published", ""),
"content": entry.get("content", "")[:2000],
},
ensure_ascii=False,
)
)
# 如果当前批次加上这个entry会超出限制，且当前批次不为空，则创建新批次
if current_chars + entry_chars + overhead > max_prompt_chars and current_batch:
batches.append(current_batch)
current_batch = [entry]
current_chars = entry_chars
else:
current_batch.append(entry)
current_chars += entry_chars
# 添加最后一个批次
if current_batch:
batches.append(current_batch)
return batches
def _reconcile_batch_results(
entries: List[Dict], results: List[Dict], batch_index: int
) -> Tuple[List[Dict], List[str]]:
"""对单批评分结果按 link 过滤，保留可回收结果"""
entry_links = {entry.get("link") for entry in entries if entry.get("link")}
matched_results = []
result_links = set()
for item in results:
if not isinstance(item, dict):
continue
link = item.get("link")
if link:
result_links.add(link)
if link in entry_links:
matched_results.append(item)
errors = []
if len(results) != len(entries) or len(matched_results) != len(entries):
missing_links = sorted(entry_links - result_links)
error_message = (
"批次{batch} 评分结果异常: 输入{input_count}, 返回{output_count}, "
"匹配{matched_count}, 未评分链接({missing_count}): {missing}"
).format(
batch=batch_index + 1,
input_count=len(entries),
output_count=len(results),
matched_count=len(matched_results),
missing_count=len(missing_links),
missing=missing_links,
)
print(f"⚠️ {error_message}")
errors.append(error_message)
return matched_results, errors
async def _score_single_batch(
entries: List[Dict], config: Dict, batch_index: int = 0
) -> Tuple[List[Dict], List[str]]:
"""对单批entries进行评分"""
# 从config获取批量评分提示词路径
prompt_path = config.get("prompts", {}).get("score_batch", "prompts/score_batch.md")
prompt = _build_batch_prompt(entries, prompt_path)
try:
response = await call_llm(
prompt, config, response_format={"type": "json_object"}
)
results = _parse_score_response(response)
if not isinstance(results, list):
raise ValueError(f"LLM返回的不是数组: {type(results)}")
return _reconcile_batch_results(entries, results, batch_index)
except Exception as e:
error_message = f"批次{batch_index + 1} 评分失败: {e}"
print(f"⚠️ {error_message}")
return [], [error_message]
async def score_batch(
entries: List[Dict], config: Dict
) -> Tuple[List[Dict], List[str]]:
"""
批量评分 - 智能分批处理
根据数据量自动决定分批策略：
- 小批量：一次性发送
- 大批量：分成多个批次并行处理
"""
if not entries:
return [], []
# 获取分批配置
max_prompt_chars = config.get("max_prompt_chars", 10000)
max_concurrent_batches = config.get("max_concurrent_batches", 3)
# 分批
batches = _split_entries_for_batch(entries, max_prompt_chars)
print(f"📦 分成 {len(batches)} 个批次评分 (共 {len(entries)} 条)")
# 如果只有一批，直接处理
if len(batches) == 1:
scores, errors = await _score_single_batch(batches[0], config, batch_index=0)
return _merge_scores(entries, scores), errors
# 多批并行处理（限制并发数）
semaphore = asyncio.Semaphore(max_concurrent_batches)
async def score_with_limit(batch_index: int, batch: List[Dict]):
async with semaphore:
return await _score_single_batch(batch, config, batch_index=batch_index)
# 并发处理所有批次
batch_tasks = [
score_with_limit(batch_index, batch)
for batch_index, batch in enumerate(batches)
]
batch_results = await asyncio.gather(*batch_tasks)
# 合并所有评分结果
all_scores = []
all_errors = []
for scores, errors in batch_results:
all_scores.extend(scores)
all_errors.extend(errors)
return _merge_scores(entries, all_scores), all_errors
def _merge_scores(entries: List[Dict], scores: List[Dict]) -> List[Dict]:
"""将评分结果合并到原始entries中"""
# 构建link到score的映射
score_map = {s.get("link"): s for s in scores if s.get("link")}
merged = []
for entry in entries:
link = entry.get("link")
score_data = score_map.get(link, {})
# 确保 score 为整数类型
score_value = score_data.get("score", entry.get("score"))
if isinstance(score_value, str):
try:
score_value = int(score_value)
except (ValueError, TypeError):
score_value = 0
merged.append(
{
**entry,
"tags": score_data.get("tags", entry.get("tags", [])),
"score": score_value,
"summary": score_data.get("summary", entry.get("summary", "")),
}
)
return merged
async def generate_immediate_push(
entries: List[Dict], config: Dict, recent_push_context: str = ""
) -> Tuple[str, Optional[str]]:
"""生成即时推送内容
Args:
entries: 原始entries列表（调用方已筛选好高分条目）
config: LLM配置
recent_push_context: 近期推送上下文，用于去重
"""
prompt_path = config.get("prompts", {}).get(
"immediate_push", "prompts/immediate_push.txt"
)
# 直接使用传入的entries，转为JSON格式传给prompt
prompt = load_prompt(
prompt_path,
count=len(entries),
entries=json.dumps(entries, ensure_ascii=False, indent=2),
recent_push_context=recent_push_context,
)
try:
return await call_llm(prompt, config), None
except Exception as e:
error_message = f"生成即时推送失败: {e}"
print(f"⚠️ {error_message}")
return "", error_message
async def compose_digest(
entries: List[Dict],
context: List[Dict],
config: Dict,
recent_push_context: str = "",
) -> str:
"""生成定时汇总推送内容
Args:
entries: 原始entries列表
context: 历史碎片化信息（用于去重参考），只保留 title, published, tags, summary, source
config: LLM配置
recent_push_context: 近期汇总推送上下文，用于去重
"""
prompt_path = config.get("prompts", {}).get("digest", "prompts/digest.md")
# context 只保留必要字段，拼接成字符串
context_text = []
for c in context:
tags_str = ", ".join(c.get("tags", [])) if c.get("tags") else ""
context_text.append(
f"[score: {c.get('score', 0)}] title:{c.get('title', '')}\n"
f"published: {c.get('published', '')}\n"
f"tags: {tags_str}\n"
f"source: {c.get('source', '')}\n"
f"summary: {c.get('summary', '')}"
)
prompt = load_prompt(
prompt_path,
count=len(entries),
entries=json.dumps(entries, ensure_ascii=False, indent=2),
context="\n\n".join(context_text),
recent_push_context=recent_push_context,
date=datetime.now().strftime("%Y-%m-%d"),
)
try:
return await call_llm(prompt, config)
except Exception:
raise
async def summarize_github_trending(
enriched_repos: List[Dict], config: Dict
) -> Tuple[str, Optional[str]]:
"""GH 板块总结:从 enriched 候选中选 1-max_items + 写 markdown。不传历史上下文。"""
prompt_path = config.get("prompts", {}).get(
"section_github", "prompts/section_github.md"
)
max_items = (
config.get("sections", {}).get("github_trending", {}).get("max_items", 3)
)
prompt = load_prompt(
prompt_path,
repos_json=json.dumps(enriched_repos, ensure_ascii=False, indent=2),
max_items=max_items,
)
try:
return await call_llm(prompt, config), None
except Exception as e:
msg = f"summarize_github_trending 失败: {e}"
print(f"⚠️ {msg}")
return "", msg
async def select_ai_related_hn(
candidates: List[Dict], k: int, config: Dict
) -> Tuple[List[str], Optional[str]]:
"""轻 LLM:从 HN 首页候选元数据中挑 k 个 AI 相关 id。
输入候选只含 id/title/site/points/comments 字段(不含正文)。
"""
prompt_path = config.get("prompts", {}).get(
"section_hackernews_select", "prompts/section_hackernews_select.md"
)
slim = [
{
"id": c.get("id"),
"title": c.get("title", ""),
"site": c.get("site", ""),
"points": c.get("points", 0),
"comments": c.get("comments", 0),
}
for c in candidates
]
prompt = load_prompt(
prompt_path,
k=k,
candidates_json=json.dumps(slim, ensure_ascii=False, indent=2),
)
try:
response = await call_llm(prompt, config)
except Exception as e:
msg = f"select_ai_related_hn 失败: {e}"
print(f"⚠️ {msg}")
return [], msg
try:
ids = _parse_llm_json_response(response)
except ValueError as e:
msg = f"select_ai_related_hn 解析失败: {e}"
print(f"⚠️ {msg}")
return [], msg
if not isinstance(ids, list):
return [], "select_ai_related_hn 返回非数组"
return [str(x) for x in ids][:k], None
async def summarize_hackernews(
enriched_stories: List[Dict], config: Dict
) -> Tuple[str, Optional[str]]:
"""对输入的 K 个 enriched stories 行文(K 由 select_k 决定)。不传历史上下文。"""
prompt_path = config.get("prompts", {}).get(
"section_hackernews", "prompts/section_hackernews.md"
)
prompt = load_prompt(
prompt_path,
stories_json=json.dumps(enriched_stories, ensure_ascii=False, indent=2),
)
try:
return await call_llm(prompt, config), None
except Exception as e:
msg = f"summarize_hackernews 失败: {e}"
print(f"⚠️ {msg}")
return "", msg
async def generate_trend_insights(
sections: Dict[str, str], config: Dict
) -> Tuple[str, Optional[str]]:
"""输入三段成品,返回洞察段 markdown(含 frontmatter)。"""
prompt_path = config.get("prompts", {}).get("insights", "prompts/insights.md")
prompt = load_prompt(
prompt_path,
rss=sections.get("rss", ""),
github=sections.get("github", ""),
hackernews=sections.get("hackernews", ""),
)
try:
return await call_llm(prompt, config), None
except Exception as e:
msg = f"generate_trend_insights 失败: {e}"
print(f"⚠️ {msg}")
return "", msg
def parse_insights_with_metadata(llm_output: str, date: str) -> Tuple[str, Dict]:
"""解析 insights LLM 输出,返回 (insights_md, metadata)。
metadata 字段:title / excerpt / seotitle / seodescription / lead / highlights /
profile / date。缺失字段补默认值。
"""
meta, body = parse_frontmatter(llm_output)
insights_md = body if meta else llm_output
metadata = {
"title": meta.get("title") or f"📰 AI Daily 每日精选 | {date}",
"excerpt": meta.get("excerpt", ""),
"seotitle": meta.get("seotitle", ""),
"seodescription": meta.get("seodescription", ""),
"lead": meta.get("lead", ""),
"highlights": normalize_str_list(meta.get("highlights")),
"profile": "morning",
"date": date,
}
return insights_md, metadata
def parse_digest_with_metadata(llm_output: str, date: str) -> Tuple[str, Dict]:
"""解析 digest LLM 输出,返回 (digest_md, metadata)。
metadata 字段:title / lead / highlights / profile / date。
无 frontmatter 时回退到 "🌙 AI Daily 晚报 | {date}" 标题。
"""
meta, body = parse_frontmatter(llm_output)
digest_md = body if meta else llm_output
metadata = {
"title": meta.get("title") or f"🌙 AI Daily 晚报 | {date}",
"lead": meta.get("lead", ""),
"highlights": normalize_str_list(meta.get("highlights")),
"profile": "default",
"date": date,
}
return digest_md, metadata
def parse_immediate_push_with_metadata(
llm_output: str, default_title: str
) -> Tuple[str, Dict]:
"""解析即时推送 LLM 输出,返回 (body, metadata)。
metadata 仅含 title / profile。无 frontmatter 时降级到旧式 `# ` 标题提取,
再降级到 default_title。
"""
meta, body = parse_frontmatter(llm_output)
if meta and meta.get("title"):
return body, {"title": meta["title"], "profile": "hotspot"}
# 兼容旧格式:从正文一级标题提取
match = re.search(r"^\s*#\s+(.+?)\s*\n(.*)$", llm_output, re.DOTALL | re.MULTILINE)
if match:
return match.group(2).rstrip(), {
"title": match.group(1).strip(),
"profile": "hotspot",
}
return llm_output, {"title": default_title, "profile": "hotspot"}
ai-daily-main/src/main.py
+714
View File
@@ -0,0 +1,714 @@
"""AI每日资讯推送系统 - 主程序"""
import argparse
import asyncio
import os
import sys
from datetime import date, datetime, timedelta, timezone
from typing import Dict, List, Optional
# 加载 .env 文件
from dotenv import load_dotenv
load_dotenv()
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from croniter import croniter
from src.config import get_timezone, load_config, merge_sources
from src.fetcher import fetch_all_feeds
from src.llm import (
check_llm_available,
generate_immediate_push,
parse_immediate_push_with_metadata,
score_batch,
)
from src.processor import html_to_markdown
from src.push import send_to_platforms
from src.sections.github.section import run_github_section
from src.sections.hackernews.section import run_hackernews_section
from src.sections.insights.section import run_insights_section
from src.sections.rss.section import run_rss_section
from src.storage import (
append_entries,
assemble_with_sentinels,
cleanup_old_files,
get_fetch_file,
get_notify_file,
get_push_file,
load_existing_links,
load_recent_notify_content,
load_recent_push_content,
read_entries,
save_notify_file,
save_push_file,
)
async def notify_llm_errors(stage: str, errors: List[str], config: Dict):
"""发送简单的 LLM 异常通知"""
if not errors:
return
lines = [
"## LLM异常",
"",
f"stage: {stage}",
f"time: {now_local(config).strftime('%Y-%m-%d %H:%M:%S')}",
"",
]
lines.extend(f"- {error}" for error in errors)
try:
await send_to_platforms("\n".join(lines), config["push"])
except Exception as e:
print(f"⚠️ LLM异常通知发送失败: {e}")
def now_local(config: Dict = None) -> datetime:
"""获取配置时区的当前时间"""
return datetime.now(get_timezone(config))
def parse_time_to_local(time_str: str, config: Dict = None) -> Optional[datetime]:
"""解析时间字符串为配置时区的datetime"""
try:
dt = datetime.fromisoformat(time_str.replace("Z", "+00:00"))
return dt.astimezone(get_timezone(config))
except (ValueError, TypeError):
return None
def calculate_push_times(
cron_list: List[str], offset_days: int = 0, config: Dict = None
) -> List[datetime]:
base_date = datetime.now(get_timezone(config)).date() + timedelta(days=offset_days)
times = []
for cron in cron_list:
try:
minute, hour, _, _, _ = cron.split()
t = datetime.combine(
base_date,
datetime.strptime(f"{hour}:{minute}", "%H:%M").time(),
tzinfo=get_timezone(config),
)
times.append(t)
except ValueError:
continue
return sorted(times)
def is_morning_push(now: datetime, config: Dict) -> bool:
"""判定当前 push 是否为「早报」(触发 GH/HN/insights 三段)。
规则:在 `schedule.push_cron` 列表里,now 离哪条 cron 最近,就归为那条;
最近的那条若是当天最早的 cron,则视为早报。
特例:
- `push_cron` 为空 → 不视为早报
- `push_cron` 只有一条 → 该条即"最早"也即"最近",任何触发都视为早报
"""
cron_list = config.get("schedule", {}).get("push_cron", [])
if not cron_list:
return False
if len(cron_list) == 1:
return True
base = now.replace(hour=0, minute=0, second=0, microsecond=0)
today_fires = [croniter(c, base).get_next(datetime) for c in cron_list]
closest = min(today_fires, key=lambda f: abs(now - f))
return closest == min(today_fires)
def collect_entries_for_push(
last_push_time: Optional[datetime],
context_days: int = 2,
min_score: int = 60,
data_dir: str = "news-data",
) -> tuple[List[Dict], List[Dict]]:
"""
收集推送所需的条目，返回 (待推送条目, 上下文条目)
逻辑：
1. 获取 context_days 天内的所有条目
2. 按 min_score 过滤
3. push_time = max(last_push_time, now - 24h)
4. 晚于 push_time 的 → 待推送条目
5. 早于 push_time 的 → 上下文条目（用于LLM去重参考）
"""
tz = get_timezone()
now = datetime.now(tz)
# 获取 context_days 天的所有条目
all_entries = []
today = now.date()
for i in range(context_days):
d = today - timedelta(days=i)
fetch_file = get_fetch_file(d, data_dir)
for entry in read_entries(fetch_file):
all_entries.append(entry)
print(
f"📋 收集总条目: {len(all_entries)} 条 , context_days: {context_days}, min_score:{min_score}"
)
# 按 min_score 过滤
qualified_entries = [e for e in all_entries if (e.get("score") or 0) >= min_score]
print(f"📋 过滤后条目: {len(qualified_entries)} 条 ")
# 计算推送时间边界：max(last_push_time, now - 24h)
past_24h = now - timedelta(hours=24)
push_cutoff = (
last_push_time if last_push_time and last_push_time > past_24h else past_24h
)
print(f"推送时间边界: {push_cutoff.strftime('%Y-%m-%d %H:%M:%S')}")
# 分割条目
to_push = []
context = []
# context 只供 LLM 做去重/历史参考,不需要 content/link/fetched_at 等大字段
CONTEXT_FIELDS = ("title", "source", "score", "summary", "tags", "published")
for entry in qualified_entries:
entry_time = parse_time_to_local(entry.get("fetched_at", ""))
if entry_time and entry_time > push_cutoff:
to_push.append(entry)
else:
context.append({k: entry.get(k) for k in CONTEXT_FIELDS})
# 上下文按分数排序，取前50
context = sorted(context, key=lambda x: x.get("score", 0), reverse=True)[:50]
return to_push, context
async def run_fetch_job(config: Dict):
print(f"\n{'=' * 50}")
print(f"🔄 Fetch Job | {now_local().strftime('%Y-%m-%d %H:%M:%S')}")
print(f"{'=' * 50}")
interval = config["schedule"]["fetch_interval_minutes"]
lookback = config["schedule"].get("fetch_lookback_minutes", 120)
lookback = max(lookback, interval)
threshold = lookback + interval
cutoff = datetime.now(timezone.utc) - timedelta(minutes=lookback)
sources = merge_sources(config["sources"])
print(f"📂 共 {len(sources)} 个订阅源")
if not sources:
print("⚠️ 没有可用的订阅源")
return
max_workers = config.get("fetch", {}).get("max_workers", 20)
timeout = config.get("fetch", {}).get("timeout", 30)
entries = await fetch_all_feeds(
sources, cutoff, max_workers=max_workers, timeout=timeout
)
print(f"📥 抓取到 {len(entries)} 条原始消息")
if not entries:
return
for entry in entries:
entry["content"] = html_to_markdown(
entry.get("content", ""), entry.get("link", "")
)
fetch_file = get_fetch_file()
existing_links = load_existing_links(fetch_file, threshold)
new_entries = [
e for e in entries if e.get("link") and e["link"] not in existing_links
]
print(f"🆕 新消息 {len(new_entries)} 条 | 链接数：{len(existing_links)}")
if not new_entries:
return
print("🤖 LLM评分中...")
# 预处理：将所有 datetime 转换为字符串，避免 JSON 序列化错误
for entry in new_entries:
if isinstance(entry.get("published"), datetime):
entry["published"] = (
entry["published"].astimezone(get_timezone(config)).isoformat()
)
scored, score_errors = await score_batch(new_entries, config["llm"])
if score_errors:
print(f"⚠️ [score_batch] {len(score_errors)} 个错误: {score_errors[0]}")
await notify_llm_errors("score_batch", score_errors, config)
is_new_file = not os.path.exists(fetch_file)
if is_new_file:
cleanup_old_files(days=config["filter"]["keep_days"])
# 添加 fetched_at 时间戳
for entry in scored:
entry["fetched_at"] = now_local().isoformat()
if isinstance(entry.get("published"), datetime):
entry["published"] = (
entry["published"].astimezone(get_timezone(config)).isoformat()
)
# 批量保存到 JSON 文件
from datetime import date
meta = {"date": date.today().isoformat()}
append_entries(fetch_file, scored, meta)
print(f"💾 已保存到 {fetch_file}")
hot_threshold = config["filter"]["hot_threshold"]
no_content_marker = config["filter"].get("no_content_marker", "[NO_NEW_CONTENT]")
hot_entries = [e for e in scored if (e.get("score") or 0) >= hot_threshold]
if hot_entries:
print(f"🔥 发现 {len(hot_entries)} 条热点消息，即时推送...")
# 加载近期已推送内容（仅供 LLM 查重，避免风格趋同）
context_days = config["filter"]["context_days"]
recent_notify = load_recent_notify_content(context_days)
recent_push = load_recent_push_content(context_days)
recent_context = (
f"=== 近期即时推送 ===\n{recent_notify}\n\n"
f"=== 近期汇总推送 ===\n{recent_push}"
)
push_content, immediate_push_error = await generate_immediate_push(
hot_entries, config["llm"], recent_push_context=recent_context
)
if immediate_push_error:
print(f"⚠️ [generate_immediate_push] {immediate_push_error}")
await notify_llm_errors(
"generate_immediate_push", [immediate_push_error], config
)
if not push_content:
print("⚠️ 即时推送内容生成失败，跳过本次热点推送")
print(
f"✅ Fetch Job 完成 | 新消息: {len(scored)} 条 | 热点: {len(hot_entries)} 条"
)
return
# 检查是否有实际内容需要推送
if no_content_marker in push_content:
print(f"ℹ️ 无新内容需要推送 (LLM判定为重复内容)")
else:
# 提取标题并构建 metadata
now = now_local(config)
timestamp = now.strftime("%Y-%m-%d %H:%M")
content_without_title, metadata = parse_immediate_push_with_metadata(
push_content, f"🚨 AI Daily 快讯 | {timestamp}"
)
metadata["pushTime"] = now.isoformat()
await send_to_platforms(
content_without_title,
config["push"],
"🚨 AI Daily 快讯 | " + metadata["title"],
metadata=metadata,
)
# 保存即时推送内容到notify文件
notify_file = get_notify_file()
save_notify_file(notify_file, content_without_title, metadata)
print(f"💾 已保存即时推送到 {notify_file}")
print(f"✅ Fetch Job 完成 | 新消息: {len(scored)} 条 | 热点: {len(hot_entries)} 条")
async def run_push_job(config: Dict):
print(f"\n{'=' * 50}")
print(f"📤 Push Job | {now_local().strftime('%Y-%m-%d %H:%M:%S')}")
print(f"{'=' * 50}")
if is_morning_push(now_local(config), config):
await _run_morning_push(config)
else:
await _run_default_push(config)
async def _run_default_push(config: Dict):
"""晚报或非早报时段:沿用原有纯 RSS digest 流程,委托给 run_rss_section"""
now = now_local(config)
rss_md, metadata, rss_err = await run_rss_section(config, now)
if rss_err and not rss_md:
print(f"⚠️ [compose_digest] {rss_err}")
await notify_llm_errors("compose_digest", [rss_err], config)
raise RuntimeError(f"RSS section failed: {rss_err}")
if not rss_md:
# run_rss_section 在无新消息时已打印 "ℹ️ RSS: 无新消息"
return
# metadata 缺失兜底:parse_digest_with_metadata 失败 / LLM 输出无 frontmatter 时可能返回 None
if not metadata:
date_str = now.strftime("%Y-%m-%d")
metadata = {
"title": f"🌙 AI Daily 晚报 | {date_str}",
"lead": "",
"highlights": [],
"profile": "default",
"date": date_str,
}
metadata.setdefault("pushTime", now.isoformat())
await send_to_platforms(
rss_md,
config["push"],
title="📰 AI Daily 每日精选 | " + metadata["title"],
metadata=metadata,
)
push_file = get_push_file()
rss_count = rss_md.count("###")
save_push_file(
push_file,
rss_md,
rss_count,
rss_count,
profile="default",
metadata=metadata,
)
print(f"💾 已保存到 {push_file}")
print(f"✅ Push Job 完成 | 推送条目: {rss_count}")
async def _run_morning_push(config: Dict):
"""早报四模块编排:RSS/GH/HN 并发 → insights 串行 → sentinel 拼装 → 推送 → 落盘。
失败语义:
- RSS 失败 → 整体抛 RuntimeError (核心承诺不变)
- GH/HN/insights 失败 → 该段省略 + 告警,其他段照推
"""
now = now_local(config)
rss_result, gh_result, hn_result = await asyncio.gather(
run_rss_section(config, now),
run_github_section(config, now),
run_hackernews_section(config, now),
)
# 早报场景:digest 的 metadata 通常会被 insights 段覆盖,
# 但保留以便在 insights 失败时作为兜底来源(title 关键词 / lead / highlights)
rss_md, digest_meta, rss_err = rss_result
gh_md, gh_err = gh_result
hn_md, hn_err = hn_result
if gh_err:
print(f"⚠️ [section_github] {gh_err}")
await notify_llm_errors("section_github", [gh_err], config)
if hn_err:
print(f"⚠️ [section_hackernews] {hn_err}")
await notify_llm_errors("section_hackernews", [hn_err], config)
if rss_err and not rss_md:
print(f"⚠️ [compose_digest] {rss_err}")
await notify_llm_errors("compose_digest", [rss_err], config)
raise RuntimeError(f"RSS section failed: {rss_err}")
insights_md, metadata, insights_err = await run_insights_section(
rss_md, gh_md, hn_md, config, now
)
if insights_err:
print(f"⚠️ [insights] {insights_err}")
await notify_llm_errors("insights", [insights_err], config)
# 如果 insights 失败,优先用 digest metadata 兜底;两者都缺再走默认
if not metadata:
date_str = now.strftime("%Y-%m-%d")
fallback = digest_meta or {}
digest_title = fallback.get("title", "")
title = digest_title if digest_title else f"📰 AI Daily 每日精选 | {date_str}"
metadata = {
"date": date_str,
"pushTime": now.isoformat(),
"title": title,
"excerpt": "",
"seotitle": "",
"seodescription": "",
"lead": fallback.get("lead", ""),
"highlights": fallback.get("highlights", []),
"profile": "morning",
}
else:
metadata.setdefault("pushTime", now.isoformat())
final = assemble_with_sentinels(
{
"rss": rss_md,
"github": gh_md,
"hackernews": hn_md,
"insights": insights_md,
}
)
if not final.strip():
print("ℹ️ 早报无任何段输出,跳过推送")
return
await send_to_platforms(
final,
config["push"],
title="📰 AI Daily 每日精选 | " + metadata["title"],
metadata=metadata,
)
push_file = get_push_file()
rss_count = rss_md.count("###") if rss_md else 0
save_push_file(
push_file, final, rss_count, rss_count, profile="morning", metadata=metadata
)
print(f"💾 已保存早报到 {push_file}")
async def fetch_loop(config: Dict):
"""Fetch循环 - 修复时间漂移并支持优雅退出"""
import time
interval_seconds = config["schedule"]["fetch_interval_minutes"] * 60
print(f"🔄 Fetch循环已启动 | 严格间隔: {interval_seconds / 60}分钟")
while True:
start_time = time.monotonic() # 使用 monotonic 避免系统时间修改影响
try:
await run_fetch_job(config)
except asyncio.CancelledError:
print("⚠️ Fetch循环被外部取消，正在安全退出...")
break # 允许外部取消任务
except Exception as e:
print(f"❌ Fetch Job 失败: {e}")
# 计算任务耗时
elapsed = time.monotonic() - start_time
# 计算还需要睡多久（如果任务耗时超过间隔，则不睡，立刻进入下一次）
sleep_time = max(0.0, interval_seconds - elapsed)
if sleep_time > 0:
print(f"⏰ 下次抓取: {sleep_time / 60:.1f}分钟后")
try:
await asyncio.sleep(sleep_time)
except asyncio.CancelledError:
print("⚠️ 睡眠被中断，Fetch循环安全退出...")
break
async def push_loop(config: Dict):
"""Push循环 - 无状态 croniter + 原生异步睡眠"""
cron_list = config["schedule"]["push_cron"]
tz = get_timezone(config)
# 1. 启动前预校验 cron 表达式，过滤掉无效配置
valid_crons = []
for cron in cron_list:
if croniter.is_valid(cron):
valid_crons.append(cron)
else:
print(f"⚠️ 忽略无效的 cron 表达式: '{cron}'")
if not valid_crons:
print("❌ 没有有效的推送时间配置，Push循环退出")
return
print(f"📤 Push循环已启动 | 定时: {', '.join(valid_crons)} | 时区: {tz}")
# 2. 主循环
while True:
try:
now = datetime.now(tz)
# 💡 核心优化：无状态计算。
# 每次都基于此刻的真实时间，动态计算所有有效 cron 的下一次时间，取最近的一个。
# 这样无论 run_push_job 执行多久，或者系统休眠过，永远都不会算错。
next_push = min(
croniter(cron, now).get_next(datetime) for cron in valid_crons
)
wait_seconds = (next_push - datetime.now(tz)).total_seconds()
if wait_seconds > 0:
print(
f"⏰ 下次推送: {next_push.strftime('%Y-%m-%d %H:%M:%S')} (等待 {wait_seconds / 60:.1f} 分钟)"
)
# 💡 核心优化：直接 Sleep。asyncio 天生支持被 CancelledError 瞬间打断
await asyncio.sleep(wait_seconds)
# 到达推送时间，执行推送
print(f"📤 执行推送: {datetime.now(tz).strftime('%Y-%m-%d %H:%M:%S')}")
await run_push_job(config)
# 增加 1 秒缓冲：防止 run_push_job 执行过快（不到 1 秒），
# 导致下一个循环的 now 仍停留在当前秒，croniter 算出重复的时间点。
await asyncio.sleep(1)
except asyncio.CancelledError:
print("⚠️ Push循环收到取消信号，安全退出...")
break # 直接 break 退出循环即可
except Exception as e:
print(f"❌ Push 循环异常: {e}")
# 遇到未知异常时休眠 60 秒，防止死循环疯狂报错打满日志
await asyncio.sleep(60)
async def cmd_check(config: Dict) -> int:
"""校验 LLM 接口可达性（部署期使用，运行期不再校验）"""
print("🔍 校验 LLM 接口...")
try:
await check_llm_available(config["llm"])
except Exception as e:
print(f"❌ LLM 接口不可用: {e}")
return 1
print("✅ LLM 接口可用")
return 0
async def cmd_fetch(config: Dict) -> int:
"""单次抓取（systemd timer 调用）"""
try:
await run_fetch_job(config)
return 0
except Exception as e:
print(f"❌ Fetch 任务失败: {e}")
return 1
async def cmd_push(config: Dict) -> int:
"""单次推送（systemd timer 调用）"""
try:
await run_push_job(config)
return 0
except Exception as e:
print(f"❌ Push 任务失败: {e}")
return 1
async def cmd_loop(config: Dict) -> int:
"""长跑模式（本地开发/调试用）"""
print("🔍 检查 LLM 接口可用性...")
try:
await check_llm_available(config["llm"])
print("✅ LLM 接口可用")
except Exception as e:
print(f"❌ LLM 接口不可用: {e}")
return 1
await asyncio.gather(fetch_loop(config), push_loop(config))
return 0
async def cmd_rss(config: Dict) -> int:
"""单独跑一次 RSS digest 板块,打印结果不推送"""
print("📰 RSS Digest 单板块运行")
try:
md, meta, err = await run_rss_section(config, now=now_local(config))
except Exception as e:
print(f"❌ RSS 板块失败: {e}")
return 1
if err:
print(f"❌ {err}")
return 1
if not md:
print("ℹ️ 本次无内容")
return 0
print("\n" + "=" * 60)
print("📑 metadata:")
if meta:
import json as _json
print(_json.dumps(meta, ensure_ascii=False, indent=2))
else:
print("(none)")
print("=" * 60)
print(md)
print("=" * 60)
return 0
async def cmd_github(config: Dict) -> int:
"""单独跑一次 GitHub trending 板块,打印结果不推送"""
print("⭐ GitHub Trending 单板块运行")
try:
md, err = await run_github_section(config, now=now_local(config))
except Exception as e:
print(f"❌ GitHub 板块失败: {e}")
return 1
if err:
print(f"❌ {err}")
return 1
if not md:
print("ℹ️ 本次无内容")
return 0
print("\n" + "=" * 60)
print(md)
print("=" * 60)
return 0
async def cmd_hackernews(config: Dict) -> int:
"""单独跑一次 Hacker News 板块,打印结果不推送"""
print("🟧 Hacker News 单板块运行")
try:
md, err = await run_hackernews_section(config, now=now_local(config))
except Exception as e:
print(f"❌ Hacker News 板块失败: {e}")
return 1
if err:
print(f"❌ {err}")
return 1
if not md:
print("ℹ️ 本次无内容")
return 0
print("\n" + "=" * 60)
print(md)
print("=" * 60)
return 0
def _parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="daily-news",
description="AI 每日资讯推送系统",
)
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser("check", help="校验 LLM 接口可达性")
sub.add_parser("fetch", help="单次抓取并退出")
sub.add_parser("push", help="单次推送并退出")
sub.add_parser("loop", help="长跑模式（开发/调试用）")
sub.add_parser("rss", help="单独跑一次 RSS Digest 板块（仅打印,不推送）")
sub.add_parser("github", help="单独跑一次 GitHub Trending 板块（仅打印,不推送）")
sub.add_parser("hackernews", help="单独跑一次 Hacker News 板块（仅打印,不推送）")
return parser.parse_args()
def main() -> int:
print("🚀 AI每日资讯推送系统")
args = _parse_args()
try:
config = load_config()
print("✅ 配置加载成功")
except Exception as e:
print(f"❌ 加载配置失败: {e}")
return 1
handlers = {
"check": cmd_check,
"fetch": cmd_fetch,
"push": cmd_push,
"loop": cmd_loop,
"rss": cmd_rss,
"github": cmd_github,
"hackernews": cmd_hackernews,
}
return asyncio.run(handlers[args.command](config))
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
print("\n👋 程序已退出")
sys.exit(0)
ai-daily-main/src/markdown_utils.py
+83
View File
@@ -0,0 +1,83 @@
"""Markdown / YAML frontmatter 公共辅助。
供 storage、llm 等模块复用,避免重复实现。本模块零业务依赖,仅依赖 yaml/json/re。
"""
import json
import re
from typing import Any, Dict, List, Tuple
import yaml
__all__ = [
"yaml_value",
"dump_frontmatter",
"parse_frontmatter",
"normalize_str_list",
]
def yaml_value(v: Any) -> str:
"""把单个值序列化为 YAML 合法的 token,借道 JSON 语法。
依据:JSON 是 YAML 1.2 的真子集,任何 json.dumps 的输出都是合法 YAML 标量/序列/映射。
始终带引号的字符串可以避免 PyYAML 的若干怪癖(折行、未引号字符串歧义、unicode 转义)。
"""
if isinstance(v, (dict, list, str)):
return json.dumps(v, ensure_ascii=False)
if isinstance(v, bool):
return "true" if v else "false"
if v is None:
return ""
return str(v)
def dump_frontmatter(meta: Dict) -> str:
"""把扁平 metadata dict 序列化为 frontmatter 文本(不含包围的 `---`)。
保留插入顺序(title 在前,bookkeeping 字段在后)。仅支持扁平结构 —— 当前所有
metadata 都是扁平的,无需处理嵌套。
"""
return "".join(f"{k}: {yaml_value(v)}\n" for k, v in meta.items())
_FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n?(.*)$", re.DOTALL)
def parse_frontmatter(text: str) -> Tuple[Dict, str]:
"""从 markdown 文本中分离 YAML frontmatter 与正文。
返回 (metadata_dict, body)。无 frontmatter / YAML 解析失败 / 非 dict 时返回
({}, 原文) —— 保证降级路径不丢内容,调用方可凭 metadata_dict 是否为空判定。
"""
match = _FRONTMATTER_RE.match(text.strip())
if not match:
return {}, text
try:
meta = yaml.safe_load(match.group(1)) or {}
except yaml.YAMLError:
print("frontmatter 数据段解析失败")
return {}, text
if not isinstance(meta, dict):
return {}, text
return meta, match.group(2).strip()
def normalize_str_list(value: Any) -> List[str]:
"""将 str/list/None 规整为非空字符串列表。
用于兜底 LLM 输出的列表型字段(如 frontmatter 中的 highlights):单字符串包裹为单元素列表,
null/非列表/非字符串返回空列表,列表中夹杂的空白项过滤掉。不做数量截断。
"""
if not value:
return []
if isinstance(value, str):
items = [value]
elif isinstance(value, list):
items = value
else:
return []
return [str(x).strip() for x in items if str(x).strip()]
ai-daily-main/src/processor.py
+37
View File
@@ -0,0 +1,37 @@
"""内容处理模块 - HTML转Markdown"""
import re
from urllib.parse import urljoin
from markdownify import markdownify as md
def html_to_markdown(html: str, base_url: str = "") -> str:
"""
将HTML转换为Markdown，保留链接和图片
使用markdownify库，并进行后处理优化
"""
# markdownify会自动处理<img>为![](url)，<a>为[text](url)
markdown = md(html, heading_style="ATX")
# 处理相对链接
if base_url:
def replace_rel_link(m):
prefix, path, suffix = m.groups()
if path.startswith(("http://", "https://", "data:")):
return m.group(0)
abs_url = urljoin(base_url, path)
return f"{prefix}{abs_url}{suffix}"
markdown = re.sub(r"(!?\[.*?\]\()(.*?)(\))", replace_rel_link, markdown)
# 后处理优化
# 1. 直接匹配移除 xgo.ing 推广链接
markdown = markdown.replace("[⚡ Powered by xgo.ing](https://xgo.ing)", "")
markdown = markdown.replace("[⚡ Powered by xgo.ing](https://xgo.ing/)", "")
# 2. 清理多余空行
markdown = re.sub(r"\n{3,}", "\n\n", markdown)
return markdown.strip()
ai-daily-main/src/push/__init__.py
+42
View File
@@ -0,0 +1,42 @@
"""推送平台模块"""
from typing import Dict, Optional
from .base import PushPlatform
from .discord import DiscordPlatform
from .feishu import FeishuPlatform
from .custom import CustomPlatform
def create_platform(name: str, config: Dict) -> Optional[PushPlatform]:
"""工厂函数，创建推送平台实例"""
platforms = {
"discord": DiscordPlatform,
"feishu": FeishuPlatform,
"custom": CustomPlatform,
}
if name not in platforms:
raise ValueError(f"未知推送平台: {name}")
platform_class = platforms[name]
platform = platform_class(config)
if not platform.validate_config(config):
return None
return platform
async def send_to_platforms(content: str, push_config: Dict, title: str = None, metadata: Optional[Dict] = None):
"""发送内容到所有已启用且配置有效的平台"""
for platform_name, platform_conf in push_config.items():
platform = create_platform(platform_name, platform_conf)
if platform is None:
continue
try:
await platform.send(content, title, metadata)
print(f"✅ 已推送到 {platform_name}")
except Exception as e:
print(f"❌ 推送到 {platform_name} 失败: {e}")
ai-daily-main/src/push/base.py
+26
View File
@@ -0,0 +1,26 @@
"""推送平台基类"""
from abc import ABC, abstractmethod
from typing import Dict
class PushPlatform(ABC):
"""推送平台抽象基类"""
def __init__(self, config: Dict):
self.config = config
@abstractmethod
def validate_config(self, config: Dict) -> bool:
"""验证配置是否有效"""
pass
@abstractmethod
async def send(self, content: str, title: str = None, metadata: Dict = None):
"""发送内容
Args:
content: 正文内容
title: 标题（可选，兼容旧接口）
metadata: 元信息（可选，新增参数）
"""
pass
ai-daily-main/src/push/custom.py
+54
View File
@@ -0,0 +1,54 @@
"""自定义 API 推送平台"""
import os
from typing import Dict, Optional
import aiohttp
from .base import PushPlatform
class CustomPlatform(PushPlatform):
"""自定义 API 推送平台"""
def validate_config(self, config: Dict) -> bool:
"""验证配置"""
if not config.get("enabled", False):
return False
api_key_name = config.get("apiKeyName")
token_key_name = config.get("tokenKeyName")
if not api_key_name or not token_key_name:
print("❌ Custom 平台配置缺少 apiKeyName 或 tokenKeyName")
return False
url = os.getenv(api_key_name)
token = os.getenv(token_key_name)
if not url or not token:
print(f"❌ 环境变量 {api_key_name} 或 {token_key_name} 未设置")
return False
return True
async def send(self, content: str, title: str = None, metadata: Optional[Dict] = None):
"""发送到自定义 API"""
api_key_name = self.config.get("apiKeyName")
token_key_name = self.config.get("tokenKeyName")
url = os.getenv(api_key_name)
token = os.getenv(token_key_name)
payload = {
"content": content,
"metadata": metadata
}
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers) as resp:
if resp.status != 200:
error_text = await resp.text()
raise Exception(f"API 返回错误 {resp.status}: {error_text}")
ai-daily-main/src/push/discord.py
+60
View File
@@ -0,0 +1,60 @@
"""Discord推送平台"""
import os
from typing import Dict
import aiohttp
from .base import PushPlatform
class DiscordPlatform(PushPlatform):
"""Discord Webhook推送"""
def __init__(self, config: Dict):
super().__init__(config)
self.api_key_name = config.get("apiKeyName", "DISCORD_WEBHOOK_URL")
self.webhook_url = os.environ.get(self.api_key_name, "")
def validate_config(self, config: Dict) -> bool:
"""检查Discord配置是否有效"""
if not config.get("enabled", False):
return False
api_key_name = config.get("apiKeyName", "DISCORD_WEBHOOK_URL")
webhook = os.environ.get(api_key_name, "")
return bool(webhook and webhook.startswith("https://discord.com/api/webhooks/"))
async def send(self, content: str, title: str = None, metadata: Dict = None):
"""发送到Discord（忽略 metadata）"""
full_content = f"# {title}\n\n{content}" if title else content
chunks = self._split_content(full_content, limit=2000)
async with aiohttp.ClientSession() as session:
for chunk in chunks:
payload = {"content": chunk}
async with session.post(self.webhook_url, json=payload) as resp:
if resp.status != 204:
text = await resp.text()
raise RuntimeError(f"Discord推送失败: {resp.status} - {text}")
def _split_content(self, content: str, limit: int = 2000) -> list:
"""Discord限制2000字符，需要分割"""
if len(content) <= limit:
return [content]
chunks = []
lines = content.split("\n")
current = ""
for line in lines:
if len(current) + len(line) + 1 > limit:
if current:
chunks.append(current)
current = line
else:
current += "\n" + line if current else line
if current:
chunks.append(current)
return chunks
ai-daily-main/src/push/feishu.py
+92
View File
@@ -0,0 +1,92 @@
"""飞书推送平台"""
import os
from typing import Dict
import aiohttp
from .base import PushPlatform
class FeishuPlatform(PushPlatform):
"""飞书 Webhook 推送"""
def __init__(self, config: Dict):
super().__init__(config)
self.api_key_name = config.get("apiKeyName", "FEISHU_WEBHOOK_URL")
self.webhook_url = os.environ.get(self.api_key_name, "")
def validate_config(self, config: Dict) -> bool:
"""检查飞书配置是否有效"""
if not config.get("enabled", False):
return False
api_key_name = config.get("apiKeyName", "FEISHU_WEBHOOK_URL")
webhook = os.environ.get(api_key_name, "")
return bool(webhook)
async def send(self, content: str, title: str = None, metadata: Dict = None):
"""发送到飞书（忽略 metadata）"""
chunks = self._split_content(content, limit=8000)
async with aiohttp.ClientSession() as session:
for chunk in chunks:
payload = self._build_payload(chunk, title)
async with session.post(self.webhook_url, json=payload) as resp:
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"飞书推送失败: {resp.status} - {text}")
data = await resp.json()
if data.get("code") != 0:
raise RuntimeError(f"飞书推送失败: {data.get('msg')}")
def _build_payload(self, content: str, title: str = None) -> Dict:
"""
构建飞书卡片消息 payload，支持 Markdown，
参考 https://open.feishu.cn/document/feishu-cards/card-json-v2-structure
"""
header = {}
if title:
header = {
"title": {"content": title, "tag": "plain_text"},
"template": "blue",
}
return {
"msg_type": "interactive",
"card": {
"schema": "2.0", # 【重点1】显式声明使用 V2 版本结构
"header": header,
"body": { # 【重点2】V2 中，所有的内容元素都必须放在 body 里面
"elements": [
{
"tag": "markdown",
"content": content,
"text_align": "left", # 可选：left / center / right
},
],
},
},
}
def _split_content(self, content: str, limit: int = 8000) -> list:
"""飞书卡片消息 markdown 元素限制 8000 字符"""
if len(content) <= limit:
return [content]
chunks = []
lines = content.split("\n")
current = ""
for line in lines:
if len(current) + len(line) + 1 > limit:
if current:
chunks.append(current)
current = line
else:
current += "\n" + line if current else line
if current:
chunks.append(current)
return chunks
ai-daily-main/src/sections/__init__.py
+1
View File
@@ -0,0 +1 @@
"""板块模块包。每个子模块导出 run_<board>_section(config, now) -> (markdown, error)"""
ai-daily-main/src/sections/github/__init__.py
View File
ai-daily-main/src/sections/github/repo_enricher.py
+111
View File
@@ -0,0 +1,111 @@
"""GitHub REST API enrich:metadata + README → enriched repo dict
匿名调用受 60 req/hr 限,设置 GITHUB_TOKEN 环境变量后走 5000 req/hr。
"""
import asyncio
import base64
import os
from typing import Dict, List, Optional, Tuple
import aiohttp
API_BASE = "https://api.github.com"
def _auth_headers(token_env: str = "GITHUB_TOKEN") -> Dict[str, str]:
headers = {"Accept": "application/vnd.github+json"}
token = os.environ.get(token_env)
if token:
headers["Authorization"] = f"Bearer {token}"
print(f"🔑 GH: 已配置 {token_env},鉴权调用 (5000 req/hr)")
else:
print(f"⚠️ GH: 未配置 {token_env},匿名调用 (60 req/hr)")
return headers
async def _get_json(
session: aiohttp.ClientSession, url: str, timeout: int = 10
) -> Optional[Dict]:
async with session.get(
url, timeout=aiohttp.ClientTimeout(total=timeout)
) as resp:
if resp.status == 404:
return None
if resp.status != 200:
raise RuntimeError(f"GitHub API {resp.status} for {url}")
return await resp.json()
async def enrich_repo(
session: aiohttp.ClientSession,
repo: Dict,
token_env: str = "GITHUB_TOKEN",
readme_max_chars: int = 3000,
timeout: int = 10,
) -> Optional[Dict]:
"""单 repo 双调用 enrich。返回 None 表示该 repo 应剔除(archived 或 metadata 不可达)。
任一调用失败 raise → 调用方按 return_exceptions 模式聚合错误。
"""
full_name = repo["full_name"]
meta_url = f"{API_BASE}/repos/{full_name}"
readme_url = f"{API_BASE}/repos/{full_name}/readme"
meta, readme = await asyncio.gather(
_get_json(session, meta_url, timeout=timeout),
_get_json(session, readme_url, timeout=timeout),
)
if meta is None:
return None
if meta.get("archived"):
return None
license_spdx = ""
if isinstance(meta.get("license"), dict):
license_spdx = meta["license"].get("spdx_id") or ""
readme_excerpt = ""
if readme and readme.get("content"):
try:
raw = base64.b64decode(readme["content"]).decode("utf-8", errors="replace")
readme_excerpt = raw[:readme_max_chars]
except Exception:
readme_excerpt = ""
return {
**repo,
"topics": meta.get("topics") or [],
"license": license_spdx,
"pushed_at": meta.get("pushed_at") or "",
"readme_excerpt": readme_excerpt,
}
async def enrich_repos(
candidates: List[Dict],
token_env: str = "GITHUB_TOKEN",
readme_max_chars: int = 3000,
timeout: int = 10,
) -> Tuple[List[Dict], List[str]]:
"""并发 enrich 多个 repo。返回 (enriched_list_with_archived_filtered, errors)"""
errors: List[str] = []
headers = _auth_headers(token_env)
async with aiohttp.ClientSession(headers=headers) as session:
results = await asyncio.gather(
*[
enrich_repo(session, r, token_env, readme_max_chars, timeout)
for r in candidates
],
return_exceptions=True,
)
enriched: List[Dict] = []
for r, candidate in zip(results, candidates):
if isinstance(r, Exception):
errors.append(f"enrich {candidate['full_name']} 失败: {r}")
elif r is not None:
enriched.append(r)
return enriched, errors
ai-daily-main/src/sections/github/section.py
+104
View File
@@ -0,0 +1,104 @@
"""GitHub Trending 板块入口。
流程:trending 抓取 → history 过滤 → 候选写回 history → deep-dive → LLM 总结
"""
from datetime import datetime
from typing import Dict, Optional, Tuple
from src.config import get_timezone
from src.sections.github.repo_enricher import enrich_repos
from src.sections.github.trending_scraper import (
fetch_trending_page,
parse_trending_html,
)
from src.storage import load_trending_history
async def run_github_section(
config: Dict, now: Optional[datetime] = None
) -> Tuple[str, Optional[str]]:
cfg = config.get("sections", {}).get("github_trending", {})
if not cfg.get("enabled", False):
return "", None
# 延迟 import,Task 11 才提供 summarize_github_trending
from src.llm import summarize_github_trending
today = (now or datetime.now(get_timezone())).date()
keep_days = config["filter"]["keep_days"]
timeout = cfg.get("request_timeout", 10)
max_deep_dive = cfg.get("max_deep_dive", 10)
readme_max_chars = cfg.get("readme_max_chars", 3000)
history_path = cfg.get("history_file", "news-data/trending-history.json")
token_env = cfg.get("tokenName", "GITHUB_TOKEN")
# 1. 抓取
print("📥 GH: 抓取 trending 页...")
try:
html = await fetch_trending_page(timeout=timeout)
except Exception as e:
return "", f"GH 抓取失败: {e}"
all_repos = parse_trending_html(html)
print(f"📋 GH: 解析 {len(all_repos)} 个 repo")
if not all_repos:
return "", None
# 2. history 加载 + 清理
history = load_trending_history(history_path)
before_cleanup = len(history.repos)
history.cleanup(today=today, keep_days=keep_days)
after_cleanup = len(history.repos)
if before_cleanup != after_cleanup:
print(
f"🧹 GH: history 清理过期 {before_cleanup - after_cleanup} 条 (剩 {after_cleanup})"
)
# 3. 候选筛选(按 spec §4.3 语义)
candidates = []
already_seen = 0
for repo in all_repos:
if repo["url"] in history:
history.touch(repo["url"], today)
already_seen += 1
else:
candidates.append(repo)
print(f"🔍 GH: history 过滤掉 {already_seen} 条已见,新候选 {len(candidates)} 条")
if not candidates:
print("ℹ️ GH: 无新候选,跳过")
return "", None
if len(candidates) > max_deep_dive:
print(f"✂️ GH: 候选 {len(candidates)} 超 max_deep_dive={max_deep_dive},截断")
candidates = candidates[:max_deep_dive]
# 4. 候选写回 history + 持久化
for repo in candidates:
history.touch(repo["url"], today)
history.save()
# 5. 并发 enrich
print(f"🌐 GH: 并发 enrich {len(candidates)} 个 repo (REST API)...")
enriched, enrich_errors = await enrich_repos(
candidates,
token_env=token_env,
readme_max_chars=readme_max_chars,
timeout=timeout,
)
for e in enrich_errors:
print(f"⚠️ GH enrich: {e}")
print(
f"✅ GH: enrich 成功 {len(enriched)} / 失败 {len(enrich_errors)} / "
f"输入 {len(candidates)}"
)
if not enriched:
return "", None
# 6. LLM 总结
print(f"🤖 GH: summarize {len(enriched)} 个候选...")
md, err = await summarize_github_trending(enriched, config["llm"])
if err:
return "", f"summarize_github_trending: {err}"
print(f"✅ GH: 板块输出 {len(md or '')} chars")
return md or "", None
ai-daily-main/src/sections/github/trending_scraper.py
+98
View File
@@ -0,0 +1,98 @@
"""GitHub Trending 单页 HTML 抓取与解析。
数据源: https://github.com/trending (无 language / since 过滤)
"""
import re
from typing import Dict, List
import aiohttp
from bs4 import BeautifulSoup
TRENDING_URL = "https://github.com/trending"
USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
_NUM_RE = re.compile(r"[\d,]+")
def _parse_int(s: str) -> int:
m = _NUM_RE.search(s or "")
if not m:
return 0
return int(m.group(0).replace(",", ""))
async def fetch_trending_page(timeout: int = 10) -> str:
"""抓取 trending 页 HTML;非 200 抛 RuntimeError"""
async with aiohttp.ClientSession(
headers={"User-Agent": USER_AGENT}
) as session:
async with session.get(
TRENDING_URL, timeout=aiohttp.ClientTimeout(total=timeout)
) as resp:
if resp.status != 200:
text = await resp.text()
raise RuntimeError(
f"GitHub trending 返回 {resp.status}: {text[:200]}"
)
return await resp.text()
def parse_trending_html(html: str) -> List[Dict]:
"""解析 trending HTML,返回去重后的 repo 字典数组。"""
if not html:
return []
soup = BeautifulSoup(html, "html.parser")
seen_urls = set()
repos: List[Dict] = []
for article in soup.select("article.Box-row"):
h2 = article.find("h2")
a = h2.find("a") if h2 else None
if not a or not a.get("href"):
continue
href = a["href"].strip()
full_name = href.lstrip("/")
url = f"https://github.com/{full_name}"
if url in seen_urls:
continue
seen_urls.add(url)
# description
desc_tag = article.find("p")
description = (desc_tag.get_text(strip=True) if desc_tag else "") or ""
# language
lang_tag = article.find("span", attrs={"itemprop": "programmingLanguage"})
language = (lang_tag.get_text(strip=True) if lang_tag else "") or ""
# stars_total: 第一个指向 /stargazers 的链接
stars_total = 0
star_a = article.find("a", href=re.compile(r"/stargazers$"))
if star_a:
stars_total = _parse_int(star_a.get_text(strip=True))
# stars_today: 末尾的 "N stars today" span
stars_today = 0
for span in article.find_all("span"):
t = span.get_text(strip=True)
if "stars today" in t or "stars this week" in t or "stars this month" in t:
stars_today = _parse_int(t)
break
repos.append(
{
"url": url,
"full_name": full_name,
"description": description,
"language": language,
"stars_today": stars_today,
"stars_total": stars_total,
}
)
return repos
ai-daily-main/src/sections/hackernews/__init__.py
View File
ai-daily-main/src/sections/hackernews/frontpage_scraper.py
+91
View File
@@ -0,0 +1,91 @@
"""HN 首页 HTML 抓取与解析。
数据源: https://news.ycombinator.com/news (30 条)
"""
import re
from typing import Dict, List
import aiohttp
from bs4 import BeautifulSoup
FRONTPAGE_URL = "https://news.ycombinator.com/news"
USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
_NUM_RE = re.compile(r"\d+")
def _first_int(text: str) -> int:
m = _NUM_RE.search(text or "")
return int(m.group(0)) if m else 0
async def fetch_frontpage(timeout: int = 10) -> str:
async with aiohttp.ClientSession(headers={"User-Agent": USER_AGENT}) as session:
async with session.get(
FRONTPAGE_URL, timeout=aiohttp.ClientTimeout(total=timeout)
) as resp:
if resp.status != 200:
raise RuntimeError(f"HN frontpage 返回 {resp.status}")
return await resp.text()
def parse_frontpage_html(html: str) -> List[Dict]:
"""解析首页 HTML,返回 [{id, title, url, site, points, comments, comments_url}]
注:HN frontpage 的 athing 行的实际 class 是 'athing submission' (多类),
用 CSS 选择器 'tr.athing' 仍然匹配。
"""
if not html:
return []
soup = BeautifulSoup(html, "html.parser")
stories: List[Dict] = []
for athing in soup.select("tr.athing"):
item_id = athing.get("id")
if not item_id:
continue
title_a = athing.select_one("span.titleline > a")
if not title_a:
continue
title = title_a.get_text(strip=True)
href = title_a.get("href", "")
if href.startswith("item?id="):
url = f"https://news.ycombinator.com/{href}"
site = ""
else:
url = href
site_tag = athing.select_one("span.sitestr")
site = site_tag.get_text(strip=True) if site_tag else ""
sub_tr = athing.find_next_sibling("tr")
points = 0
comments = 0
comments_url = f"https://news.ycombinator.com/item?id={item_id}"
if sub_tr:
score = sub_tr.select_one("span.score")
if score:
points = _first_int(score.get_text(strip=True))
comment_a = None
for a in sub_tr.find_all("a", href=re.compile(r"^item\?id=")):
comment_a = a
if comment_a:
comments = _first_int(comment_a.get_text(strip=True))
comments_url = f"https://news.ycombinator.com/{comment_a['href']}"
stories.append(
{
"id": item_id,
"title": title,
"url": url,
"site": site,
"points": points,
"comments": comments,
"comments_url": comments_url,
}
)
return stories
ai-daily-main/src/sections/hackernews/item_enricher.py
+237
View File
@@ -0,0 +1,237 @@
"""HN 单 story enrich:Algolia 评论树 + 外链正文。
Algolia API: GET /api/v1/items/{id}
- root.text 是 Show HN / Ask HN 的 post 正文
- root.children[] 是顶层评论(按 HN ranking 排序)
- 每条 child 自己还有 children[],承载嵌套回复
enrich 策略:取 L1 + 每个 L1 下前 N 条 L2 回复,合并为 tree JSON:
[{"l1": "顶层评论", "replies": ["回复 1", "回复 2"]}, ...]
外链正文优先走 Jina Reader (https://r.jina.ai/<url>, 返回 markdown),
失败回退到直接 GET + html_to_markdown。Jina API key 可选(配置后免费额度更高),
环境变量名通过 `sections.hackernews.jinaTokenName` 配置(默认 JINA_API_KEY)。
"""
import asyncio
import os
from typing import Dict, List, Optional, Tuple
import aiohttp
from src.processor import html_to_markdown
USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
JINA_READER_BASE = "https://r.jina.ai"
def _is_internal_hn_url(url: str) -> bool:
return url.startswith("https://news.ycombinator.com/item?id=")
async def _fetch_algolia_item(
session: aiohttp.ClientSession, item_id: str, algolia_base: str, timeout: int
) -> Dict:
url = f"{algolia_base}/items/{item_id}"
async with session.get(
url, timeout=aiohttp.ClientTimeout(total=timeout)
) as resp:
if resp.status != 200:
raise RuntimeError(f"Algolia /items/{item_id} 返回 {resp.status}")
return await resp.json()
async def _fetch_url_html(
session: aiohttp.ClientSession, url: str, timeout: int
) -> str:
async with session.get(
url, timeout=aiohttp.ClientTimeout(total=timeout)
) as resp:
if resp.status != 200:
raise RuntimeError(f"外链 {url} 返回 {resp.status}")
return await resp.text()
async def _fetch_via_jina(
session: aiohttp.ClientSession,
url: str,
timeout: int,
jina_token_env: str = "JINA_API_KEY",
) -> str:
"""通过 Jina Reader 拉取外链 markdown。`jina_token_env` 指定 API key 环境变量名,配置后免费额度更高。"""
jina_url = f"{JINA_READER_BASE}/{url}"
headers = {"Accept": "text/markdown"}
api_key = os.environ.get(jina_token_env) if jina_token_env else None
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
async with session.get(
jina_url, headers=headers, timeout=aiohttp.ClientTimeout(total=timeout)
) as resp:
if resp.status != 200:
raise RuntimeError(f"Jina Reader {url} 返回 {resp.status}")
return await resp.text()
async def _fetch_external_markdown(
session: aiohttp.ClientSession,
url: str,
timeout: int,
jina_token_env: str = "JINA_API_KEY",
) -> str:
"""获取外链正文 markdown:先走 Jina Reader,失败回退到直接 GET + html_to_markdown。"""
try:
return await _fetch_via_jina(session, url, timeout, jina_token_env=jina_token_env)
except Exception:
html = await _fetch_url_html(session, url, timeout)
return html_to_markdown(html, base_url=url)
def _collect_comments_tree(
root: Dict,
top_comments: int,
top_l2_per_l1: int,
comment_max_chars: int,
comments_total_chars: int,
) -> List[Dict]:
"""从 Algolia 根节点提取评论树,返回 [{l1, replies}]。
规则:
- L1 上限 `top_comments`,每个 L1 下取前 `top_l2_per_l1` 条 L2 作为 replies
- 每条 text 过 html_to_markdown,单条截断到 `comment_max_chars`
- 累计字符达 `comments_total_chars` 时立即停止(防离群 story 撑爆 prompt)
- 跳过空 text;空 replies 仍保留 `replies: []`,schema 一致
"""
out: List[Dict] = []
consumed = 0
l1_children = (root.get("children") or [])[:top_comments]
for l1 in l1_children:
if consumed >= comments_total_chars:
break
l1_raw = (l1 or {}).get("text") or ""
if not l1_raw:
continue
l1_md = html_to_markdown(l1_raw)[:comment_max_chars]
consumed += len(l1_md)
replies: List[str] = []
for l2 in ((l1 or {}).get("children") or [])[:top_l2_per_l1]:
if consumed >= comments_total_chars:
break
l2_raw = (l2 or {}).get("text") or ""
if not l2_raw:
continue
l2_md = html_to_markdown(l2_raw)[:comment_max_chars]
consumed += len(l2_md)
replies.append(l2_md)
out.append({"l1": l1_md, "replies": replies})
return out
async def enrich_story(
session: aiohttp.ClientSession,
story: Dict,
top_comments: int,
top_l2_per_l1: int,
comment_max_chars: int,
comments_total_chars: int,
link_content_max_chars: int,
algolia_base: str,
timeout: int,
jina_token_env: str = "JINA_API_KEY",
) -> Dict:
"""对单 story enrich。任一子任务失败 → 对应字段留空,不抛。"""
item_id = story["id"]
is_internal = _is_internal_hn_url(story["url"])
tasks = [
_fetch_algolia_item(
session, item_id, algolia_base=algolia_base, timeout=timeout
)
]
if not is_internal:
tasks.append(
_fetch_external_markdown(
session, story["url"], timeout=timeout, jina_token_env=jina_token_env
)
)
results = await asyncio.gather(*tasks, return_exceptions=True)
algolia_result = results[0]
external_markdown_result = results[1] if not is_internal else None
comments_tree: List[Dict] = []
post_text = ""
if not isinstance(algolia_result, Exception) and algolia_result:
post_text = algolia_result.get("text") or ""
comments_tree = _collect_comments_tree(
algolia_result,
top_comments=top_comments,
top_l2_per_l1=top_l2_per_l1,
comment_max_chars=comment_max_chars,
comments_total_chars=comments_total_chars,
)
link_content = ""
if is_internal:
if post_text:
link_content = html_to_markdown(post_text)[:link_content_max_chars]
else:
if (
not isinstance(external_markdown_result, Exception)
and external_markdown_result
):
link_content = external_markdown_result[:link_content_max_chars]
return {
**story,
"link_content": link_content,
"top_comments": comments_tree,
}
async def enrich_stories(
stories: List[Dict],
top_comments: int,
top_l2_per_l1: int,
comment_max_chars: int,
comments_total_chars: int,
link_content_max_chars: int,
algolia_base: str = "https://hn.algolia.com/api/v1",
timeout: int = 10,
jina_token_env: str = "JINA_API_KEY",
) -> Tuple[List[Dict], List[str]]:
"""并发 enrich 多个 stories。"""
errors: List[str] = []
if os.environ.get(jina_token_env):
print(f"🔑 HN: 已配置 {jina_token_env},Jina Reader 鉴权调用")
else:
print(f"⚠️ HN: 未配置 {jina_token_env},Jina Reader 匿名调用 (额度受限)")
async with aiohttp.ClientSession(headers={"User-Agent": USER_AGENT}) as session:
results = await asyncio.gather(
*[
enrich_story(
session,
s,
top_comments=top_comments,
top_l2_per_l1=top_l2_per_l1,
comment_max_chars=comment_max_chars,
comments_total_chars=comments_total_chars,
link_content_max_chars=link_content_max_chars,
algolia_base=algolia_base,
timeout=timeout,
jina_token_env=jina_token_env,
)
for s in stories
],
return_exceptions=True,
)
enriched: List[Dict] = []
for r, src in zip(results, stories):
if isinstance(r, Exception):
errors.append(f"enrich story {src['id']} 失败: {r}")
else:
enriched.append(r)
return enriched, errors
ai-daily-main/src/sections/hackernews/section.py
+90
View File
@@ -0,0 +1,90 @@
"""HN 板块入口。流程:首页 → 轻 LLM 选 K → enrich → 最终 LLM 行文"""
from datetime import datetime
from typing import Dict, Optional, Tuple
from src.sections.hackernews.frontpage_scraper import (
fetch_frontpage,
parse_frontpage_html,
)
from src.sections.hackernews.item_enricher import enrich_stories
async def run_hackernews_section(
config: Dict, now: Optional[datetime] = None
) -> Tuple[str, Optional[str]]:
cfg = config.get("sections", {}).get("hackernews", {})
if not cfg.get("enabled", False):
return "", None
# 延迟 import (Task 16 提供这两个函数)
from src.llm import select_ai_related_hn, summarize_hackernews
timeout = cfg.get("request_timeout", 10)
select_k = cfg.get("select_k", 1)
top_comments = cfg.get("top_comments", 30)
top_l2_per_l1 = cfg.get("top_l2_per_l1", 3)
comment_max_chars = cfg.get("comment_max_chars", 2000)
comments_total_chars = cfg.get("comments_total_chars", 60000)
link_content_max_chars = cfg.get("link_content_max_chars", 50000)
algolia_base = cfg.get("algolia_base", "https://hn.algolia.com/api/v1")
jina_token_env = cfg.get("jinaTokenName", "JINA_API_KEY")
# 1. 抓首页
print("📥 HN: 抓取首页...")
try:
html = await fetch_frontpage(timeout=timeout)
except Exception as e:
return "", f"HN 首页抓取失败: {e}"
front = parse_frontpage_html(html)
print(f"📋 HN: 解析 {len(front)} 条 frontpage stories")
if not front:
return "", None
# 2. 轻 LLM 初筛
print(f"🤖 HN: 轻 LLM 初筛 (k={select_k})...")
selected_ids, select_err = await select_ai_related_hn(front, k=select_k, config=config["llm"])
if select_err:
return "", f"select_ai_related_hn: {select_err}"
if not selected_ids:
print("ℹ️ HN: 初筛未挑出 AI 相关内容,跳过")
return "", None
selected = [s for s in front if s["id"] in set(selected_ids)]
if not selected:
return "", None
print(
f"🎯 HN: 初筛选出 {len(selected)} 个: "
f"{', '.join(s['id'] for s in selected)}"
)
# 3. enrich
print(f"🌐 HN: enrich {len(selected)} 个 story (Algolia + 外链)...")
enriched, enrich_errors = await enrich_stories(
selected,
top_comments=top_comments,
top_l2_per_l1=top_l2_per_l1,
comment_max_chars=comment_max_chars,
comments_total_chars=comments_total_chars,
link_content_max_chars=link_content_max_chars,
algolia_base=algolia_base,
timeout=timeout,
jina_token_env=jina_token_env,
)
for e in enrich_errors:
print(f"⚠️ HN enrich: {e}")
print(
f"✅ HN: enrich 成功 {len(enriched)} / 失败 {len(enrich_errors)} / "
f"输入 {len(selected)}"
)
if not enriched:
return "", None
# 4. LLM 总结
print(f"🤖 HN: summarize {len(enriched)} 个 enriched story...")
md, err = await summarize_hackernews(enriched, config["llm"])
if err:
return "", f"summarize_hackernews: {err}"
print(f"✅ HN: 板块输出 {len(md or '')} chars")
return md or "", None
ai-daily-main/src/sections/insights/__init__.py
View File
ai-daily-main/src/sections/insights/section.py
+37
View File
@@ -0,0 +1,37 @@
"""Insights 板块:基于 RSS/GH/HN 三段成品做跨板块小结"""
from datetime import datetime
from typing import Dict, Optional, Tuple
EMPTY_MARKER = "(本次无内容)"
async def run_insights_section(
rss_md: str,
gh_md: str,
hn_md: str,
config: Dict,
now: Optional[datetime] = None,
) -> Tuple[str, Optional[Dict], Optional[str]]:
cfg = config.get("sections", {}).get("insights", {})
if not cfg.get("enabled", False):
return "", None, None
from src.llm import generate_trend_insights, parse_insights_with_metadata
sections = {
"rss": rss_md or EMPTY_MARKER,
"github": gh_md or EMPTY_MARKER,
"hackernews": hn_md or EMPTY_MARKER,
}
md, err = await generate_trend_insights(sections, config["llm"])
if err:
return "", None, f"generate_trend_insights: {err}"
if now is None:
now = datetime.now()
date_str = now.strftime("%Y-%m-%d")
insights_md, metadata = parse_insights_with_metadata(md or "", date_str)
return insights_md, metadata, None
ai-daily-main/src/sections/rss/__init__.py
+3
View File
@@ -0,0 +1,3 @@
from src.sections.rss.section import run_rss_section
__all__ = ["run_rss_section"]
ai-daily-main/src/sections/rss/section.py
+62
View File
@@ -0,0 +1,62 @@
"""RSS 板块:沿用现有 collect_entries_for_push + compose_digest 流程
迁移自 src/main.py::run_push_job 中 RSS digest 部分,行为保持一致。
"""
from datetime import datetime
from typing import Dict, Optional, Tuple
from src.llm import compose_digest, parse_digest_with_metadata
from src.storage import (
extract_push_time,
get_last_push_file,
load_recent_push_content,
)
async def run_rss_section(
config: Dict, now: Optional[datetime] = None
) -> Tuple[str, Optional[Dict], Optional[str]]:
"""生成 RSS digest markdown 段(不含 sentinel)。
返回:
(markdown_body, metadata, error)
- 无新内容时返回 ("", None, None)
- compose_digest 失败时返回 ("", None, error_message)
- metadata 字段:title / lead / highlights / profile=default / date
早报场景下调用方可丢弃 metadata(由 insights 段覆盖)
"""
# 延迟 import 避免循环:Task 20-21 后 main.py 会反向 import run_rss_section
from src.main import collect_entries_for_push
last_push_file = get_last_push_file()
last_push_time = extract_push_time(last_push_file) if last_push_file else None
min_score = config["filter"]["min_score"]
context_days = config["filter"]["context_days"]
to_push, context = collect_entries_for_push(
last_push_time=last_push_time,
context_days=context_days,
min_score=min_score,
)
if not to_push:
print("ℹ️ RSS: 无新消息")
return "", None, None
push_context_days = config["filter"].get("push_context_days", 5)
recent = load_recent_push_content(push_context_days)
try:
raw = await compose_digest(
to_push, context, config["llm"], recent_push_context=recent
)
except Exception as e:
msg = f"compose_digest 失败: {e}"
print(f"⚠️ RSS: {msg}")
return "", None, msg
date_str = (now or datetime.now()).strftime("%Y-%m-%d")
body, metadata = parse_digest_with_metadata(raw or "", date_str)
return body, metadata, None
ai-daily-main/src/storage.py
+546
View File
@@ -0,0 +1,546 @@
"""数据存储模块 - JSON文件读写"""
import json
import re
from datetime import date, datetime, timedelta
from pathlib import Path
from typing import Dict, List, Optional
from src.config import get_timezone
from src.markdown_utils import dump_frontmatter, parse_frontmatter
def get_fetch_file(d: date = None, data_dir: str = "news-data") -> str:
"""获取fetch文件路径 (使用配置时区)"""
if d is None:
d = datetime.now(get_timezone()).date()
return f"{data_dir}/fetch-{d.isoformat()}.json"
def get_push_file(push_time: datetime = None, data_dir: str = "news-data") -> str:
"""生成push文件路径"""
if push_time is None:
push_time = datetime.now(get_timezone())
time_str = push_time.strftime("%Y-%m-%d-%H-%M-%S")
return f"{data_dir}/push-{time_str}.md"
def get_notify_file(d: date = None, data_dir: str = "news-data") -> str:
"""获取notify文件路径 (使用配置时区)"""
if d is None:
d = datetime.now(get_timezone()).date()
return f"{data_dir}/notify-{d.isoformat()}.md"
def save_notify_file(
filepath: str,
content: str,
metadata: Dict = None,
):
"""保存即时推送文件（Markdown格式），同一天的内容追加到同一文件"""
path = Path(filepath)
path.parent.mkdir(parents=True, exist_ok=True)
notify_time = datetime.now(get_timezone()).isoformat()
if metadata:
frontmatter_dict = metadata.copy()
else:
frontmatter_dict = {"pushTime": notify_time}
frontmatter = dump_frontmatter(frontmatter_dict)
new_content = f"---\n{frontmatter}---\n\n{content}\n\n------\n"
with open(path, "a", encoding="utf-8") as f:
f.write(new_content)
_SECTION_RE_CACHE: Dict[str, re.Pattern] = {}
def _section_re(section: str) -> re.Pattern:
"""获取/缓存 sentinel 正则。section 名做转义,允许字母数字下划线"""
if section not in _SECTION_RE_CACHE:
s = re.escape(section)
pattern = (
rf"<!--\s*SECTION:{s}\s*BEGIN\s*-->(.*?)<!--\s*SECTION:{s}\s*END\s*-->"
)
_SECTION_RE_CACHE[section] = re.compile(pattern, flags=re.DOTALL)
return _SECTION_RE_CACHE[section]
def extract_section(push_md: str, section: str) -> str:
"""从 push 文件内容中切出 <!-- SECTION:{section} BEGIN/END --> 之间的 markdown。
向后兼容:
- 新文件(带 sentinel): 返回 sentinel 边界内的原文(不去边界空行)
- 老文件(无 sentinel) 且 section == 'rss': 返回整个 push_md
- 老文件(无 sentinel) 且 section != 'rss': 返回空字符串
- sentinel 残缺(只有 BEGIN 没有 END): 返回空字符串
"""
match = _section_re(section).search(push_md)
if match:
return match.group(1)
# 老文件兜底:rss 段视为整个 body
has_any_sentinel = "<!-- SECTION:" in push_md
if section == "rss" and not has_any_sentinel:
return push_md
return ""
def load_recent_notify_content(
context_days: int = 3, data_dir: str = "news-data"
) -> str:
"""加载最近 context_days 天 notify 文件正文（去除 frontmatter，仅供 LLM 查重）
notify 文件由多个推送块用 `------` 分隔，每块带各自 frontmatter；这里逐块剥离
frontmatter 后用 `------` 重新拼接，保留事件全文。
"""
data_path = Path(data_dir)
if not data_path.exists():
return ""
tz = get_timezone()
today = datetime.now(tz).date()
blocks: List[str] = []
loaded_files = []
for i in range(context_days):
d = today - timedelta(days=i)
notify_file = data_path / f"notify-{d.isoformat()}.md"
if not notify_file.exists() or notify_file.stat().st_size == 0:
continue
try:
with open(notify_file, "r", encoding="utf-8") as f:
content = f.read()
except Exception:
continue
for block in content.split("------"):
if not block.strip():
continue
_, body = parse_frontmatter(block)
if body:
blocks.append(body)
loaded_files.append(notify_file.name)
if loaded_files:
print(
f" 📂 已加载 {len(loaded_files)} 个 notify 文件: {', '.join(loaded_files)}"
)
return "\n\n------\n\n".join(blocks)
def load_recent_push_content(
context_days: int = 3, data_dir: str = "news-data", section: str = "rss"
) -> str:
"""加载最近 context_days 天 push 文件中指定 section 的正文(去除 frontmatter,仅供 LLM 查重)。
Args:
section: sentinel 段名,默认 "rss"。老文件(无 sentinel) 且 section == "rss"
时会兜底返回整个 body(由 extract_section 处理),其它 section 在
老文件上返回空。
"""
data_path = Path(data_dir)
if not data_path.exists():
return ""
tz = get_timezone()
today = datetime.now(tz).date()
bodies: List[str] = []
loaded_files = []
for i in range(context_days):
d = today - timedelta(days=i)
pattern = f"push-{d.isoformat()}-*.md"
for push_file in sorted(data_path.glob(pattern)):
if push_file.stat().st_size == 0:
continue
try:
with open(push_file, "r", encoding="utf-8") as f:
content = f.read()
except Exception:
continue
section_md = extract_section(content, section)
if not section_md:
continue
# 老文件兜底路径会把整篇文件还回来,此时仍需剥离 frontmatter;
# 新文件 sentinel 内不含 frontmatter,parse_frontmatter 会原样返回。
_, body = parse_frontmatter(section_md)
body = body or section_md
body = body.strip()
if body:
bodies.append(body)
loaded_files.append(push_file.name)
if loaded_files:
print(
f" 📂 已加载 {len(loaded_files)} 个 push 文件 (section={section}): "
f"{', '.join(loaded_files)}"
)
return "\n\n------\n\n".join(bodies)
def get_last_push_file(data_dir: str = "news-data") -> Optional[str]:
"""从news-data目录找到最新的push文件"""
data_path = Path(data_dir)
if not data_path.exists():
return None
push_files = sorted(data_path.glob("push-*.md"))
return str(push_files[-1]) if push_files else None
def extract_push_time(filepath: str) -> Optional[datetime]:
"""从push文件名提取时间"""
try:
basename = Path(filepath).name
time_str = basename.replace("push-", "").replace(".md", "")
dt = datetime.strptime(time_str, "%Y-%m-%d-%H-%M-%S")
return dt.replace(tzinfo=get_timezone())
except (ValueError, AttributeError):
return None
def read_entries(filepath: str) -> List[Dict]:
"""读取fetch文件，返回entries列表"""
path = Path(filepath)
if not path.exists():
return []
with open(path, "r", encoding="utf-8") as f:
data = json.load(f)
return data.get("entries", [])
def read_fetch_data(filepath: str) -> Dict:
"""读取完整的fetch文件数据（包含meta和entries）"""
path = Path(filepath)
if not path.exists():
return {"meta": {}, "entries": []}
# 检查文件是否为空
if path.stat().st_size == 0:
return {"meta": {}, "entries": []}
with open(path, "r", encoding="utf-8") as f:
return json.load(f)
def save_fetch_file(filepath: str, meta: Dict, entries: List[Dict]):
"""保存fetch文件（JSON格式）"""
path = Path(filepath)
path.parent.mkdir(parents=True, exist_ok=True)
data = {"meta": meta, "entries": entries}
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=2)
def append_entries(filepath: str, new_entries: List[Dict], meta: Dict = None):
"""追加条目到fetch文件"""
path = Path(filepath)
# 读取现有数据
if path.exists():
data = read_fetch_data(filepath)
else:
data = {"meta": meta or {}, "entries": []}
# 更新meta（如果提供了）
if meta:
data["meta"].update(meta)
# 去重：基于link字段
existing_links = {e.get("link") for e in data["entries"]}
for entry in new_entries:
if entry.get("link") not in existing_links:
data["entries"].append(entry)
existing_links.add(entry.get("link"))
# 保存
save_fetch_file(filepath, data["meta"], data["entries"])
return len(new_entries)
def format_entry(entry: Dict) -> str:
"""格式化单条条目为Markdown字符串"""
tags = entry.get("tags", [])
tags_str = json.dumps(tags, ensure_ascii=False) if tags else "[]"
score = entry.get("score", "")
summary = entry.get("summary", "")
return f"""## {entry["title"]}
---
source: {entry["source"]}
link: {entry["link"]}
published: {entry["published"]}
fetched_at: {entry["fetched_at"]}
tags: {tags_str}
score: {score}
summary: {summary}
---
{entry["content"]}
------
"""
def json_to_md(data: Dict) -> str:
"""
将JSON格式的fetch数据转换为Markdown格式，便于阅读
Args:
data: {"meta": {...}, "entries": [...]}
Returns:
Markdown格式的字符串
"""
meta = data.get("meta", {})
entries = data.get("entries", [])
lines = []
# 文件头部YAML frontmatter
if meta.get("date"):
lines.append("---")
lines.append(f'date: "{meta["date"]}"')
lines.append("---")
lines.append("")
# 条目
for entry in entries:
lines.append(format_entry(entry))
return "\n".join(lines)
def convert_fetch_json_to_md(json_filepath: str, md_filepath: str = None) -> str:
"""
将fetch JSON文件转换为Markdown文件
Args:
json_filepath: JSON文件路径
md_filepath: 输出MD文件路径，默认为同名.md
Returns:
生成的Markdown内容
"""
data = read_fetch_data(json_filepath)
md_content = json_to_md(data)
if md_filepath:
path = Path(md_filepath)
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
f.write(md_content)
return md_content
def save_push_file(
filepath: str,
content: str,
source_count: int,
total_entries: int,
profile: str = "default",
metadata: Dict = None,
):
"""保存推送文件（Markdown格式）
Args:
profile: "morning" | "default" ← 早报或常规;写入 frontmatter,便于按 profile 分析
metadata: 元信息（可选），如果提供则使用 metadata，否则使用默认格式
"""
path = Path(filepath)
path.parent.mkdir(parents=True, exist_ok=True)
if metadata:
# 使用提供的 metadata
frontmatter_dict = metadata.copy()
# 添加推送时间和统计信息
frontmatter_dict["pushDate"] = datetime.now(get_timezone()).isoformat()
frontmatter_dict["sourceCount"] = source_count
frontmatter_dict["totalEntries"] = total_entries
else:
# 降级：使用默认格式
push_time = datetime.now(get_timezone())
frontmatter_dict = {
"profile": profile,
"pushDate": push_time.isoformat(),
"sourceCount": source_count,
"totalEntries": total_entries,
}
frontmatter = dump_frontmatter(frontmatter_dict)
full_content = f"---\n{frontmatter}---\n\n{content}"
with open(path, "w", encoding="utf-8") as f:
f.write(full_content)
def load_existing_links(filepath: str, threshold: int = 150) -> set:
"""加载文件中已有的链接（用于去重）
如果当天时间已超过 threshold 分钟，则只需加载当天文件；
否则需要同时加载当天和昨天的文件（用于处理跨天边界情况）。
Args:
filepath: 当天的 fetch 文件路径
threshold: 阈值（分钟），超过此时间只加载当天文件
"""
tz = get_timezone()
now = datetime.now(tz)
current_minutes = now.hour * 60 + now.minute
need_yesterday = current_minutes < threshold
if not need_yesterday:
if not filepath or not Path(filepath).exists():
return set()
entries = read_entries(filepath)
return {e.get("link") for e in entries if e.get("link")}
all_links = set()
if filepath and Path(filepath).exists():
all_links.update(
{e.get("link") for e in read_entries(filepath) if e.get("link")}
)
yesterday = (now - timedelta(days=1)).date()
yesterday_file = get_fetch_file(yesterday)
if Path(yesterday_file).exists():
all_links.update(
{e.get("link") for e in read_entries(yesterday_file) if e.get("link")}
)
return all_links
def cleanup_old_files(days: int = 7, data_dir: str = "news-data"):
"""清理超过days天的旧文件"""
data_path = Path(data_dir)
if not data_path.exists():
return
cutoff = datetime.now() - timedelta(days=days)
deleted_count = 0
for pattern in ["fetch-*.json", "fetch-*.md", "push-*.md", "notify-*.md"]:
for file in data_path.glob(pattern):
try:
date_str = (
file.name.replace("fetch-", "")
.replace("push-", "")
.replace("notify-", "")
.replace(".json", "")
.replace(".md", "")
)
date_parts = date_str.split("-")
if len(date_parts) >= 3:
file_date = date(
int(date_parts[0]), int(date_parts[1]), int(date_parts[2])
)
if file_date < cutoff.date():
file.unlink()
deleted_count += 1
print(f" 🗑️ 删除旧文件: {file.name}")
except (ValueError, OSError):
continue
# trending-history.json: 剪枝过期条目,保留文件本身
trending_path = data_path / "trending-history.json"
if trending_path.exists() and trending_path.stat().st_size > 0:
try:
history = load_trending_history(str(trending_path))
before = len(history.repos)
history.cleanup(today=datetime.now().date(), keep_days=days)
after = len(history.repos)
if after < before:
history.save()
print(f" ✂️ trending-history 剪枝: {before} → {after} 条")
except Exception as e:
print(f" ⚠️ trending-history 剪枝失败: {e}")
if deleted_count > 0:
print(f" ✅ 清理完成: 删除了 {deleted_count} 个旧文件")
class TrendingHistory:
"""GitHub trending 已查阅 repo 索引。
repos 字段:url → last_seen_date (ISO YYYY-MM-DD)。
每次早报 cleanup 一次,touch 完所有今日 URL 后 save。
"""
def __init__(self, path: str, repos: Dict[str, str]):
self._path = path
self.repos: Dict[str, str] = dict(repos)
def __contains__(self, url: str) -> bool:
return url in self.repos
def touch(self, url: str, today: date) -> None:
self.repos[url] = today.isoformat()
def cleanup(self, today: date, keep_days: int) -> None:
cutoff = today - timedelta(days=keep_days)
self.repos = {
url: d
for url, d in self.repos.items()
if _parse_iso_date_safe(d) is not None and _parse_iso_date_safe(d) >= cutoff
}
def save(self) -> None:
path = Path(self._path)
path.parent.mkdir(parents=True, exist_ok=True)
payload = {
"repos": self.repos,
"updated_at": datetime.now(get_timezone()).isoformat(),
}
with open(path, "w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
def _parse_iso_date_safe(s: str) -> Optional[date]:
try:
return date.fromisoformat(s)
except (ValueError, TypeError):
return None
def load_trending_history(path: str) -> TrendingHistory:
"""读取 trending-history.json;不存在返回空实例。"""
p = Path(path)
if not p.exists() or p.stat().st_size == 0:
return TrendingHistory(path, {})
try:
with open(p, "r", encoding="utf-8") as f:
data = json.load(f)
return TrendingHistory(path, data.get("repos", {}))
except (json.JSONDecodeError, OSError):
print(f"⚠️ trending-history 读取失败,使用空索引: {path}")
return TrendingHistory(path, {})
_SECTION_ORDER = ("rss", "github", "hackernews", "insights")
def assemble_with_sentinels(sections: Dict[str, str]) -> str:
"""按固定顺序拼装四段 markdown,每段包 sentinel;空段整段省略。"""
parts: List[str] = []
for key in _SECTION_ORDER:
body = (sections.get(key) or "").strip()
if not body:
continue
parts.append(
f"<!-- SECTION:{key} BEGIN -->\n{body}\n<!-- SECTION:{key} END -->"
)
return "\n\n".join(parts)
ai-daily-main/systemd/dnews-fetch.service.tmpl
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=Daily News - Fetch Job
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User={{USER}}
Group={{GROUP}}
WorkingDirectory={{PROJECT_DIR}}
EnvironmentFile={{PROJECT_DIR}}/.env
ExecStart={{UV_BIN}} run --project {{PROJECT_DIR}} python -m src.main fetch
LogNamespace=dnews
TimeoutStartSec=20min
Nice=10
ai-daily-main/systemd/dnews-fetch.timer.tmpl
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Daily News - Fetch Timer
[Timer]
OnActiveSec=1s
OnUnitActiveSec={{FETCH_INTERVAL_MIN}}min
AccuracySec=1s
Unit=dnews-fetch.service
[Install]
WantedBy=timers.target
ai-daily-main/systemd/dnews-push.service.tmpl
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=Daily News - Push Job
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User={{USER}}
Group={{GROUP}}
WorkingDirectory={{PROJECT_DIR}}
EnvironmentFile={{PROJECT_DIR}}/.env
ExecStart={{UV_BIN}} run --project {{PROJECT_DIR}} python -m src.main push
LogNamespace=dnews
TimeoutStartSec=10min
Nice=10
ai-daily-main/systemd/dnews-push.timer.tmpl
+10
View File
@@ -0,0 +1,10 @@
[Unit]
Description=Daily News - Push Timer
[Timer]
{{PUSH_ONCALENDAR_LINES}}
AccuracySec=1s
Unit=dnews-push.service
[Install]
WantedBy=timers.target
ai-daily-main/systemd/journald-dnews.conf.tmpl
+2
View File
@@ -0,0 +1,2 @@
[Journal]
MaxRetentionSec={{LOG_RETENTION_DAYS}}day
ai-daily-main/tests/__init__.py
View File
ai-daily-main/tests/fetch_hackernews.py
+45
View File
@@ -0,0 +1,45 @@
"""手动跑一次 HN 首页 + Algolia enrich,验证选择器与 API"""
import asyncio
import json
import os
import sys
from dotenv import load_dotenv
load_dotenv()
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from src.sections.hackernews.frontpage_scraper import (
fetch_frontpage,
parse_frontpage_html,
)
from src.sections.hackernews.item_enricher import enrich_stories
async def main():
print("📥 抓取 HN 首页...")
html = await fetch_frontpage(timeout=15)
stories = parse_frontpage_html(html)
print(f"📋 解析出 {len(stories)} 条")
for s in stories[:5]:
print(f" - [{s['points']} pts · {s['comments']} comments] {s['title']} ({s['site']})")
print("\n🔍 enrich 前 1 个外链类故事...")
target = next((s for s in stories if not s["url"].startswith("https://news.ycombinator.com/")), stories[0])
enriched, errors = await enrich_stories(
[target],
top_comments=5,
top_l2_per_l1=3,
comment_max_chars=2000,
comments_total_chars=60000,
link_content_max_chars=1500,
timeout=15,
)
for e in errors:
print(f" ⚠️ {e}")
print(json.dumps(enriched, ensure_ascii=False, indent=2))
if __name__ == "__main__":
asyncio.run(main())
ai-daily-main/tests/fetch_news.py
+201
View File
@@ -0,0 +1,201 @@
#!/usr/bin/env python3
"""获取新闻脚本 - 模块化测试第一步：获取RSS并存储"""
import argparse
import asyncio
import os
import sys
from collections import defaultdict
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import urlparse
# 添加项目根目录到路径
sys.path.insert(0, str(Path(__file__).parent.parent))
from dotenv import load_dotenv
load_dotenv()
from src.config import get_timezone, load_config, merge_sources
from src.fetcher import fetch_all_feeds
from src.processor import html_to_markdown
from src.storage import append_entries, get_fetch_file, load_existing_links
def parse_args():
"""解析命令行参数"""
parser = argparse.ArgumentParser(description="RSS新闻获取测试")
parser.add_argument(
"--hours", "-H", type=int, default=1, help="获取过去多少小时的新闻 (默认: 1)"
)
parser.add_argument("--minutes", "-m", type=int, help="获取过去多少分钟的新闻")
parser.add_argument(
"--output-dir",
"-o",
type=str,
default="tests/news-data",
help="输出目录 (默认: tests/news-data)",
)
parser.add_argument(
"--max-per-domain",
type=int,
default=30,
help="同一域名最大保留源数量 (默认: 30)",
)
return parser.parse_args()
def get_cutoff_time(args) -> datetime:
"""根据参数计算截止时间 (返回 UTC 时间)"""
now = datetime.now(timezone.utc)
if args.minutes:
return now - timedelta(minutes=args.minutes)
return now - timedelta(hours=args.hours)
def limit_sources_by_domain(sources: list, max_per_domain: int = 30) -> list:
"""限制同一域名的源数量"""
domain_sources = defaultdict(list)
for source in sources:
url = source.get("xmlUrl", "")
try:
domain = urlparse(url).netloc.lower()
# 移除 www 前缀
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = "unknown"
domain_sources[domain].append(source)
limited_sources = []
domain_counts = {}
for domain, src_list in domain_sources.items():
kept = src_list[:max_per_domain]
limited_sources.extend(kept)
domain_counts[domain] = {"total": len(src_list), "kept": len(kept)}
return limited_sources, domain_counts
async def fetch_news():
"""主函数：获取RSS新闻并存储"""
args = parse_args()
tz = get_timezone()
print("=" * 60)
print("📰 RSS新闻获取测试 (Step 1)")
print("=" * 60)
# 1. 加载配置
print("\n📋 加载配置...")
config = load_config()
all_sources = merge_sources(config["sources"])
print(f" OPML 解析完成: {len(all_sources)} 个源")
# 2. 域名限制
print(f"\n🔍 域名限制 (每域名最多 {args.max_per_domain} 个)...")
sources, domain_stats = limit_sources_by_domain(all_sources, args.max_per_domain)
# 打印域名统计
total_domains = len(domain_stats)
limited_domains = sum(
1 for d in domain_stats.values() if d["total"] > args.max_per_domain
)
print(f" 域名总数: {total_domains}")
print(f" 受限域名: {limited_domains}")
print(f" 最终保留: {len(sources)} 个源")
# 显示受限域名详情
for domain, stats in sorted(domain_stats.items(), key=lambda x: -x[1]["total"])[:5]:
if stats["total"] > args.max_per_domain:
print(f" - {domain}: {stats['total']} → {stats['kept']}")
# 3. 计算时间窗口
cutoff = get_cutoff_time(args)
print(f"\n⏰ 时间窗口")
print(f" UTC: {cutoff.strftime('%Y-%m-%d %H:%M')}")
print(
f" Local: {(datetime.now(tz) - (datetime.now(timezone.utc) - cutoff)).strftime('%Y-%m-%d %H:%M')}"
)
# 4. 获取RSS数据
print(f"\n📡 开始获取...")
max_workers = config.get("fetch", {}).get("max_workers", 10)
timeout = config.get("fetch", {}).get("timeout", 5)
entries = await fetch_all_feeds(
sources, cutoff, max_workers=max_workers, timeout=timeout
)
# 5. 统计结果
print(f"\n📊 获取统计")
print(f" 读取源数: {len(all_sources)}")
print(f" 保留源数: {len(sources)}")
print(f" 获取条目: {len(entries)}")
if not entries:
print("\n⚠️ 没有获取到新消息")
return 0
# 6. 转换HTML到Markdown
print("\n📝 处理内容...")
for entry in entries:
entry["content"] = html_to_markdown(
entry.get("content", ""), entry.get("link", "")
)
# 7. 保存到文件
print(f"\n💾 保存到文件...")
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
# 使用今天的日期作为文件名（JSON格式）
today = datetime.now(tz).date()
fetch_file = output_dir / f"fetch-{today.isoformat()}.json"
# 加载已有链接去重
existing_links = (
load_existing_links(str(fetch_file)) if fetch_file.exists() else set()
)
new_entries = [
e for e in entries if e.get("link") and e["link"] not in existing_links
]
# 添加时间戳并格式化
for entry in entries:
entry["fetched_at"] = datetime.now(tz).isoformat()
if isinstance(entry.get("published"), datetime):
entry["published"] = entry["published"].astimezone(tz).isoformat()
# 使用新的 append_entries 批量保存
meta = {"date": today.isoformat()}
append_entries(str(fetch_file), entries, meta)
print(f" 文件: {fetch_file}")
print(f" 保存: {len(entries)} 条")
print(f" 新增: {len(new_entries)} 条")
print(f" 重复: {len(entries) - len(new_entries)} 条")
print("\n" + "=" * 60)
print("✅ Step 1 完成: RSS获取并存储")
print("=" * 60)
return len(entries)
if __name__ == "__main__":
try:
count = asyncio.run(fetch_news())
sys.exit(0 if count > 0 else 1)
except KeyboardInterrupt:
print("\n\n👋 已取消")
sys.exit(130)
except Exception as e:
print(f"\n❌ 错误: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
ai-daily-main/tests/fetch_nitter.py
+46
View File
@@ -0,0 +1,46 @@
"""手动跑一次 xcancel/nitter 抓取，验证白名单 UA + requests 路径
用法：python tests/fetch_nitter.py
"""
import asyncio
import os
import sys
from datetime import datetime, timedelta, timezone
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from src.fetcher import fetch_all_feeds
FEEDS = [
{
"title": "Alibaba_Qwen",
"xmlUrl": "https://rss.xcancel.com/Alibaba_Qwen/rss",
},
{
"title": "trq212",
"xmlUrl": "https://rss.xcancel.com/trq212/rss",
},
]
async def main():
cutoff = datetime.now(timezone.utc) - timedelta(days=7)
print(f"📥 抓取 {len(FEEDS)} 个 nitter 源（cutoff={cutoff.isoformat()}）...")
entries = await fetch_all_feeds(FEEDS, cutoff)
print(f"📋 拿到 {len(entries)} 条")
for e in entries[:10]:
published = e["published"].isoformat() if e["published"] else "?"
print(f" - [{published}] {e['source']} {e['title']}")
print(f" {e['link']}")
print(f" {e['content']}\n\n")
if not entries:
print("⚠️ 一条都没拿到 —— 检查 UA / TLS 路径是否生效")
sys.exit(1)
if __name__ == "__main__":
asyncio.run(main())
ai-daily-main/tests/fetch_trending.py
+44
View File
@@ -0,0 +1,44 @@
"""手动跑一次 GH trending 抓取 + deep-dive,验证选择器与 API 接入"""
import asyncio
import json
import os
import sys
from dotenv import load_dotenv
load_dotenv()
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from src.config import load_config
from src.sections.github.repo_enricher import enrich_repos
from src.sections.github.trending_scraper import (
fetch_trending_page,
parse_trending_html,
)
async def main():
config = load_config()
print("📥 抓取 GitHub Trending...")
html = await fetch_trending_page(timeout=15)
repos = parse_trending_html(html)
print(f"📋 解析出 {len(repos)} 个 repo")
for r in repos[:5]:
print(f" - {r['full_name']} ⭐{r['stars_today']}/{r['stars_total']} | {r['description'][:80]}")
cfg = config["sections"]["github_trending"]
print(f"\n🔍 enrich 前 {min(3, len(repos))} 个...")
enriched, errors = await enrich_repos(
repos[:3],
token_env=cfg.get("tokenName", "GITHUB_TOKEN"),
readme_max_chars=cfg.get("readme_max_chars", 3000),
timeout=15,
)
for e in errors:
print(f" ⚠️ {e}")
print(json.dumps(enriched, ensure_ascii=False, indent=2))
if __name__ == "__main__":
asyncio.run(main())
ai-daily-main/tests/manual_trigger_fetch_loop_llm_error.py
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""手工触发 LLM 错误告警。
用法：
source ../.venv/bin/activate
python tests/manual_trigger_fetch_loop_llm_error.py --scenario score
python tests/manual_trigger_fetch_loop_llm_error.py --scenario immediate
python tests/manual_trigger_fetch_loop_llm_error.py --scenario digest
python tests/manual_trigger_fetch_loop_llm_error.py --scenario all
场景说明：
- score: 走 fetch_loop -> run_fetch_job -> score_batch，触发评分错误通知
- immediate: 走 run_fetch_job -> generate_immediate_push，触发即时推送生成错误通知
- digest: 走 run_push_job -> compose_digest，触发汇总生成错误通知
- all: 依次触发以上三类真实通知
"""
import argparse
import asyncio
import sys
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.config import load_config
from src.main import fetch_loop, run_fetch_job, run_push_job
MOCK_FETCH_ENTRIES = [
{
"title": "Manual Mock News 1",
"link": "https://example.com/mock-1",
"published": "2026-03-08T20:00:00+08:00",
"source": "Manual Mock Source",
"content": "<p>Mock content 1</p>",
"tags": [],
"score": 0,
"summary": "",
},
{
"title": "Manual Mock News 2",
"link": "https://example.com/mock-2",
"published": "2026-03-08T20:05:00+08:00",
"source": "Manual Mock Source",
"content": "<p>Mock content 2</p>",
"tags": [],
"score": 0,
"summary": "",
},
]
MOCK_HOT_SCORED_ENTRIES = [
{
"title": "Manual Hot News",
"link": "https://example.com/hot-1",
"published": "2026-03-08T21:00:00+08:00",
"source": "Manual Mock Source",
"content": "mock content",
"tags": ["AI"],
"score": 95,
"summary": "manual hot summary",
}
]
MOCK_DIGEST_ENTRIES = [
{
"title": "Manual Digest News",
"link": "https://example.com/digest-1",
"published": "2026-03-08T19:00:00+08:00",
"fetched_at": "2026-03-08T21:00:00+08:00",
"source": "Manual Mock Source",
"content": "mock digest content",
"tags": ["AI"],
"score": 88,
"summary": "manual digest summary",
}
]
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="手工触发 LLM 错误告警")
parser.add_argument(
"--scenario",
choices=["score", "immediate", "digest", "all"],
default="score",
help="要触发的错误场景，默认 score",
)
return parser.parse_args()
async def _trigger_score_error(config):
mock_sources = [
{
"title": "Manual Mock Feed",
"xmlUrl": "https://example.com/rss.xml",
"category": "AI",
}
]
tmp_dir = Path("test/news-data")
tmp_dir.mkdir(parents=True, exist_ok=True)
fetch_file = tmp_dir / "manual-fetch-loop-error.json"
async def fake_fetch_all_feeds(*args, **kwargs):
return MOCK_FETCH_ENTRIES
async def fake_sleep(seconds: float):
if seconds <= 0:
return
raise asyncio.CancelledError()
with (
patch("src.main.merge_sources", return_value=mock_sources),
patch("src.main.fetch_all_feeds", side_effect=fake_fetch_all_feeds),
patch("src.main.load_existing_links", return_value=set()),
patch("src.main.get_fetch_file", return_value=str(fetch_file)),
patch(
"src.llm.call_llm",
side_effect=RuntimeError("manual mock llm scoring error"),
),
patch("src.main.asyncio.sleep", side_effect=fake_sleep),
):
await fetch_loop(config)
async def _trigger_immediate_push_error(config):
tmp_dir = Path("test/news-data")
tmp_dir.mkdir(parents=True, exist_ok=True)
fetch_file = tmp_dir / "manual-immediate-push-error.json"
async def fake_fetch_all_feeds(*args, **kwargs):
return [dict(entry) for entry in MOCK_HOT_SCORED_ENTRIES]
with (
patch(
"src.main.merge_sources",
return_value=[
{
"title": "Manual",
"xmlUrl": "https://example.com/rss.xml",
"category": "AI",
}
],
),
patch("src.main.fetch_all_feeds", side_effect=fake_fetch_all_feeds),
patch("src.main.load_existing_links", return_value=set()),
patch("src.main.get_fetch_file", return_value=str(fetch_file)),
patch(
"src.main.score_batch",
return_value=([dict(entry) for entry in MOCK_HOT_SCORED_ENTRIES], []),
),
patch(
"src.main.generate_immediate_push",
return_value=("", "manual mock immediate push error"),
),
):
await run_fetch_job(config)
async def _trigger_digest_error(config):
with (
patch(
"src.main.collect_entries_for_push",
return_value=([dict(entry) for entry in MOCK_DIGEST_ENTRIES], []),
),
patch("src.main.get_last_push_file", return_value=None),
patch(
"src.main.compose_digest",
side_effect=RuntimeError("manual mock compose digest error"),
),
):
await run_push_job(config)
async def _run_selected_scenarios(scenario: str) -> None:
config = load_config()
if scenario in {"score", "all"}:
print("\n🚨 触发 score_batch 错误通知")
await _trigger_score_error(config)
if scenario in {"immediate", "all"}:
print("\n🚨 触发 generate_immediate_push 错误通知")
await _trigger_immediate_push_error(config)
if scenario in {"digest", "all"}:
print("\n🚨 触发 compose_digest 错误通知")
await _trigger_digest_error(config)
def main() -> int:
args = parse_args()
print("🚨 即将触发真实的 LLM 错误提醒")
print(f" - 场景: {args.scenario}")
print(" - 推送: 使用当前 config.json 和环境变量中的真实渠道")
try:
asyncio.run(_run_selected_scenarios(args.scenario))
except KeyboardInterrupt:
print("\n已取消")
return 130
print("✅ 脚本执行完成；如果推送配置正确，你应该已经收到对应错误提醒。")
return 0
if __name__ == "__main__":
raise SystemExit(main())
ai-daily-main/tests/preview_recent_context.py
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env python3
"""预览 LLM 查重上下文 —— 直接打印 load_recent_notify_content / load_recent_push_content
返回的内容,便于调试历史去重时实际喂给 LLM 的素材。
Usage:
# 默认:读取 config.json 的 filter.context_days,打印 notify + push 两段
python tests/preview_recent_context.py
# 指定回溯天数
python tests/preview_recent_context.py --days 5
# 只看一种
python tests/preview_recent_context.py --type notify
python tests/preview_recent_context.py --type push
# 自定义数据目录(例如 tests/news-data 里的样本数据)
python tests/preview_recent_context.py --data-dir tests/news-data
"""
import argparse
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.config import load_config
from src.storage import load_recent_notify_content, load_recent_push_content
def _resolve_days(cli_days: int | None) -> int:
if cli_days is not None:
return cli_days
try:
config = load_config()
return int(config.get("filter", {}).get("context_days", 3))
except Exception:
return 3
def _print_section(label: str, content: str) -> None:
sep = "=" * 60
print(f"\n{sep}\n{label}\n{sep}")
if not content:
print("(空)")
return
print(content)
print(
f"\n--- {label} 字符数: {len(content)} | 块数(按 `------` 切分): "
f"{len([b for b in content.split('------') if b.strip()])} ---"
)
def main() -> int:
parser = argparse.ArgumentParser(
description="预览近 N 天 notify / push 文件正文 (剥离 frontmatter 后)"
)
parser.add_argument(
"--days",
type=int,
default=None,
help="回溯天数,默认读取 config.filter.context_days(兜底 3)",
)
parser.add_argument(
"--type",
choices=["notify", "push", "both"],
default="both",
help="预览哪种历史,默认 both",
)
parser.add_argument(
"--data-dir",
default="news-data",
help="数据目录,默认 news-data",
)
parser.add_argument(
"--section",
default="rss",
help="push 文件取哪个 sentinel 段,默认 rss(可选 github / hackernews / insights)",
)
args = parser.parse_args()
days = _resolve_days(args.days)
print(
f"📅 回溯天数: {days} | 📂 数据目录: {args.data_dir} | "
f"类型: {args.type} | push.section: {args.section}"
)
if args.type in ("notify", "both"):
notify_md = load_recent_notify_content(days, args.data_dir)
_print_section("近期即时推送 (notify)", notify_md)
if args.type in ("push", "both"):
push_md = load_recent_push_content(days, args.data_dir, section=args.section)
_print_section(f"近期汇总推送 (push.{args.section})", push_md)
return 0
if __name__ == "__main__":
sys.exit(main())
ai-daily-main/tests/probe_hn_comments.py
+140
View File
@@ -0,0 +1,140 @@
"""Probe HN: 抓取首页前 10 条,统计每个 story 评论树的深度/数量/字符数。
用于判断 enrich 策略:是否需要 L2 回复、压缩、过滤短评等。
字符数说明:
- "raw" = Algolia 返回的 text 字段(HTML)
- "md" = html_to_markdown 后的字符数(LLM 实际看到的)
"""
import asyncio
import json
import os
import sys
from typing import Dict, List, Tuple
import aiohttp
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from src.processor import html_to_markdown
from src.sections.hackernews.frontpage_scraper import (
fetch_frontpage,
parse_frontpage_html,
)
ALGOLIA = "https://hn.algolia.com/api/v1/items"
TIMEOUT = 30
async def fetch_item(session: aiohttp.ClientSession, item_id: str) -> Dict:
async with session.get(
f"{ALGOLIA}/{item_id}",
timeout=aiohttp.ClientTimeout(total=TIMEOUT),
) as resp:
if resp.status != 200:
raise RuntimeError(f"{item_id} -> {resp.status}")
return await resp.json()
def walk_tree(node: Dict, depth: int = 0) -> List[Tuple[int, str]]:
"""递归遍历评论树,返回 [(depth, text_html), ...]。depth=0 是 story 自身,1 是顶层评论。"""
result: List[Tuple[int, str]] = []
text = (node or {}).get("text") or ""
if depth > 0 and text:
result.append((depth, text))
for child in (node or {}).get("children") or []:
result.extend(walk_tree(child, depth + 1))
return result
def stats(samples: List[str]) -> Dict:
if not samples:
return {"count": 0, "raw_chars": 0, "md_chars": 0, "avg_md": 0, "max_md": 0}
md_lens = [len(html_to_markdown(s)) for s in samples]
raw_lens = [len(s) for s in samples]
return {
"count": len(samples),
"raw_chars": sum(raw_lens),
"md_chars": sum(md_lens),
"avg_md": round(sum(md_lens) / len(md_lens)),
"max_md": max(md_lens),
}
def by_depth(nodes: List[Tuple[int, str]], d: int) -> List[str]:
return [t for depth, t in nodes if depth == d]
async def probe_one(
session: aiohttp.ClientSession, story: Dict
) -> Dict:
item_id = story["id"]
data = await fetch_item(session, item_id)
flat = walk_tree(data)
l1 = by_depth(flat, 1)
l2 = by_depth(flat, 2)
l3plus = [t for d, t in flat if d >= 3]
all_comments = [t for _, t in flat]
return {
"id": item_id,
"title": story["title"][:60],
"page_comments": story["comments"],
"L1": stats(l1),
"L2": stats(l2),
"L3+": stats(l3plus),
"ALL": stats(all_comments),
}
def print_row(label: str, s: Dict):
print(
f" {label:5s} count={s['count']:4d} md_total={s['md_chars']:7d} "
f"avg={s['avg_md']:5d} max={s['max_md']:6d}"
)
async def main():
print("📥 抓取 HN 首页...")
html = await fetch_frontpage(timeout=15)
stories = parse_frontpage_html(html)[:10]
print(f"📋 取前 10 条 (实际 {len(stories)} 条)\n")
async with aiohttp.ClientSession() as session:
results = await asyncio.gather(
*[probe_one(session, s) for s in stories],
return_exceptions=True,
)
summary = {"L1": [], "L2": [], "L3+": [], "ALL": []}
for r in results:
if isinstance(r, Exception):
print(f"❌ {r}\n")
continue
print(
f"#{r['id']} page_comments={r['page_comments']}\n {r['title']}"
)
print_row("L1", r["L1"])
print_row("L2", r["L2"])
print_row("L3+", r["L3+"])
print_row("ALL", r["ALL"])
print()
for key in summary:
summary[key].append(r[key])
print("=" * 70)
print("📊 10 条 story 汇总(平均/总和)\n")
for key in ["L1", "L2", "L3+", "ALL"]:
rows = summary[key]
total_count = sum(r["count"] for r in rows)
total_md = sum(r["md_chars"] for r in rows)
avg_count_per_story = round(total_count / len(rows), 1) if rows else 0
avg_md_per_story = round(total_md / len(rows)) if rows else 0
print(
f" {key:5s} 总数={total_count:4d} 总md字符={total_md:8d} "
f"平均每story count={avg_count_per_story:5.1f} md_chars={avg_md_per_story:6d}"
)
if __name__ == "__main__":
asyncio.run(main())
ai-daily-main/tests/push_news.py
+168
View File
@@ -0,0 +1,168 @@
#!/usr/bin/env python3
"""推送功能测试 (Step 2)
测试内容：
1. 读取已获取的新闻数据
2. 测试推送到所有已启用的平台
3. 验证推送格式和内容
使用方法:
python tests/push_news.py # 默认从 fetch 数据读取发送 (--fake)
python tests/push_news.py --fake # 从 fetch 数据读取发送
python tests/push_news.py --real # 从 news-data/push-*.md 最新文件发送
"""
import argparse
import asyncio
import sys
from datetime import date
from pathlib import Path
from typing import Optional
sys.path.insert(0, str(Path(__file__).parent.parent))
from dotenv import load_dotenv
load_dotenv()
from src.config import load_config
from src.push import send_to_platforms
from src.storage import (
convert_fetch_json_to_md,
get_fetch_file,
get_last_push_file,
read_entries,
)
def parse_args():
"""解析命令行参数"""
parser = argparse.ArgumentParser(description="推送功能测试")
parser.add_argument(
"--fake", action="store_true", help="从 fetch 数据读取发送（默认）"
)
parser.add_argument(
"--real", action="store_true", help="从 news-data/push-*.md 最新文件发送"
)
return parser.parse_args()
def load_push_content() -> Optional[str]:
"""从 news-data 获取最新的 push 文件内容（去掉 frontmatter）"""
last_file = get_last_push_file(data_dir="tests/news-data")
if not last_file:
print(" ❌ 未找到 push 文件")
return None
print(f" 📂 读取文件: {last_file}")
with open(last_file, "r", encoding="utf-8") as f:
content = f.read()
# 去掉 frontmatter
if content.startswith("---"):
parts = content.split("---", 2)
if len(parts) >= 3:
content = parts[2].strip()
return content
async def test_push(mode: str = "fake"):
"""测试推送功能"""
print("=" * 60)
print("📤 推送功能测试 (Step 2)")
print(f" 模式: {'real' if mode == 'real' else 'fake (fetch数据)'}")
print("=" * 60)
# 加载配置
print("\n📋 加载配置...")
config = load_config()
content: Optional[str] = None
title: Optional[str] = None
if mode == "real":
# 从 push 文件读取
content = load_push_content()
if content:
# 提取标题
first_line = content.split("\n")[0] if content else ""
if first_line.startswith("# "):
title = first_line.replace("# ", "").strip()
print(f" 📝 内容长度: {len(content)} 字符")
print(f" 📝 标题: {title}")
else:
# 从 fetch 数据读取（默认）
print("\n📖 读取新闻数据...")
fetch_file = get_fetch_file(date.today(), data_dir="tests/news-data")
entries = read_entries(fetch_file)
print(f" 文件: {fetch_file}")
print(f" 获取到 {len(entries)} 条新闻")
# 同时生成 Markdown 版本便于阅读
md_file = str(fetch_file).replace(".json", ".md")
convert_fetch_json_to_md(fetch_file, md_file)
print(f" Markdown版本: {md_file}")
if not entries:
print("\n⚠️ 没有新闻数据，请先运行 fetch_news.py")
return
# 构建测试消息
print("\n📝 构建推送消息...")
content = build_test_message(entries[:5])
print(f" 消息长度: {len(content)} 字符")
if not content:
print("\n⚠️ 没有内容可推送")
return
# 推送到所有已启用平台
print("\n📤 推送消息...")
try:
await send_to_platforms(
content, config["push"], "📰 AI Daily 每日精选 | {Test:YYYY-MM-DD}"
)
except Exception as e:
print(f" ❌ 推送失败: {e}")
raise
print("\n" + "=" * 60)
print("✅ Step 2 完成: 推送测试")
print("=" * 60)
def build_test_message(entries: list) -> str:
"""构建测试消息"""
lines = [
"📰 **新闻推送测试**",
"",
f"共获取 {len(entries)} 条新闻：",
"",
]
for i, entry in enumerate(entries, 1):
title = entry.get("title", "无标题")
source = entry.get("source", "未知来源")
link = entry.get("link", "")
published = entry.get("published", "")
content = entry.get("content", "")[:100]
lines.append(f"**{i}. {title}**")
lines.append(f" 📰 来源: {source}")
if published:
lines.append(f" ⏰ 时间: {published}")
if link:
lines.append(f" 🔗 链接: {link}")
if content:
lines.append(f" 📝 内容: {content}...")
lines.append("")
return "\n".join(lines)
if __name__ == "__main__":
args = parse_args()
mode = "real" if args.real else "fake"
asyncio.run(test_push(mode))
ai-daily-main/tests/pytest/__init__.py
+1
View File
@@ -0,0 +1 @@
# pytest tests for daily-news project
ai-daily-main/tests/pytest/conftest.py
+154
View File
@@ -0,0 +1,154 @@
"""pytest fixtures for daily-news project"""
import json
import pytest
import sys
from datetime import datetime, timezone, timedelta
from pathlib import Path
from unittest.mock import MagicMock
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
@pytest.fixture
def temp_dir(tmp_path):
"""临时目录fixture"""
return tmp_path
@pytest.fixture
def sample_config():
"""示例配置"""
return {
"sources": {
"base_opml": "resources/rss.opml",
"add": [],
"block": [],
"block_domains": ["*.substack.com"],
},
"filter": {
"min_score": 60,
"hot_threshold": 90,
"context_days": 2,
"keep_days": 7,
},
"schedule": {
"fetch_interval_minutes": 30,
"push_cron": ["0 8 * * *", "0 17 * * *"],
"timezone_hours": 8,
},
"llm": {
"provider": "openai",
"model": "gpt-4o-mini",
"baseUrl": "https://api.openai.com/v1",
"apiKeyName": "OPENAI_API_KEY",
"max_prompt_chars": 10000,
"max_concurrent_batches": 3,
},
"push": {
"discord": {
"enabled": True,
"webhook_url": "https://discord.com/api/webhooks/test/abc",
},
"feishu": {"enabled": False, "apiKeyName": "FEISHU_WEBHOOK_URL"},
},
}
@pytest.fixture
def sample_opml(temp_dir):
"""示例OPML文件"""
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
<outline title="Feed1" xmlUrl="http://feed1.com/rss" type="rss" category="tech"/>
<outline title="Feed2" xmlUrl="http://feed2.com/rss" type="rss" category="ai"/>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
return str(opml_file)
@pytest.fixture
def sample_entry():
"""示例新闻条目"""
return {
"title": "Test Article Title",
"link": "https://example.com/article",
"published": datetime.now(timezone.utc).isoformat(),
"fetched_at": datetime.now(timezone.utc).isoformat(),
"source": "Test Source",
"content": "<p>Test content</p>",
"summary": "Test summary",
"tags": ["AI", "Tech"],
"score": 85,
}
@pytest.fixture
def sample_entries():
"""示例新闻条目列表"""
now = datetime.now(timezone.utc)
return [
{
"title": "Article 1",
"link": "https://example.com/1",
"published": now.isoformat(),
"fetched_at": now.isoformat(),
"source": "Source1",
"content": "Content 1",
"summary": "Summary 1",
"tags": ["AI"],
"score": 85,
},
{
"title": "Article 2",
"link": "https://example.com/2",
"published": (now - timedelta(hours=1)).isoformat(),
"fetched_at": now.isoformat(),
"source": "Source2",
"content": "Content 2",
"summary": "Summary 2",
"tags": ["Tech"],
"score": 70,
},
{
"title": "Article 3",
"link": "https://example.com/3",
"published": (now - timedelta(hours=2)).isoformat(),
"fetched_at": now.isoformat(),
"source": "Source3",
"content": "Content 3",
"summary": "Summary 3",
"tags": ["News"],
"score": 55,
},
]
@pytest.fixture
def sample_fetch_json(temp_dir, sample_entries):
"""示例fetch JSON文件"""
data = {
"meta": {"date": datetime.now().date().isoformat()},
"entries": sample_entries,
}
json_file = temp_dir / "fetch-test.json"
json_file.write_text(json.dumps(data, ensure_ascii=False, indent=2))
return str(json_file)
@pytest.fixture
def mock_httpx_session():
"""Mock httpx/aiohttp session"""
mock_response = MagicMock()
mock_response.status = 200
mock_response.text = "<rss></rss>"
mock_session = MagicMock()
mock_session.__aenter__ = MagicMock(return_value=mock_session)
mock_session.__aexit__ = MagicMock(return_value=None)
mock_session.get = MagicMock(return_value=mock_response)
return mock_session
ai-daily-main/tests/pytest/fixtures/github_trending.html
+3771
View File
File diff suppressed because one or more lines are too long
ai-daily-main/tests/pytest/fixtures/hn_frontpage.html
+3
View File
File diff suppressed because one or more lines are too long
ai-daily-main/tests/pytest/test_config.py
+249
View File
@@ -0,0 +1,249 @@
"""配置模块测试"""
import json
import pytest
import sys
from datetime import datetime, timezone, timedelta
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from config import (
load_config,
parse_opml,
merge_sources,
get_timezone,
)
class TestLoadConfig:
"""测试配置加载"""
def test_load_valid_config(self, temp_dir):
config = {
"sources": {"base_opml": "test.opml", "add": [], "block": []},
"filter": {"min_score": 60},
"schedule": {"fetch_interval_minutes": 30},
"llm": {"provider": "groq", "model": "moonshotai/kimi-k2-instruct"},
"push": {"discord": {"enabled": False}},
}
config_file = temp_dir / "config.json"
config_file.write_text(json.dumps(config))
result = load_config(str(config_file))
assert result["filter"]["min_score"] == 60
assert result["llm"]["provider"] == "groq"
def test_load_missing_file(self):
with pytest.raises(FileNotFoundError):
load_config("nonexistent.json")
def test_load_invalid_json(self, temp_dir):
config_file = temp_dir / "config.json"
config_file.write_text("invalid json")
with pytest.raises(json.JSONDecodeError):
load_config(str(config_file))
def test_load_config_with_sources(self, temp_dir, sample_opml):
config = {
"sources": {
"base_opml": sample_opml,
"add": [
{
"title": "Add1",
"xmlUrl": "http://add1.com/rss",
"category": "test",
}
],
"block": [],
},
"filter": {"min_score": 60},
"schedule": {"fetch_interval_minutes": 30, "timezone_hours": 8},
"llm": {"provider": "test"},
"push": {"discord": {"enabled": False}},
}
config_file = temp_dir / "config.json"
config_file.write_text(json.dumps(config))
result = load_config(str(config_file))
assert len(result["sources"]["add"]) == 1
class TestParseOpml:
"""测试OPML解析"""
def test_parse_valid_opml(self, temp_dir):
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
<outline title="Feed1" xmlUrl="http://feed1.com/rss" type="rss"/>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
feeds = parse_opml(str(opml_file))
assert len(feeds) == 1
assert feeds[0]["title"] == "Feed1"
assert feeds[0]["xmlUrl"] == "http://feed1.com/rss"
def test_parse_missing_file(self):
feeds = parse_opml("nonexistent.opml")
assert feeds == []
def test_parse_opml_with_category(self, temp_dir):
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
<outline title="TechFeed" xmlUrl="http://tech.com/rss" type="rss" category="技术"/>
<outline title="AIFeed" xmlUrl="http://ai.com/rss" type="rss" category="AI"/>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
feeds = parse_opml(str(opml_file))
assert len(feeds) == 2
assert feeds[0]["category"] == "技术"
assert feeds[1]["category"] == "AI"
def test_parse_opml_empty_body(self, temp_dir):
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
feeds = parse_opml(str(opml_file))
assert feeds == []
class TestMergeSources:
"""测试源合并"""
def test_merge_base_and_add(self, sample_opml):
config = {
"base_opml": sample_opml,
"add": [
{"title": "Feed3", "xmlUrl": "http://feed3.com/rss", "category": "test"}
],
"block": [],
}
sources = merge_sources(config)
assert len(sources) == 3
def test_block_by_xmlUrl(self, sample_opml):
config = {
"base_opml": sample_opml,
"add": [],
"block": [{"title": "Block1", "xmlUrl": "http://feed1.com/rss"}],
}
sources = merge_sources(config)
assert all(s["xmlUrl"] != "http://feed1.com/rss" for s in sources)
assert len(sources) == 1
def test_deduplicate_by_xmlUrl(self, sample_opml):
config = {
"base_opml": sample_opml,
"add": [
{
"title": "Duplicate",
"xmlUrl": "http://feed1.com/rss",
"category": "test",
}
],
"block": [],
}
sources = merge_sources(config)
urls = [s["xmlUrl"] for s in sources]
assert len(urls) == len(set(urls))
assert len(sources) == 2
def test_block_domains_wildcard(self, temp_dir):
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
<outline title="Substack" xmlUrl="https://tech.substack.com/rss" type="rss"/>
<outline title="Blog" xmlUrl="https://tech.blog/rss" type="rss"/>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
config = {
"base_opml": str(opml_file),
"add": [],
"block": [],
"block_domains": ["*.substack.com"],
}
sources = merge_sources(config)
assert len(sources) == 1
assert sources[0]["xmlUrl"] == "https://tech.blog/rss"
def test_block_domains_exact(self, temp_dir):
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
<outline title="YouTube" xmlUrl="https://youtube.com/feed" type="rss"/>
<outline title="Blog" xmlUrl="https://tech.blog/rss" type="rss"/>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
config = {
"base_opml": str(opml_file),
"add": [],
"block": [],
"block_domains": ["youtube.com"],
}
sources = merge_sources(config)
assert len(sources) == 1
assert sources[0]["xmlUrl"] == "https://tech.blog/rss"
def test_block_domains_subdomain(self, temp_dir):
opml_content = """<?xml version="1.0"?>
<opml version="2.0">
<body>
<outline title="Substack1" xmlUrl="https://substack.com/feed" type="rss"/>
<outline title="Substack2" xmlUrl="https://ai.substack.com/feed" type="rss"/>
<outline title="Blog" xmlUrl="https://tech.blog/rss" type="rss"/>
</body>
</opml>"""
opml_file = temp_dir / "test.opml"
opml_file.write_text(opml_content)
config = {
"base_opml": str(opml_file),
"add": [],
"block": [],
"block_domains": ["*.substack.com"],
}
sources = merge_sources(config)
assert len(sources) == 1
assert sources[0]["xmlUrl"] == "https://tech.blog/rss"
class TestGetTimezone:
"""测试时区获取"""
def test_get_timezone_from_config(self, sample_config):
tz = get_timezone(sample_config)
assert isinstance(tz, timezone)
assert tz.utcoffset(datetime.now()).total_seconds() == 8 * 3600
def test_get_timezone_none_config(self):
tz = get_timezone(None)
assert isinstance(tz, timezone)
def test_get_timezone_no_timezone_hours(self):
config = {"schedule": {}}
tz = get_timezone(config)
assert isinstance(tz, timezone)
def test_get_timezone_custom_hours(self):
config = {"schedule": {"timezone_hours": -5}}
tz = get_timezone(config)
assert tz.utcoffset(datetime.now()).total_seconds() == -5 * 3600
ai-daily-main/tests/pytest/test_fetcher.py
+190
View File
@@ -0,0 +1,190 @@
"""RSS抓取模块测试"""
import pytest
import sys
from datetime import datetime, timezone, timedelta
from pathlib import Path
from unittest.mock import MagicMock, patch, AsyncMock
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from fetcher import (
parse_entry_time,
fetch_single_feed_async,
fetch_all_feeds,
DEFAULT_FEED_TIMEOUT,
)
class TestParseEntryTime:
"""测试时间解析"""
def test_parse_published_parsed(self):
entry = MagicMock()
entry.published_parsed = (2024, 1, 15, 10, 30, 0, 0, 0, 0)
result = parse_entry_time(entry)
assert result is not None
assert result.year == 2024
assert result.month == 1
assert result.day == 15
assert result.tzinfo == timezone.utc
def test_parse_updated_parsed(self):
entry = MagicMock()
entry.published_parsed = None
entry.updated_parsed = (2024, 1, 15, 10, 30, 0, 0, 0, 0)
result = parse_entry_time(entry)
assert result is not None
assert result.year == 2024
assert result.month == 1
assert result.day == 15
def test_parse_no_time(self):
entry = MagicMock()
entry.published_parsed = None
entry.updated_parsed = None
result = parse_entry_time(entry)
assert result is None
class TestFetchSingleFeedAsync:
"""测试单源抓取"""
@pytest.mark.asyncio
async def test_fetch_success(self, temp_dir):
rss_content = """<?xml version="1.0"?>
<rss version="2.0">
<channel>
<title>Test Feed</title>
<item>
<title>Article 1</title>
<link>https://example.com/1</link>
<pubDate>Mon, 15 Jan 2024 10:00:00 GMT</pubDate>
<description>Test description</description>
</item>
</channel>
</rss>"""
feed_info = {"title": "Test Feed", "xmlUrl": "http://test.com/rss"}
cutoff = datetime(2024, 1, 1, tzinfo=timezone.utc)
mock_response = MagicMock()
mock_response.status = 200
mock_response.text = AsyncMock(return_value=rss_content)
mock_session = MagicMock()
mock_session.__aenter__ = AsyncMock(return_value=mock_session)
mock_session.__aexit__ = AsyncMock(return_value=None)
mock_session.get = MagicMock(
return_value=AsyncMock(
__aenter__=AsyncMock(return_value=mock_response),
__aexit__=AsyncMock(return_value=None),
)
)
with patch("aiohttp.ClientSession", return_value=mock_session):
entries = await fetch_single_feed_async(
feed_info, cutoff, session=mock_session
)
assert len(entries) == 1
assert entries[0]["title"] == "Article 1"
assert entries[0]["link"] == "https://example.com/1"
assert entries[0]["source"] == "Test Feed"
@pytest.mark.asyncio
async def test_fetch_http_error(self):
feed_info = {"title": "Test Feed", "xmlUrl": "http://test.com/rss"}
cutoff = datetime(2024, 1, 1, tzinfo=timezone.utc)
mock_response = MagicMock()
mock_response.status = 404
mock_session = MagicMock()
mock_session.get = MagicMock(
return_value=AsyncMock(
__aenter__=AsyncMock(return_value=mock_response),
__aexit__=AsyncMock(return_value=None),
)
)
entries = await fetch_single_feed_async(feed_info, cutoff, session=mock_session)
assert entries == []
@pytest.mark.asyncio
async def test_fetch_timeout(self):
feed_info = {"title": "Test Feed", "xmlUrl": "http://test.com/rss"}
cutoff = datetime(2024, 1, 1, tzinfo=timezone.utc)
import aiohttp
mock_session = MagicMock()
mock_session.get = MagicMock(side_effect=aiohttp.ServerTimeoutError())
entries = await fetch_single_feed_async(feed_info, cutoff, session=mock_session)
assert entries == []
@pytest.mark.asyncio
async def test_fetch_cutoff_filter(self):
feed_info = {"title": "Test Feed", "xmlUrl": "http://test.com/rss"}
cutoff = datetime(2024, 1, 10, tzinfo=timezone.utc)
import feedparser
with patch(
"fetcher.fetch_single_feed_async", new_callable=AsyncMock
) as mock_fetch:
mock_fetch.return_value = [
{
"title": "New",
"link": "https://example.com/new",
"published": datetime(2024, 1, 15, tzinfo=timezone.utc),
}
]
result = await mock_fetch(feed_info, cutoff)
assert len(result) == 1
class TestFetchAllFeeds:
"""测试并发抓取"""
@pytest.mark.asyncio
async def test_concurrent_limit(self):
feeds = [
{"title": f"Feed{i}", "xmlUrl": f"http://feed{i}.com/rss"}
for i in range(20)
]
cutoff = datetime(2024, 1, 1, tzinfo=timezone.utc)
with patch(
"fetcher.fetch_single_feed_async", new_callable=AsyncMock
) as mock_fetch:
mock_fetch.return_value = []
await fetch_all_feeds(feeds, cutoff, max_workers=5)
assert mock_fetch.call_count == 20
@pytest.mark.asyncio
async def test_empty_feeds(self):
cutoff = datetime(2024, 1, 1, tzinfo=timezone.utc)
entries = await fetch_all_feeds([], cutoff)
assert entries == []
@pytest.mark.asyncio
async def test_default_timeout(self):
feed_info = {"title": "Test", "xmlUrl": "http://test.com"}
cutoff = datetime.now(timezone.utc)
with patch(
"fetcher.fetch_single_feed_async", new_callable=AsyncMock
) as mock_fetch:
mock_fetch.return_value = []
await mock_fetch(feed_info, cutoff, timeout=None)
mock_fetch.assert_called_once()
ai-daily-main/tests/pytest/test_llm.py
+330
View File
@@ -0,0 +1,330 @@
"""LLM模块测试"""
import json
import pytest
import sys
from datetime import datetime, timezone, timedelta
from pathlib import Path
from unittest.mock import patch, AsyncMock, MagicMock
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from llm import (
load_prompt,
_parse_llm_json_response,
_split_entries_for_batch,
_build_batch_prompt,
_merge_scores,
_score_single_batch,
call_llm,
check_llm_available,
generate_immediate_push,
score_batch,
)
class TestLoadPrompt:
"""测试提示词加载"""
def test_load_prompt_basic(self, temp_dir):
prompt_file = temp_dir / "test.txt"
prompt_file.write_text("Hello {name}!")
result = load_prompt(str(prompt_file), name="World")
assert result == "Hello World!"
def test_load_prompt_missing_file(self):
with pytest.raises(FileNotFoundError):
load_prompt("nonexistent.txt")
def test_load_prompt_with_braces(self, temp_dir):
prompt_file = temp_dir / "test.txt"
prompt_file.write_text("Hello {name}, curly braces: { }")
result = load_prompt(str(prompt_file), name="World")
assert result == "Hello World, curly braces: { }"
def test_load_prompt_multiple_vars(self, temp_dir):
prompt_file = temp_dir / "test.txt"
prompt_file.write_text("{greeting} {name}, you have {count} messages")
result = load_prompt(str(prompt_file), greeting="Hi", name="Alice", count=5)
assert result == "Hi Alice, you have 5 messages"
class TestParseLlmJsonResponse:
"""测试LLM响应解析"""
def test_parse_json_array(self):
response = '[{"link": "https://example.com", "score": 80}]'
result = _parse_llm_json_response(response)
assert len(result) == 1
assert result[0]["link"] == "https://example.com"
assert result[0]["score"] == 80
def test_parse_with_markdown_codeblock(self):
response = """```json
[{"link": "https://example.com", "score": 80}]
```"""
result = _parse_llm_json_response(response)
assert len(result) == 1
def test_parse_with_codeblock(self):
response = """```
[{"link": "https://example.com", "score": 80}]
```"""
result = _parse_llm_json_response(response)
assert len(result) == 1
def test_parse_invalid_response(self):
response = "This is not JSON at all"
with pytest.raises(ValueError):
_parse_llm_json_response(response)
class TestSplitEntriesForBatch:
"""测试条目分批"""
def test_split_empty(self):
result = _split_entries_for_batch([])
assert result == []
def test_split_single_batch(self):
entries = [
{
"link": f"https://example.com/{i}",
"title": f"Title{i}",
"content": "x" * 100,
}
for i in range(5)
]
result = _split_entries_for_batch(entries, max_prompt_chars=10000)
assert len(result) == 1
def test_split_multiple_batches(self):
entries = [
{
"link": f"https://example.com/{i}",
"title": f"Title{i}",
"content": "x" * 5000,
}
for i in range(10)
]
result = _split_entries_for_batch(entries, max_prompt_chars=10000)
assert len(result) > 1
class TestBuildBatchPrompt:
"""测试构建批量提示词"""
def test_build_batch_prompt_basic(self):
entries = [
{
"link": "https://example.com/1",
"title": "Title1",
"source": "Source1",
"published": "2024-01-15",
"content": "Content",
}
]
result = _build_batch_prompt(entries)
assert "Title1" in result
assert "https://example.com/1" in result
class TestMergeScores:
"""测试评分合并"""
def test_merge_scores_basic(self):
entries = [
{"link": "https://example.com/1", "title": "Title1"},
{"link": "https://example.com/2", "title": "Title2"},
]
scores = [
{
"link": "https://example.com/1",
"score": 85,
"tags": ["AI"],
"summary": "Summary1",
},
{
"link": "https://example.com/2",
"score": 70,
"tags": ["Tech"],
"summary": "Summary2",
},
]
result = _merge_scores(entries, scores)
assert result[0]["score"] == 85
assert result[0]["tags"] == ["AI"]
assert result[1]["score"] == 70
def test_merge_scores_partial(self):
entries = [
{"link": "https://example.com/1", "title": "Title1", "score": 50},
{"link": "https://example.com/2", "title": "Title2", "score": 60},
]
scores = [{"link": "https://example.com/1", "score": 85}]
result = _merge_scores(entries, scores)
assert result[0]["score"] == 85
assert result[1]["score"] == 60
class TestCallLlm:
"""测试LLM调用"""
@pytest.mark.asyncio
async def test_call_llm_success(self):
config = {
"model": "gpt-4",
"baseUrl": "https://api.openai.com/v1",
"apiKeyName": "OPENAI_API_KEY",
}
with patch("llm.call_llm", new_callable=AsyncMock) as mock_call:
mock_call.return_value = "Test response"
result = await mock_call("Test prompt", config)
assert result == "Test response"
@pytest.mark.asyncio
async def test_call_llm_missing_key(self):
config = {"model": "gpt-4", "apiKeyName": "MISSING_KEY"}
with pytest.raises(ValueError, match="未设置MISSING_KEY"):
await call_llm("Test prompt", config)
class TestLlmHealthCheck:
"""测试LLM可用性检查"""
@pytest.mark.asyncio
async def test_check_llm_available_success(self, sample_config):
with patch("llm.call_llm", new_callable=AsyncMock) as mock_call:
mock_call.return_value = "OK"
result = await check_llm_available(sample_config["llm"])
assert result == "OK"
@pytest.mark.asyncio
async def test_check_llm_available_empty_response(self, sample_config):
with patch("llm.call_llm", new_callable=AsyncMock) as mock_call:
mock_call.return_value = " "
with pytest.raises(RuntimeError, match="返回空响应"):
await check_llm_available(sample_config["llm"])
class TestImmediatePush:
"""测试即时推送生成"""
@pytest.mark.asyncio
async def test_generate_immediate_push_failure_returns_error(
self, sample_entries, sample_config
):
with patch("llm.load_prompt", return_value="prompt"), patch(
"llm.call_llm", new_callable=AsyncMock
) as mock_call:
mock_call.side_effect = RuntimeError("boom")
content, error = await generate_immediate_push(
sample_entries[:1], sample_config["llm"], recent_push_context=""
)
assert content == ""
assert error == "生成即时推送失败: boom"
class TestScoreBatch:
"""测试批量评分"""
@pytest.mark.asyncio
async def test_score_batch_empty(self, sample_config):
result, errors = await score_batch([], sample_config["llm"])
assert result == []
assert errors == []
@pytest.mark.asyncio
async def test_score_batch_single(self, sample_entries, sample_config):
entries = sample_entries[:1]
mock_scores = [
{
"link": entries[0]["link"],
"score": 85,
"tags": ["AI"],
"summary": "Test summary",
}
]
with patch("llm._score_single_batch", new_callable=AsyncMock) as mock_score:
mock_score.return_value = (mock_scores, [])
result, errors = await score_batch(entries, sample_config["llm"])
assert len(result) == 1
assert result[0]["score"] == 85
assert errors == []
@pytest.mark.asyncio
async def test_score_single_batch_failure_returns_empty_results(
self, sample_entries, sample_config
):
with patch("llm.call_llm", new_callable=AsyncMock) as mock_call:
mock_call.side_effect = RuntimeError("boom")
results, errors = await _score_single_batch(
sample_entries[:2], sample_config["llm"]
)
assert results == []
assert errors == ["批次1 评分失败: boom"]
@pytest.mark.asyncio
async def test_score_single_batch_reconcile_partial_results(
self, sample_entries, sample_config
):
entries = sample_entries[:2]
llm_results = [
{
"link": entries[0]["link"],
"score": 91,
"tags": ["AI"],
"summary": "Matched result",
}
]
with patch("llm.call_llm", new_callable=AsyncMock) as mock_call:
mock_call.return_value = json.dumps(llm_results, ensure_ascii=False)
results, errors = await _score_single_batch(entries, sample_config["llm"])
assert len(results) == 1
assert results[0]["score"] == 91
assert len(errors) == 1
assert "评分结果异常" in errors[0]
assert "输入2" in errors[0]
assert "返回1" in errors[0]
assert "匹配1" in errors[0]
@pytest.mark.asyncio
async def test_score_single_batch_keeps_full_results(self, sample_entries, sample_config):
entries = sample_entries[:3]
llm_results = [
{
"link": entry["link"],
"score": 88,
"tags": ["AI"],
"summary": f"Summary for {index}",
}
for index, entry in enumerate(entries, start=1)
]
with patch("llm.call_llm", new_callable=AsyncMock) as mock_call:
mock_call.return_value = json.dumps(llm_results, ensure_ascii=False)
results, errors = await _score_single_batch(entries, sample_config["llm"])
assert len(results) == 3
assert [result["link"] for result in results] == [entry["link"] for entry in entries]
assert errors == []
ai-daily-main/tests/pytest/test_llm_extra_sections.py
+101
View File
@@ -0,0 +1,101 @@
"""测试新增 LLM 函数 (summarize_github_trending 等)"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from llm import summarize_github_trending, select_ai_related_hn, summarize_hackernews
@pytest.mark.asyncio
async def test_summarize_github_trending_happy_path(tmp_path):
prompt_path = tmp_path / "section_github.md"
prompt_path.write_text("Repos: {repos_json}\nmax_items={max_items}", encoding="utf-8")
config = {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"section_github": str(prompt_path)},
"sections": {"github_trending": {"max_items": 3}},
}
enriched = [{"full_name": "o/r", "readme_excerpt": "rm"}]
with patch("llm.call_llm", new=AsyncMock(return_value="## md")):
md, err = await summarize_github_trending(enriched, config)
assert md == "## md"
assert err is None
@pytest.mark.asyncio
async def test_summarize_github_trending_llm_failure_returns_error(tmp_path):
prompt_path = tmp_path / "section_github.md"
prompt_path.write_text("x {repos_json} {max_items}", encoding="utf-8")
config = {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"section_github": str(prompt_path)},
"sections": {"github_trending": {"max_items": 3}},
}
with patch("llm.call_llm", new=AsyncMock(side_effect=RuntimeError("boom"))):
md, err = await summarize_github_trending([{"full_name": "o/r"}], config)
assert md == ""
assert "boom" in err
@pytest.mark.asyncio
async def test_select_ai_related_hn_parses_id_array(tmp_path):
prompt_path = tmp_path / "select.md"
prompt_path.write_text("k={k} candidates={candidates_json}", encoding="utf-8")
config = {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"section_hackernews_select": str(prompt_path)},
}
with patch("llm.call_llm", new=AsyncMock(return_value='["111", "222"]')):
ids, err = await select_ai_related_hn(
[{"id": "111"}, {"id": "222"}, {"id": "333"}], k=2, config=config
)
assert ids == ["111", "222"]
assert err is None
@pytest.mark.asyncio
async def test_select_ai_related_hn_empty_array(tmp_path):
prompt_path = tmp_path / "select.md"
prompt_path.write_text("{k}{candidates_json}", encoding="utf-8")
config = {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"section_hackernews_select": str(prompt_path)},
}
with patch("llm.call_llm", new=AsyncMock(return_value="[]")):
ids, err = await select_ai_related_hn([{"id": "1"}], k=1, config=config)
assert ids == []
assert err is None
@pytest.mark.asyncio
async def test_summarize_hackernews_happy(tmp_path):
prompt_path = tmp_path / "hn.md"
prompt_path.write_text("{stories_json}", encoding="utf-8")
config = {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"section_hackernews": str(prompt_path)},
}
with patch("llm.call_llm", new=AsyncMock(return_value="## HN summary")):
md, err = await summarize_hackernews(
[{"id": "1", "title": "t", "link_content": "x", "top_comments": []}], config
)
assert md == "## HN summary"
assert err is None
ai-daily-main/tests/pytest/test_main.py
+285
View File
@@ -0,0 +1,285 @@
"""主程序逻辑测试"""
import json
import pytest
import sys
from datetime import datetime, timezone, timedelta, date
from pathlib import Path
from unittest.mock import patch, MagicMock, AsyncMock
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from main import (
now_local,
parse_time_to_local,
calculate_push_times,
collect_entries_for_push,
main as run_main,
)
class TestNowLocal:
"""测试获取本地时间"""
def test_now_local_with_config(self, sample_config):
result = now_local(sample_config)
assert isinstance(result, datetime)
assert result.tzinfo is not None
def test_now_local_without_config(self):
result = now_local()
assert isinstance(result, datetime)
class TestParseTimeToLocal:
"""测试时间解析"""
def test_parse_iso_format(self, sample_config):
result = parse_time_to_local("2024-01-15T10:30:00+00:00", sample_config)
assert result is not None
assert result.year == 2024
assert result.month == 1
assert result.day == 15
def test_parse_with_z_suffix(self, sample_config):
result = parse_time_to_local("2024-01-15T10:30:00Z", sample_config)
assert result is not None
assert result.year == 2024
def test_parse_invalid_format(self, sample_config):
result = parse_time_to_local("not-a-date", sample_config)
assert result is None
def test_parse_none(self, sample_config):
result = parse_time_to_local("", sample_config)
assert result is None
class TestCalculatePushTimes:
"""测试推送时间计算"""
def test_calculate_single_cron(self, sample_config):
times = calculate_push_times(["30 8 * * *"], config=sample_config)
assert len(times) == 1
assert times[0].hour == 8
assert times[0].minute == 30
def test_calculate_multiple_crons(self, sample_config):
times = calculate_push_times(["0 8 * * *", "0 17 * * *"], config=sample_config)
assert len(times) == 2
hours = [t.hour for t in times]
assert 8 in hours
assert 17 in hours
def test_calculate_with_offset(self, sample_config):
times = calculate_push_times(["0 8 * * *"], offset_days=1, config=sample_config)
assert len(times) == 1
expected_date = (datetime.now(timezone.utc) + timedelta(days=1)).date()
assert times[0].date() == expected_date
def test_calculate_invalid_cron(self, sample_config):
times = calculate_push_times(["invalid cron"], config=sample_config)
assert times == []
class TestCollectEntriesForPush:
"""测试收集推送条目"""
def test_collect_no_files(self, temp_dir):
to_push, context = collect_entries_for_push(
last_push_time=None, context_days=2, min_score=60, data_dir=str(temp_dir)
)
assert to_push == []
assert context == []
def test_collect_with_low_score(self, temp_dir):
now = datetime.now(timezone.utc)
data = {
"meta": {"date": now.date().isoformat()},
"entries": [
{
"title": "Low Score",
"link": "https://example.com/1",
"score": 30,
"fetched_at": now.isoformat(),
}
],
}
fetch_file = temp_dir / f"fetch-{now.date().isoformat()}.json"
with open(fetch_file, "w") as f:
json.dump(data, f)
to_push, context = collect_entries_for_push(
last_push_time=None, context_days=2, min_score=60, data_dir=str(temp_dir)
)
assert len(to_push) == 0
def test_collect_with_high_score(self, temp_dir):
now = datetime.now(timezone.utc)
data = {
"meta": {"date": now.date().isoformat()},
"entries": [
{
"title": "High Score",
"link": "https://example.com/1",
"score": 85,
"fetched_at": now.isoformat(),
}
],
}
fetch_file = temp_dir / f"fetch-{now.date().isoformat()}.json"
with open(fetch_file, "w") as f:
json.dump(data, f)
to_push, context = collect_entries_for_push(
last_push_time=None, context_days=2, min_score=60, data_dir=str(temp_dir)
)
assert len(to_push) == 1
assert to_push[0]["score"] == 85
def test_collect_with_last_push_time(self, temp_dir):
now = datetime.now(timezone.utc)
last_push = now - timedelta(hours=2)
data = {
"meta": {"date": now.date().isoformat()},
"entries": [
{
"title": "New Entry",
"link": "https://example.com/1",
"score": 80,
"fetched_at": now.isoformat(),
},
{
"title": "Old Entry",
"link": "https://example.com/2",
"score": 80,
"fetched_at": last_push.isoformat(),
},
],
}
fetch_file = temp_dir / f"fetch-{now.date().isoformat()}.json"
with open(fetch_file, "w") as f:
json.dump(data, f)
to_push, context = collect_entries_for_push(
last_push_time=last_push,
context_days=2,
min_score=60,
data_dir=str(temp_dir),
)
assert len(to_push) == 1
assert to_push[0]["title"] == "New Entry"
def test_collect_context_limit(self, temp_dir):
now = datetime.now(timezone.utc)
entries = [
{
"title": f"Entry{i}",
"link": f"https://example.com/{i}",
"score": 50 + i,
"fetched_at": now.isoformat(),
}
for i in range(60)
]
data = {"meta": {"date": now.date().isoformat()}, "entries": entries}
fetch_file = temp_dir / f"fetch-{now.date().isoformat()}.json"
with open(fetch_file, "w") as f:
json.dump(data, f)
to_push, context = collect_entries_for_push(
last_push_time=None, context_days=2, min_score=60, data_dir=str(temp_dir)
)
assert len(context) <= 50
def test_collect_multi_day(self, temp_dir):
from src.config import get_timezone
tz = get_timezone()
today = datetime.now(tz)
yesterday = today - timedelta(days=1)
today_data = {
"meta": {"date": today.date().isoformat()},
"entries": [
{
"title": "Today Entry",
"link": "https://example.com/1",
"score": 80,
"fetched_at": today.isoformat(),
}
],
}
yesterday_data = {
"meta": {"date": yesterday.date().isoformat()},
"entries": [
{
"title": "Yesterday Entry",
"link": "https://example.com/2",
"score": 75,
"fetched_at": yesterday.isoformat(),
}
],
}
(temp_dir / f"fetch-{today.date().isoformat()}.json").write_text(
json.dumps(today_data)
)
(temp_dir / f"fetch-{yesterday.date().isoformat()}.json").write_text(
json.dumps(yesterday_data)
)
to_push, context = collect_entries_for_push(
last_push_time=None, context_days=2, min_score=60, data_dir=str(temp_dir)
)
assert len(to_push) >= 1
class TestMainStartup:
"""测试主程序启动流程"""
@pytest.mark.asyncio
async def test_main_checks_llm_before_starting_loops(self, sample_config):
with patch("main.load_config", return_value=sample_config), patch(
"main.check_llm_available", new_callable=AsyncMock
) as mock_check, patch(
"main.fetch_loop", new_callable=AsyncMock
) as mock_fetch_loop, patch(
"main.push_loop", new_callable=AsyncMock
) as mock_push_loop:
await run_main()
mock_check.assert_awaited_once_with(sample_config["llm"])
mock_fetch_loop.assert_awaited_once_with(sample_config)
mock_push_loop.assert_awaited_once_with(sample_config)
@pytest.mark.asyncio
async def test_main_exits_when_llm_health_check_fails(self, sample_config):
with patch("main.load_config", return_value=sample_config), patch(
"main.check_llm_available", new_callable=AsyncMock
) as mock_check, patch(
"main.fetch_loop", new_callable=AsyncMock
) as mock_fetch_loop, patch(
"main.push_loop", new_callable=AsyncMock
) as mock_push_loop:
mock_check.side_effect = RuntimeError("health failed")
await run_main()
mock_check.assert_awaited_once_with(sample_config["llm"])
mock_fetch_loop.assert_not_called()
mock_push_loop.assert_not_called()
ai-daily-main/tests/pytest/test_main_morning_push.py
+123
View File
@@ -0,0 +1,123 @@
"""测试早报四模块编排:gather + insights 串行 + sentinel 拼装 + 失败隔离"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.main import _run_morning_push
def _insights_meta() -> dict:
return {
"title": "📰 AI Daily 每日精选 | 2026-05-22",
"lead": "lead text",
"highlights": ["a"],
"profile": "morning",
"date": "2026-05-22",
"excerpt": "",
"seotitle": "",
"seodescription": "",
}
@pytest.mark.asyncio
async def test_assembles_all_four_sections(sample_config):
sample_config["filter"]["push_context_days"] = 5
sent = {}
async def fake_send(content, push_cfg, title=None, metadata=None):
sent["content"] = content
sent["metadata"] = metadata
saved = {}
def fake_save(
filepath,
content,
source_count,
total_entries,
profile="default",
metadata=None,
):
saved["profile"] = profile
saved["content"] = content
saved["metadata"] = metadata
with patch(
"src.main.run_rss_section", new=AsyncMock(return_value=("R", None, None))
), patch(
"src.main.run_github_section", new=AsyncMock(return_value=("G", None))
), patch(
"src.main.run_hackernews_section", new=AsyncMock(return_value=("H", None))
), patch(
"src.main.run_insights_section",
new=AsyncMock(return_value=("I", _insights_meta(), None)),
), patch(
"src.main.send_to_platforms", new=AsyncMock(side_effect=fake_send)
), patch(
"src.main.save_push_file", side_effect=fake_save
):
await _run_morning_push(sample_config)
assert "SECTION:rss" in sent["content"]
assert "SECTION:github" in sent["content"]
assert "SECTION:hackernews" in sent["content"]
assert "SECTION:insights" in sent["content"]
assert sent["metadata"]["profile"] == "morning"
assert saved["profile"] == "morning"
assert saved["metadata"]["lead"] == "lead text"
@pytest.mark.asyncio
async def test_rss_failure_raises_to_caller(sample_config):
sample_config["filter"]["push_context_days"] = 5
with patch(
"src.main.run_rss_section",
new=AsyncMock(return_value=("", None, "compose_digest 失败")),
), patch(
"src.main.run_github_section", new=AsyncMock(return_value=("G", None))
), patch(
"src.main.run_hackernews_section", new=AsyncMock(return_value=("H", None))
), patch(
"src.main.notify_llm_errors", new=AsyncMock()
):
with pytest.raises(RuntimeError):
await _run_morning_push(sample_config)
@pytest.mark.asyncio
async def test_section_failure_degrades_to_omission(sample_config):
sample_config["filter"]["push_context_days"] = 5
sent = {}
async def fake_send(content, push_cfg, title=None, metadata=None):
sent["content"] = content
with patch(
"src.main.run_rss_section", new=AsyncMock(return_value=("R", None, None))
), patch(
"src.main.run_github_section", new=AsyncMock(return_value=("", "gh down"))
), patch(
"src.main.run_hackernews_section", new=AsyncMock(return_value=("H", None))
), patch(
"src.main.run_insights_section",
new=AsyncMock(return_value=("I", _insights_meta(), None)),
), patch(
"src.main.notify_llm_errors", new=AsyncMock()
), patch(
"src.main.send_to_platforms", new=AsyncMock(side_effect=fake_send)
), patch(
"src.main.save_push_file"
):
await _run_morning_push(sample_config)
assert "SECTION:rss" in sent["content"]
assert "SECTION:github" not in sent["content"]
assert "SECTION:hackernews" in sent["content"]
ai-daily-main/tests/pytest/test_main_run_push_job.py
+43
View File
@@ -0,0 +1,43 @@
"""测试 run_push_job 的早报/默认路径分发"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.main import run_push_job
@pytest.mark.asyncio
async def test_default_path_when_not_morning(sample_config):
sample_config["schedule"]["push_cron"] = ["0 8 * * *", "0 17 * * *"]
sample_config["filter"]["push_context_days"] = 5
with patch("src.main.is_morning_push", return_value=False), patch(
"src.main._run_default_push", new=AsyncMock(return_value=None)
) as default_path, patch(
"src.main._run_morning_push", new=AsyncMock(return_value=None)
) as morning_path:
await run_push_job(sample_config)
default_path.assert_awaited_once()
morning_path.assert_not_awaited()
@pytest.mark.asyncio
async def test_morning_path_when_morning(sample_config):
sample_config["schedule"]["push_cron"] = ["0 8 * * *", "0 17 * * *"]
sample_config["filter"]["push_context_days"] = 5
with patch("src.main.is_morning_push", return_value=True), patch(
"src.main._run_default_push", new=AsyncMock(return_value=None)
) as default_path, patch(
"src.main._run_morning_push", new=AsyncMock(return_value=None)
) as morning_path:
await run_push_job(sample_config)
morning_path.assert_awaited_once()
default_path.assert_not_awaited()
ai-daily-main/tests/pytest/test_morning_detection.py
+60
View File
@@ -0,0 +1,60 @@
"""测试早报判定:push_cron 最近最早匹配"""
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.main import is_morning_push
TZ = timezone(timedelta(hours=8))
def _cfg(push_cron):
return {"schedule": {"push_cron": push_cron, "timezone_hours": 8}}
def test_returns_false_when_push_cron_empty():
assert is_morning_push(datetime(2026, 5, 17, 8, 0, tzinfo=TZ), {"schedule": {}}) is False
assert is_morning_push(datetime(2026, 5, 17, 8, 0, tzinfo=TZ), _cfg([])) is False
def test_single_cron_always_morning():
"""push_cron 只有一项时,任何触发(含手动 / 非整点)都视为早报"""
cfg = _cfg(["0 8 * * *"])
assert is_morning_push(datetime(2026, 5, 17, 8, 0, tzinfo=TZ), cfg) is True
assert is_morning_push(datetime(2026, 5, 17, 12, 30, tzinfo=TZ), cfg) is True
assert is_morning_push(datetime(2026, 5, 17, 23, 59, tzinfo=TZ), cfg) is True
def test_earliest_cron_match_is_morning():
"""触发时刻离最早 cron 最近 → 早报"""
cfg = _cfg(["0 8 * * *", "0 17 * * *"])
assert is_morning_push(datetime(2026, 5, 17, 8, 0, tzinfo=TZ), cfg) is True
assert is_morning_push(datetime(2026, 5, 17, 8, 30, tzinfo=TZ), cfg) is True
assert is_morning_push(datetime(2026, 5, 17, 6, 0, tzinfo=TZ), cfg) is True
def test_later_cron_match_not_morning():
"""触发时刻离非最早 cron 最近 → 默认"""
cfg = _cfg(["0 8 * * *", "0 17 * * *"])
assert is_morning_push(datetime(2026, 5, 17, 17, 0, tzinfo=TZ), cfg) is False
assert is_morning_push(datetime(2026, 5, 17, 16, 30, tzinfo=TZ), cfg) is False
assert is_morning_push(datetime(2026, 5, 17, 22, 0, tzinfo=TZ), cfg) is False
def test_drift_tolerance_via_closest_match():
"""无显式容差,但「最近匹配」自动吸附小幅漂移 (08:01 仍归 08:00)"""
cfg = _cfg(["0 8 * * *", "0 17 * * *"])
assert is_morning_push(datetime(2026, 5, 17, 8, 1, tzinfo=TZ), cfg) is True
assert is_morning_push(datetime(2026, 5, 17, 17, 1, tzinfo=TZ), cfg) is False
def test_three_crons_only_earliest_is_morning():
"""三条 cron 时,只有最早那条对应的触发是早报"""
cfg = _cfg(["0 8 * * *", "0 12 * * *", "0 20 * * *"])
assert is_morning_push(datetime(2026, 5, 17, 8, 0, tzinfo=TZ), cfg) is True
assert is_morning_push(datetime(2026, 5, 17, 12, 0, tzinfo=TZ), cfg) is False
assert is_morning_push(datetime(2026, 5, 17, 20, 0, tzinfo=TZ), cfg) is False
ai-daily-main/tests/pytest/test_processor.py
+87
View File
@@ -0,0 +1,87 @@
"""内容处理模块测试"""
import pytest
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from processor import html_to_markdown
class TestHtmlToMarkdown:
"""测试HTML转Markdown"""
def test_convert_basic_html(self):
html = "<p>Hello World</p>"
result = html_to_markdown(html)
assert "Hello World" in result
def test_convert_with_links(self):
html = '<a href="https://example.com">Click here</a>'
result = html_to_markdown(html)
assert "[Click here](https://example.com)" in result
def test_convert_with_images(self):
html = '<img src="https://example.com/image.png" alt="Image">'
result = html_to_markdown(html)
assert "![Image](https://example.com/image.png)" in result
def test_convert_with_headings(self):
html = "<h1>Title</h1><h2>Subtitle</h2>"
result = html_to_markdown(html)
assert "# Title" in result
assert "## Subtitle" in result
def test_convert_with_lists(self):
html = "<ul><li>Item 1</li><li>Item 2</li></ul>"
result = html_to_markdown(html)
assert "Item 1" in result
assert "Item 2" in result
def test_convert_with_strong_emphasis(self):
html = "<strong>Bold</strong> and <em>italic</em>"
result = html_to_markdown(html)
assert "**Bold**" in result
assert "*italic*" in result
def test_relative_link_conversion(self):
html = '<a href="/article/123">Read more</a>'
result = html_to_markdown(html, base_url="https://example.com/blog")
assert "https://example.com/article/123" in result
def test_relative_image_conversion(self):
html = '<img src="/images/logo.png">'
result = html_to_markdown(html, base_url="https://example.com")
assert "https://example.com/images/logo.png" in result
def test_absolute_link_unchanged(self):
html = '<a href="https://other.com/page">Link</a>'
result = html_to_markdown(html, base_url="https://example.com")
assert "https://other.com/page" in result
def test_remove_xgo_ing_link(self):
html = "<p>Content</p><p>[⚡ Powered by xgo.ing](https://xgo.ing)</p>"
result = html_to_markdown(html)
assert "xgo.ing" not in result
assert "Content" in result
def test_remove_xgo_ing_link_with_slash(self):
html = "<p>Content</p><p>[⚡ Powered by xgo.ing](https://xgo.ing/)</p>"
result = html_to_markdown(html)
assert "xgo.ing" not in result
def test_clean_extra_newlines(self):
html = "<p>Line 1</p>\n\n\n\n<p>Line 2</p>"
result = html_to_markdown(html)
assert "\n\n\n\n" not in result
def test_empty_html(self):
result = html_to_markdown("")
assert result.strip() == ""
def test_html_with_nbsp(self):
html = "<p>Hello&nbsp;World</p>"
result = html_to_markdown(html)
assert "Hello" in result
assert "World" in result
ai-daily-main/tests/pytest/test_push.py
+209
View File
@@ -0,0 +1,209 @@
"""推送模块测试"""
import os
import pytest
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock, MagicMock
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from push.discord import DiscordPlatform
from push.feishu import FeishuPlatform
from push import create_platform
class TestDiscordPlatform:
"""测试Discord推送"""
def test_validate_config_valid(self):
config = {
"enabled": True,
"apiKeyName": "DISCORD_WEBHOOK_URL",
}
with patch.dict(
os.environ,
{"DISCORD_WEBHOOK_URL": "https://discord.com/api/webhooks/123456/abcdef"},
):
platform = DiscordPlatform(config)
assert platform.validate_config(config) is True
def test_validate_config_disabled(self):
config = {
"enabled": False,
"apiKeyName": "DISCORD_WEBHOOK_URL",
}
with patch.dict(
os.environ,
{"DISCORD_WEBHOOK_URL": "https://discord.com/api/webhooks/123456/abcdef"},
):
platform = DiscordPlatform(config)
assert platform.validate_config(config) is False
def test_validate_config_missing_webhook(self):
config = {"enabled": True, "apiKeyName": "DISCORD_WEBHOOK_URL"}
with patch.dict(os.environ, {"DISCORD_WEBHOOK_URL": ""}):
platform = DiscordPlatform(config)
assert platform.validate_config(config) is False
def test_validate_config_invalid_url(self):
config = {"enabled": True, "apiKeyName": "DISCORD_WEBHOOK_URL"}
with patch.dict(os.environ, {"DISCORD_WEBHOOK_URL": "not-a-valid-url"}):
platform = DiscordPlatform(config)
assert platform.validate_config(config) is False
def test_validate_config_wrong_domain(self):
config = {"enabled": True, "apiKeyName": "DISCORD_WEBHOOK_URL"}
with patch.dict(
os.environ, {"DISCORD_WEBHOOK_URL": "https://example.com/webhook"}
):
platform = DiscordPlatform(config)
assert platform.validate_config(config) is False
def test_split_content_short(self):
with patch.dict(os.environ, {"DISCORD_WEBHOOK_URL": "https://test.com"}):
config = {"apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = DiscordPlatform(config)
short_content = "Hello"
chunks = platform._split_content(short_content, limit=2000)
assert len(chunks) == 1
assert chunks[0] == "Hello"
def test_split_content_long_message(self):
with patch.dict(os.environ, {"DISCORD_WEBHOOK_URL": "https://test.com"}):
config = {"apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = DiscordPlatform(config)
long_content = "A\n" * 2500
chunks = platform._split_content(long_content, limit=2000)
assert len(chunks) > 1
assert all(len(c) <= 2000 for c in chunks)
def test_split_content_exact_boundary(self):
with patch.dict(os.environ, {"DISCORD_WEBHOOK_URL": "https://test.com"}):
config = {"apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = DiscordPlatform(config)
content = "A" * 2000
chunks = platform._split_content(content, limit=2000)
assert len(chunks) == 1
def test_split_content_unicode(self):
with patch.dict(os.environ, {"DISCORD_WEBHOOK_URL": "https://test.com"}):
config = {"apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = DiscordPlatform(config)
content = "你好" * 500
chunks = platform._split_content(content, limit=100)
assert len(chunks) >= 1
@pytest.mark.asyncio
async def test_send_success(self):
with patch.dict(
os.environ,
{"DISCORD_WEBHOOK_URL": "https://discord.com/api/webhooks/test/abc"},
):
config = {"apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = DiscordPlatform(config)
with patch.object(platform, "send", new_callable=AsyncMock) as mock_send:
mock_send.return_value = True
result = await mock_send("Test message")
assert result is True
@pytest.mark.asyncio
async def test_send_failure(self):
with patch.dict(
os.environ,
{"DISCORD_WEBHOOK_URL": "https://discord.com/api/webhooks/test/abc"},
):
config = {"apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = DiscordPlatform(config)
with patch.object(platform, "send", new_callable=AsyncMock) as mock_send:
mock_send.return_value = False
result = await mock_send("Test message")
assert result is False
class TestFeishuPlatform:
"""测试飞书推送"""
def test_validate_config_valid(self):
config = {
"enabled": True,
"apiKeyName": "FEISHU_WEBHOOK_URL",
}
with patch.dict(os.environ, {"FEISHU_WEBHOOK_URL": "https://open.feishu.cn/open-apis/bot/v2/hook/test"}):
platform = FeishuPlatform(config)
assert platform.validate_config(config) is True
def test_validate_config_disabled(self):
config = {
"enabled": False,
"apiKeyName": "FEISHU_WEBHOOK_URL",
}
with patch.dict(os.environ, {"FEISHU_WEBHOOK_URL": "https://open.feishu.cn/open-apis/bot/v2/hook/test"}):
platform = FeishuPlatform(config)
assert platform.validate_config(config) is False
def test_validate_config_missing_key(self):
config = {"enabled": True, "apiKeyName": "FEISHU_WEBHOOK_URL"}
with patch.dict(os.environ, {"FEISHU_WEBHOOK_URL": ""}):
platform = FeishuPlatform(config)
assert platform.validate_config(config) is False
def test_validate_config_any_non_empty_webhook(self):
config = {"enabled": True, "apiKeyName": "FEISHU_WEBHOOK_URL"}
with patch.dict(os.environ, {"FEISHU_WEBHOOK_URL": "https://open.feishu.cn/open-apis/bot/v2/hook/test"}):
platform = FeishuPlatform(config)
assert platform.validate_config(config) is True
class TestPushFactory:
"""测试平台工厂"""
def test_create_enabled_platform(self):
config = {
"enabled": True,
"apiKeyName": "DISCORD_WEBHOOK_URL",
}
with patch.dict(
os.environ,
{"DISCORD_WEBHOOK_URL": "https://discord.com/api/webhooks/123/abc"},
):
platform = create_platform("discord", config)
assert platform is not None
def test_create_disabled_platform_returns_none(self):
config = {"enabled": False, "apiKeyName": "DISCORD_WEBHOOK_URL"}
platform = create_platform("discord", config)
assert platform is None
def test_create_unknown_platform_raises(self):
with pytest.raises(ValueError):
create_platform("unknown", {})
def test_create_feishu_platform(self):
config = {
"enabled": True,
"apiKeyName": "FEISHU_WEBHOOK_URL",
}
with patch.dict(os.environ, {"FEISHU_WEBHOOK_URL": "https://open.feishu.cn/open-apis/bot/v2/hook/test"}):
platform = create_platform("feishu", config)
assert platform is not None
assert isinstance(platform, FeishuPlatform)
def test_create_discord_platform(self):
config = {
"enabled": True,
"apiKeyName": "DISCORD_WEBHOOK_URL",
}
with patch.dict(
os.environ,
{"DISCORD_WEBHOOK_URL": "https://discord.com/api/webhooks/test/abc"},
):
platform = create_platform("discord", config)
assert platform is not None
assert isinstance(platform, DiscordPlatform)
ai-daily-main/tests/pytest/test_sections_github_enricher.py
+107
View File
@@ -0,0 +1,107 @@
"""测试 GitHub REST API enrich 字段映射、archived 过滤、token 鉴权头"""
import base64
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock, MagicMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.sections.github.repo_enricher import enrich_repo, _auth_headers
def test_auth_headers_with_token(monkeypatch):
monkeypatch.setenv("GITHUB_TOKEN", "ghp_secret")
headers = _auth_headers(token_env="GITHUB_TOKEN")
assert headers["Authorization"] == "Bearer ghp_secret"
assert headers["Accept"] == "application/vnd.github+json"
def test_auth_headers_without_token(monkeypatch):
monkeypatch.delenv("GITHUB_TOKEN", raising=False)
headers = _auth_headers(token_env="GITHUB_TOKEN")
assert "Authorization" not in headers
assert headers["Accept"] == "application/vnd.github+json"
@pytest.mark.asyncio
async def test_enrich_repo_merges_metadata_and_readme():
readme_body = "# Title\n\nProject description here."
readme_b64 = base64.b64encode(readme_body.encode("utf-8")).decode("ascii")
metadata_payload = {
"description": "real desc",
"topics": ["llm", "rag"],
"license": {"spdx_id": "MIT"},
"pushed_at": "2026-05-16T10:00:00Z",
"archived": False,
}
readme_payload = {"content": readme_b64, "encoding": "base64"}
async def fake_get_json(session, url, **kwargs):
if url.endswith("/readme"):
return readme_payload
return metadata_payload
base = {
"url": "https://github.com/o/r",
"full_name": "o/r",
"description": "from trending",
"language": "Python",
"stars_today": 100,
"stars_total": 5000,
}
with patch(
"src.sections.github.repo_enricher._get_json", new=AsyncMock(side_effect=fake_get_json)
):
enriched = await enrich_repo(
session=MagicMock(), repo=base, token_env="GITHUB_TOKEN", readme_max_chars=200
)
assert enriched["topics"] == ["llm", "rag"]
assert enriched["license"] == "MIT"
assert enriched["pushed_at"] == "2026-05-16T10:00:00Z"
assert "Project description" in enriched["readme_excerpt"]
assert enriched["stars_today"] == 100 # trending 已有字段保留
@pytest.mark.asyncio
async def test_enrich_repo_returns_none_when_archived():
metadata_payload = {"archived": True, "topics": [], "pushed_at": "x"}
async def fake_get_json(session, url, **kwargs):
if url.endswith("/readme"):
return {"content": ""}
return metadata_payload
base = {"url": "https://github.com/o/r", "full_name": "o/r"}
with patch(
"src.sections.github.repo_enricher._get_json", new=AsyncMock(side_effect=fake_get_json)
):
result = await enrich_repo(
session=MagicMock(), repo=base, token_env="GITHUB_TOKEN", readme_max_chars=200
)
assert result is None
@pytest.mark.asyncio
async def test_enrich_repo_truncates_readme():
readme_body = "x" * 5000
readme_b64 = base64.b64encode(readme_body.encode("utf-8")).decode("ascii")
async def fake_get_json(session, url, **kwargs):
if url.endswith("/readme"):
return {"content": readme_b64, "encoding": "base64"}
return {"archived": False, "topics": [], "pushed_at": "p"}
base = {"url": "https://github.com/o/r", "full_name": "o/r"}
with patch(
"src.sections.github.repo_enricher._get_json", new=AsyncMock(side_effect=fake_get_json)
):
enriched = await enrich_repo(
session=MagicMock(), repo=base, token_env="GITHUB_TOKEN", readme_max_chars=100
)
assert len(enriched["readme_excerpt"]) == 100
ai-daily-main/tests/pytest/test_sections_github_scraper.py
+40
View File
@@ -0,0 +1,40 @@
"""测试 GitHub trending HTML 解析"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.sections.github.trending_scraper import parse_trending_html
def test_parse_trending_html_returns_repo_dicts():
fixture = (
Path(__file__).parent / "fixtures" / "github_trending.html"
).read_text(encoding="utf-8")
repos = parse_trending_html(fixture)
assert len(repos) > 0
first = repos[0]
assert first["url"].startswith("https://github.com/")
assert "/" in first["full_name"]
assert isinstance(first["stars_today"], int)
assert isinstance(first["stars_total"], int)
# description / language 可为空字符串但必须是 str
assert isinstance(first["description"], str)
assert isinstance(first["language"], str)
def test_parse_trending_html_dedupes_by_url():
fixture = (
Path(__file__).parent / "fixtures" / "github_trending.html"
).read_text(encoding="utf-8")
repos = parse_trending_html(fixture)
urls = [r["url"] for r in repos]
assert len(urls) == len(set(urls))
def test_parse_trending_html_empty_input():
assert parse_trending_html("") == []
assert parse_trending_html("<html><body>no repos</body></html>") == []
ai-daily-main/tests/pytest/test_sections_github_section.py
+145
View File
@@ -0,0 +1,145 @@
"""测试 GitHub 板块编排:抓取 → history 过滤 → enrich → LLM 总结"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.sections.github.section import run_github_section
# Stub summarize_github_trending until Task 11 provides it
import src.llm as _llm
if not hasattr(_llm, "summarize_github_trending"):
async def _stub(*a, **k):
return "", None
_llm.summarize_github_trending = _stub
def _cfg(history_file: str, max_deep_dive: int = 10) -> dict:
return {
"filter": {"keep_days": 7},
"sections": {
"github_trending": {
"enabled": True,
"max_items": 3,
"max_deep_dive": max_deep_dive,
"readme_max_chars": 3000,
"history_file": history_file,
"request_timeout": 10,
"tokenName": "GITHUB_TOKEN",
}
},
"llm": {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"section_github": "prompts/section_github.md"},
},
}
@pytest.mark.asyncio
async def test_disabled_returns_empty(tmp_path):
cfg = _cfg(str(tmp_path / "h.json"))
cfg["sections"]["github_trending"]["enabled"] = False
md, err = await run_github_section(cfg, now=None)
assert md == ""
assert err is None
@pytest.mark.asyncio
async def test_no_candidates_after_history_returns_empty(tmp_path):
history_path = tmp_path / "h.json"
# 预置 history,使得今日 scrape 出来的 repo 都已存在
history_path.write_text(
'{"repos": {"https://github.com/a/b": "2026-05-16"}, "updated_at": "x"}',
encoding="utf-8",
)
cfg = _cfg(str(history_path))
with patch(
"src.sections.github.section.fetch_trending_page", new=AsyncMock(return_value="<html>")
), patch(
"src.sections.github.section.parse_trending_html",
return_value=[{"url": "https://github.com/a/b", "full_name": "a/b"}],
):
md, err = await run_github_section(cfg, now=None)
assert md == ""
assert err is None
@pytest.mark.asyncio
async def test_happy_path_enriches_and_summarizes(tmp_path):
history_path = tmp_path / "h.json"
cfg = _cfg(str(history_path), max_deep_dive=10)
repos = [
{
"url": "https://github.com/o1/r1",
"full_name": "o1/r1",
"description": "d1",
"language": "Python",
"stars_today": 100,
"stars_total": 1000,
}
]
enriched = [{**repos[0], "topics": ["llm"], "license": "MIT", "pushed_at": "p", "readme_excerpt": "rm"}]
with patch(
"src.sections.github.section.fetch_trending_page", new=AsyncMock(return_value="<html>")
), patch(
"src.sections.github.section.parse_trending_html", return_value=repos
), patch(
"src.sections.github.section.enrich_repos",
new=AsyncMock(return_value=(enriched, [])),
), patch(
"src.llm.summarize_github_trending",
new=AsyncMock(return_value=("## GH section md", None)),
):
md, err = await run_github_section(cfg, now=None)
assert md == "## GH section md"
assert err is None
import json as _j
saved = _j.loads(history_path.read_text(encoding="utf-8"))
assert "https://github.com/o1/r1" in saved["repos"]
@pytest.mark.asyncio
async def test_truncates_candidates_to_max_deep_dive(tmp_path):
cfg = _cfg(str(tmp_path / "h.json"), max_deep_dive=2)
repos = [
{"url": f"https://github.com/o/r{i}", "full_name": f"o/r{i}"} for i in range(5)
]
captured = {}
async def fake_enrich(candidates, **kwargs):
captured["count"] = len(candidates)
return [], []
with patch(
"src.sections.github.section.fetch_trending_page", new=AsyncMock(return_value="<html>")
), patch(
"src.sections.github.section.parse_trending_html", return_value=repos
), patch(
"src.sections.github.section.enrich_repos", new=AsyncMock(side_effect=fake_enrich)
):
await run_github_section(cfg, now=None)
assert captured["count"] == 2
@pytest.mark.asyncio
async def test_scrape_failure_returns_error(tmp_path):
cfg = _cfg(str(tmp_path / "h.json"))
with patch(
"src.sections.github.section.fetch_trending_page",
new=AsyncMock(side_effect=RuntimeError("HTTP 500")),
):
md, err = await run_github_section(cfg, now=None)
assert md == ""
assert "HTTP 500" in err
ai-daily-main/tests/pytest/test_sections_hackernews_enricher.py
+246
View File
@@ -0,0 +1,246 @@
"""测试 HN enrich(Algolia 评论树 + 外链正文)"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock, MagicMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.sections.hackernews.item_enricher import enrich_story
def _kwargs(**overrides):
base = dict(
top_comments=3,
top_l2_per_l1=2,
comment_max_chars=500,
comments_total_chars=60000,
link_content_max_chars=3000,
algolia_base="https://hn.algolia.com/api/v1",
timeout=10,
)
base.update(overrides)
return base
@pytest.mark.asyncio
async def test_enrich_external_link_story_returns_tree():
story = {
"id": "111",
"title": "T",
"url": "https://example.com/post",
"site": "example.com",
"points": 100,
"comments": 5,
"comments_url": "https://news.ycombinator.com/item?id=111",
}
algolia_payload = {
"text": None,
"children": [
{
"text": "<p>comment one</p>",
"children": [
{"text": "<p>reply 1a</p>"},
{"text": "<p>reply 1b</p>"},
{"text": "<p>reply 1c (should be dropped)</p>"},
],
},
{"text": "<p>comment two</p>", "children": []},
{"text": "<p>comment three</p>"},
{"text": "<p>comment four (over top_comments cap)</p>"},
],
}
async def fake_algolia(session, item_id, **kw):
return algolia_payload
async def fake_external(session, url, **kw):
return "link body"
with patch(
"src.sections.hackernews.item_enricher._fetch_algolia_item",
new=AsyncMock(side_effect=fake_algolia),
), patch(
"src.sections.hackernews.item_enricher._fetch_external_markdown",
new=AsyncMock(side_effect=fake_external),
):
enriched = await enrich_story(
session=MagicMock(), story=story, **_kwargs()
)
tree = enriched["top_comments"]
assert len(tree) == 3
assert "comment one" in tree[0]["l1"]
assert len(tree[0]["replies"]) == 2
assert "reply 1a" in tree[0]["replies"][0]
assert "reply 1b" in tree[0]["replies"][1]
assert tree[1]["replies"] == []
assert tree[2]["replies"] == []
assert "link body" in enriched["link_content"]
@pytest.mark.asyncio
async def test_enrich_show_hn_uses_root_text_no_external_fetch():
story = {
"id": "222",
"title": "Show HN: T",
"url": "https://news.ycombinator.com/item?id=222",
"site": "",
"points": 200,
"comments": 10,
"comments_url": "https://news.ycombinator.com/item?id=222",
}
algolia_payload = {
"text": "<p>post body text</p>",
"children": [{"text": "<p>c1</p>"}],
}
link_calls = []
async def fake_algolia(session, item_id, **kw):
return algolia_payload
async def fake_external(session, url, **kw):
link_calls.append(url)
return "should not be called"
with patch(
"src.sections.hackernews.item_enricher._fetch_algolia_item",
new=AsyncMock(side_effect=fake_algolia),
), patch(
"src.sections.hackernews.item_enricher._fetch_external_markdown",
new=AsyncMock(side_effect=fake_external),
):
enriched = await enrich_story(
session=MagicMock(), story=story, **_kwargs()
)
assert link_calls == []
assert "post body text" in enriched["link_content"]
assert enriched["top_comments"][0]["l1"].startswith("c1") or "c1" in enriched["top_comments"][0]["l1"]
@pytest.mark.asyncio
async def test_enrich_truncates_comments_and_link():
story = {
"id": "333",
"title": "T",
"url": "https://example.com/a",
"site": "example.com",
"points": 100,
"comments": 2,
"comments_url": "x",
}
long_comment = "<p>" + ("y" * 2000) + "</p>"
long_reply = "<p>" + ("z" * 2000) + "</p>"
long_link = "z" * 5000
async def fake_algolia(session, item_id, **kw):
return {
"text": None,
"children": [
{"text": long_comment, "children": [{"text": long_reply}]}
],
}
async def fake_external(session, url, **kw):
return long_link
with patch(
"src.sections.hackernews.item_enricher._fetch_algolia_item",
new=AsyncMock(side_effect=fake_algolia),
), patch(
"src.sections.hackernews.item_enricher._fetch_external_markdown",
new=AsyncMock(side_effect=fake_external),
):
enriched = await enrich_story(
session=MagicMock(),
story=story,
**_kwargs(comment_max_chars=100, link_content_max_chars=200),
)
assert len(enriched["top_comments"][0]["l1"]) <= 100
assert len(enriched["top_comments"][0]["replies"][0]) <= 100
assert len(enriched["link_content"]) <= 200
@pytest.mark.asyncio
async def test_enrich_failure_returns_partial():
story = {
"id": "444",
"title": "T",
"url": "https://example.com/x",
"site": "example.com",
"points": 100,
"comments": 2,
"comments_url": "x",
}
async def fake_algolia(session, item_id, **kw):
raise RuntimeError("algolia down")
async def fake_external(session, url, **kw):
return "ok"
with patch(
"src.sections.hackernews.item_enricher._fetch_algolia_item",
new=AsyncMock(side_effect=fake_algolia),
), patch(
"src.sections.hackernews.item_enricher._fetch_external_markdown",
new=AsyncMock(side_effect=fake_external),
):
enriched = await enrich_story(
session=MagicMock(), story=story, **_kwargs()
)
assert enriched["top_comments"] == []
assert "ok" in enriched["link_content"]
@pytest.mark.asyncio
async def test_enrich_total_budget_stops_early():
"""累计字符达 comments_total_chars 立即停止,后续 L1 / L2 都不再加入"""
story = {
"id": "555",
"title": "T",
"url": "https://example.com/q",
"site": "example.com",
"points": 100,
"comments": 5,
"comments_url": "x",
}
big = "<p>" + ("a" * 500) + "</p>" # markdown 约 500 chars
async def fake_algolia(session, item_id, **kw):
return {
"text": None,
"children": [{"text": big, "children": [{"text": big}, {"text": big}]} for _ in range(10)],
}
async def fake_external(session, url, **kw):
return "x"
with patch(
"src.sections.hackernews.item_enricher._fetch_algolia_item",
new=AsyncMock(side_effect=fake_algolia),
), patch(
"src.sections.hackernews.item_enricher._fetch_external_markdown",
new=AsyncMock(side_effect=fake_external),
):
enriched = await enrich_story(
session=MagicMock(),
story=story,
**_kwargs(
top_comments=10,
top_l2_per_l1=2,
comment_max_chars=500,
comments_total_chars=1500,
),
)
tree = enriched["top_comments"]
total = sum(len(n["l1"]) + sum(len(r) for r in n["replies"]) for n in tree)
# 累计应该在 1500 附近停下(允许多收一条到 ~2000),不应该收全 10*3=30 条
assert total <= 2000
assert len(tree) < 10
ai-daily-main/tests/pytest/test_sections_hackernews_scraper.py
+59
View File
@@ -0,0 +1,59 @@
"""测试 HN 首页 HTML 解析"""
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.sections.hackernews.frontpage_scraper import parse_frontpage_html
def test_parse_frontpage_returns_stories():
fixture = (Path(__file__).parent / "fixtures" / "hn_frontpage.html").read_text(
encoding="utf-8"
)
stories = parse_frontpage_html(fixture)
assert len(stories) >= 25
s = stories[0]
print(json.dumps(stories[:5], indent=4, ensure_ascii=False))
assert s["id"]
assert s["title"]
assert s["url"]
assert isinstance(s["points"], int)
assert isinstance(s["comments"], int)
assert s["comments_url"].startswith("https://news.ycombinator.com/item?id=")
def test_parse_frontpage_detects_show_hn_internal_url():
html = """
<table>
<tr class="athing" id="111">
<td class="title">
<span class="titleline">
<a href="item?id=111">Ask HN: what's new?</a>
</span>
</td>
</tr>
<tr>
<td class="subtext">
<span class="subline">
<span class="score">50 points</span>
by <a href="user?id=alice">alice</a>
<span class="age"><a href="item?id=111">2 hours ago</a></span>
| <a href="item?id=111">5&nbsp;comments</a>
</span>
</td>
</tr>
</table>
"""
stories = parse_frontpage_html(html)
assert len(stories) == 1
s = stories[0]
assert s["id"] == "111"
assert s["url"].startswith("https://news.ycombinator.com/item?id=")
assert s["site"] == ""
assert s["points"] == 50
assert s["comments"] == 5
ai-daily-main/tests/pytest/test_sections_hackernews_section.py
+111
View File
@@ -0,0 +1,111 @@
"""测试 HN 板块编排:scrape → select → enrich → LLM"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
# Stub HN LLM functions until Task 16 lands them
import src.llm as _llm
if not hasattr(_llm, "select_ai_related_hn"):
async def _stub_select(*a, **k):
return [], None
_llm.select_ai_related_hn = _stub_select
if not hasattr(_llm, "summarize_hackernews"):
async def _stub_summarize(*a, **k):
return "", None
_llm.summarize_hackernews = _stub_summarize
from src.sections.hackernews.section import run_hackernews_section
def _cfg() -> dict:
return {
"filter": {"keep_days": 7},
"sections": {
"hackernews": {
"enabled": True,
"select_k": 1,
"top_comments": 20,
"comment_max_chars": 500,
"link_content_max_chars": 3000,
"request_timeout": 10,
"algolia_base": "https://hn.algolia.com/api/v1",
}
},
"llm": {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {
"section_hackernews_select": "prompts/section_hackernews_select.md",
"section_hackernews": "prompts/section_hackernews.md",
},
},
}
@pytest.mark.asyncio
async def test_disabled_returns_empty():
cfg = _cfg()
cfg["sections"]["hackernews"]["enabled"] = False
md, err = await run_hackernews_section(cfg, now=None)
assert md == ""
assert err is None
@pytest.mark.asyncio
async def test_select_empty_returns_silent():
cfg = _cfg()
with patch(
"src.sections.hackernews.section.fetch_frontpage", new=AsyncMock(return_value="<html>")
), patch(
"src.sections.hackernews.section.parse_frontpage_html",
return_value=[{"id": "1", "title": "x"}],
), patch(
"src.llm.select_ai_related_hn",
new=AsyncMock(return_value=([], None)),
):
md, err = await run_hackernews_section(cfg, now=None)
assert md == ""
assert err is None
@pytest.mark.asyncio
async def test_happy_path():
cfg = _cfg()
front = [{"id": "1", "title": "AI thing", "url": "https://e.com/a", "site": "e.com", "points": 100, "comments": 5, "comments_url": "x"}]
enriched = [{**front[0], "link_content": "body", "top_comments": ["c1"]}]
with patch(
"src.sections.hackernews.section.fetch_frontpage", new=AsyncMock(return_value="<html>")
), patch(
"src.sections.hackernews.section.parse_frontpage_html", return_value=front
), patch(
"src.llm.select_ai_related_hn",
new=AsyncMock(return_value=(["1"], None)),
), patch(
"src.sections.hackernews.section.enrich_stories",
new=AsyncMock(return_value=(enriched, [])),
), patch(
"src.llm.summarize_hackernews",
new=AsyncMock(return_value=("## HN md", None)),
):
md, err = await run_hackernews_section(cfg, now=None)
assert md == "## HN md"
assert err is None
@pytest.mark.asyncio
async def test_scrape_failure_returns_error():
cfg = _cfg()
with patch(
"src.sections.hackernews.section.fetch_frontpage",
new=AsyncMock(side_effect=RuntimeError("net")),
):
md, err = await run_hackernews_section(cfg, now=None)
assert md == ""
assert "net" in err
ai-daily-main/tests/pytest/test_sections_insights.py
+66
View File
@@ -0,0 +1,66 @@
"""测试 insights 模块"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
# Stub Task 17's LLM function until it lands
import src.llm as _llm
if not hasattr(_llm, "generate_trend_insights"):
async def _stub(*a, **k):
return "", None
_llm.generate_trend_insights = _stub
from src.sections.insights.section import run_insights_section
def _cfg() -> dict:
return {
"filter": {"push_context_days": 5},
"sections": {"insights": {"enabled": True}},
"llm": {
"model": "x",
"baseUrl": "http://x",
"apiKeyName": "DEEPSEEK_API_KEY",
"prompts": {"insights": "prompts/insights.md"},
},
}
@pytest.mark.asyncio
async def test_disabled_returns_empty():
cfg = _cfg()
cfg["sections"]["insights"]["enabled"] = False
md, meta, err = await run_insights_section("rss", "gh", "hn", cfg, now=None)
assert md == ""
assert meta is None
assert err is None
@pytest.mark.asyncio
async def test_marks_empty_sections_for_llm():
cfg = _cfg()
captured = {}
async def fake_gen(sections, config):
captured["sections"] = sections
return "insights md", None
with patch(
"src.llm.generate_trend_insights",
new=AsyncMock(side_effect=fake_gen),
):
md, meta, err = await run_insights_section("", "gh md", "", cfg, now=None)
assert md == "insights md"
assert err is None
# metadata 由 parse_insights_with_metadata 注入默认标题/profile
assert meta["profile"] == "morning"
assert "📰 AI Daily 每日精选" in meta["title"]
assert captured["sections"]["rss"] == "(本次无内容)"
assert captured["sections"]["github"] == "gh md"
assert captured["sections"]["hackernews"] == "(本次无内容)"
ai-daily-main/tests/pytest/test_sections_rss.py
+76
View File
@@ -0,0 +1,76 @@
"""src.main.collect_entries_for_push patched at source (lazy import in section)"""
import sys
from pathlib import Path
from unittest.mock import patch, AsyncMock
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.sections.rss.section import run_rss_section
@pytest.mark.asyncio
async def test_returns_markdown_when_entries_present(sample_config, tmp_path):
digest_raw = (
"---\n"
'title: "🌙 AI Daily 晚报 | 测试"\n'
'lead: "今日测试导读"\n'
"highlights:\n - 重点1\n"
"---\n\n"
"### 1️⃣ digest body"
)
with patch(
"src.main.collect_entries_for_push",
return_value=([{"link": "x", "title": "t", "score": 80}], []),
), patch(
"src.sections.rss.section.compose_digest",
new=AsyncMock(return_value=digest_raw),
), patch(
"src.sections.rss.section.load_recent_push_content", return_value=""
), patch(
"src.sections.rss.section.get_last_push_file", return_value=None
):
md, meta, err = await run_rss_section(sample_config, now=None)
assert md == "### 1️⃣ digest body"
assert err is None
assert meta["title"] == "🌙 AI Daily 晚报 | 测试"
assert meta["lead"] == "今日测试导读"
assert meta["highlights"] == ["重点1"]
assert meta["profile"] == "default"
@pytest.mark.asyncio
async def test_returns_empty_when_no_entries(sample_config):
with patch(
"src.main.collect_entries_for_push", return_value=([], [])
), patch(
"src.sections.rss.section.get_last_push_file", return_value=None
):
md, meta, err = await run_rss_section(sample_config, now=None)
assert md == ""
assert meta is None
assert err is None
@pytest.mark.asyncio
async def test_returns_error_on_compose_failure(sample_config):
with patch(
"src.main.collect_entries_for_push",
return_value=([{"link": "x"}], []),
), patch(
"src.sections.rss.section.compose_digest",
new=AsyncMock(side_effect=RuntimeError("LLM down")),
), patch(
"src.sections.rss.section.load_recent_push_content", return_value=""
), patch(
"src.sections.rss.section.get_last_push_file", return_value=None
):
md, meta, err = await run_rss_section(sample_config, now=None)
assert md == ""
assert meta is None
assert "LLM down" in err
ai-daily-main/tests/pytest/test_storage.py
+257
View File
@@ -0,0 +1,257 @@
"""存储模块测试"""
import json
import pytest
import sys
from datetime import datetime, date, timedelta, timezone
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from storage import (
get_fetch_file,
get_push_file,
get_last_push_file,
extract_push_time,
read_entries,
read_fetch_data,
save_fetch_file,
append_entries,
format_entry,
json_to_md,
save_push_file,
load_existing_links,
cleanup_old_files,
)
class TestGetFetchFile:
"""测试获取fetch文件路径"""
def test_get_fetch_file_default(self):
result = get_fetch_file()
assert "fetch-" in result
assert result.endswith(".json")
def test_get_fetch_file_specific_date(self):
result = get_fetch_file(date(2024, 1, 15))
assert "fetch-2024-01-15.json" in result
class TestGetPushFile:
"""测试获取push文件路径"""
def test_get_push_file_default(self):
result = get_push_file()
assert "push-" in result
assert result.endswith(".md")
def test_get_push_file_specific_time(self):
dt = datetime(2024, 1, 15, 8, 30, 0)
result = get_push_file(dt)
assert "push-2024-01-15-08-30-00.md" in result
class TestExtractPushTime:
"""测试从文件名提取时间"""
def test_extract_valid_time(self):
result = extract_push_time("news-data/push-2024-01-15-08-30-00.md")
assert result is not None
assert result.year == 2024
assert result.month == 1
assert result.day == 15
def test_extract_invalid_filename(self):
result = extract_push_time("invalid.md")
assert result is None
class TestGetLastPushFile:
"""测试获取最新push文件"""
def test_get_last_push_file_empty(self, temp_dir):
result = get_last_push_file(str(temp_dir))
assert result is None
def test_get_last_push_file_exists(self, temp_dir):
(temp_dir / "push-2024-01-14-10-00-00.md").touch()
(temp_dir / "push-2024-01-15-10-00-00.md").touch()
result = get_last_push_file(str(temp_dir))
assert "2024-01-15" in result
class TestReadWriteEntries:
"""测试读写条目"""
def test_read_entries(self, sample_fetch_json):
entries = read_entries(sample_fetch_json)
assert len(entries) == 3
assert entries[0]["title"] == "Article 1"
def test_read_entries_missing_file(self):
entries = read_entries("nonexistent.json")
assert entries == []
def test_read_fetch_data(self, sample_fetch_json):
data = read_fetch_data(sample_fetch_json)
assert "meta" in data
assert "entries" in data
assert len(data["entries"]) == 3
class TestSaveFetchFile:
"""测试保存fetch文件"""
def test_save_fetch_file(self, temp_dir):
filepath = str(temp_dir / "test.json")
meta = {"date": "2024-01-15"}
entries = [{"title": "Test", "link": "https://example.com", "score": 80}]
save_fetch_file(filepath, meta, entries)
data = read_fetch_data(filepath)
assert data["meta"]["date"] == "2024-01-15"
assert len(data["entries"]) == 1
class TestAppendEntries:
"""测试追加条目"""
def test_append_new_entries(self, temp_dir):
filepath = str(temp_dir / "test.json")
meta = {"date": "2024-01-15"}
entries1 = [{"title": "Entry1", "link": "https://example.com/1", "score": 80}]
count1 = append_entries(filepath, entries1, meta)
assert count1 == 1
entries2 = [{"title": "Entry2", "link": "https://example.com/2", "score": 70}]
count2 = append_entries(filepath, entries2, meta)
assert count2 == 1
def test_append_duplicate_entries(self, temp_dir):
filepath = str(temp_dir / "test.json")
meta = {"date": "2024-01-15"}
entries = [{"title": "Entry1", "link": "https://example.com/1", "score": 80}]
append_entries(filepath, entries, meta)
all_entries = read_entries(filepath)
assert len(all_entries) == 1
count = append_entries(filepath, entries, meta)
all_entries_after = read_entries(filepath)
assert len(all_entries_after) == 1
def test_append_to_existing_file(self, temp_dir):
filepath = str(temp_dir / "test.json")
data = {
"meta": {"date": "2024-01-15"},
"entries": [{"title": "Old", "link": "https://old.com", "score": 60}],
}
with open(filepath, "w") as f:
json.dump(data, f)
new_entries = [{"title": "New", "link": "https://new.com", "score": 70}]
count = append_entries(filepath, new_entries)
entries = read_entries(filepath)
assert len(entries) == 2
class TestFormatEntry:
"""测试格式化条目"""
def test_format_entry_basic(self, sample_entry):
result = format_entry(sample_entry)
assert "## Test Article Title" in result
assert "source: Test Source" in result
assert "score: 85" in result
def test_format_entry_with_tags(self, sample_entry):
result = format_entry(sample_entry)
assert "AI" in result
assert "Tech" in result
class TestJsonToMd:
"""测试JSON转Markdown"""
def test_json_to_md_basic(self, sample_fetch_json):
data = read_fetch_data(sample_fetch_json)
result = json_to_md(data)
assert "Article 1" in result
assert "Article 2" in result
assert "Article 3" in result
def test_json_to_md_empty(self):
data = {"meta": {}, "entries": []}
result = json_to_md(data)
assert result == ""
class TestSavePushFile:
"""测试保存推送文件"""
def test_save_push_file(self, temp_dir):
filepath = str(temp_dir / "push-test.md")
content = "# Test Push\n\nContent here"
save_push_file(filepath, content, 5, 10)
with open(filepath, "r") as f:
content = f.read()
assert "pushDate:" in content
assert "sourceCount: 5" in content
assert "totalEntries: 10" in content
assert "# Test Push" in content
class TestLoadExistingLinks:
"""测试加载已有链接"""
def test_load_existing_links_json(self, sample_fetch_json):
links = load_existing_links(sample_fetch_json)
assert len(links) == 3
assert "https://example.com/1" in links
def test_load_existing_links_missing(self):
links = load_existing_links("nonexistent.json")
assert links == set()
def test_load_existing_links_empty_string(self):
links = load_existing_links("")
assert links == set()
class TestCleanupOldFiles:
"""测试清理旧文件"""
def test_cleanup_old_files(self, temp_dir):
old_date = (datetime.now() - timedelta(days=10)).date()
new_date = (datetime.now() - timedelta(days=1)).date()
(temp_dir / f"fetch-{old_date}.json").touch()
(temp_dir / f"fetch-{new_date}.json").touch()
cleanup_old_files(days=7, data_dir=str(temp_dir))
assert not (temp_dir / f"fetch-{old_date}.json").exists()
assert (temp_dir / f"fetch-{new_date}.json").exists()
def test_cleanup_push_files(self, temp_dir):
old_time = datetime.now() - timedelta(days=10)
new_time = datetime.now() - timedelta(days=1)
(temp_dir / f"push-{old_time.strftime('%Y-%m-%d-%H-%M-%S')}.md").touch()
(temp_dir / f"push-{new_time.strftime('%Y-%m-%d-%H-%M-%S')}.md").touch()
cleanup_old_files(days=7, data_dir=str(temp_dir))
files = list(temp_dir.glob("push-*.md"))
assert len(files) == 1
ai-daily-main/tests/pytest/test_storage_sections.py
+163
View File
@@ -0,0 +1,163 @@
"""测试新增的 sentinel 切片与 section-aware 读取"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "src"))
from storage import extract_section
class TestExtractSection:
def test_extract_section_with_sentinel(self):
md = (
"intro\n"
"<!-- SECTION:rss BEGIN -->\n"
"RSS body\n"
"<!-- SECTION:rss END -->\n"
"\n"
"<!-- SECTION:github BEGIN -->\n"
"GH body\n"
"<!-- SECTION:github END -->\n"
)
assert extract_section(md, "rss").strip() == "RSS body"
assert extract_section(md, "github").strip() == "GH body"
assert extract_section(md, "hackernews") == ""
def test_extract_section_legacy_file_rss(self):
legacy = "# AI Daily\n### 1️⃣ foo\n### 2️⃣ bar\n"
assert extract_section(legacy, "rss") == legacy
def test_extract_section_legacy_file_non_rss(self):
legacy = "# AI Daily\n### 1️⃣ foo\n"
assert extract_section(legacy, "github") == ""
assert extract_section(legacy, "hackernews") == ""
assert extract_section(legacy, "insights") == ""
def test_extract_section_missing_end_marker(self):
broken = "<!-- SECTION:rss BEGIN -->\ncontent only\n"
assert extract_section(broken, "rss") == ""
from datetime import date, datetime, timedelta
from storage import save_push_file
from storage import TrendingHistory, load_trending_history
class TestTrendingHistory:
def test_load_missing_file(self, tmp_path):
path = tmp_path / "trending.json"
h = load_trending_history(str(path))
assert h.repos == {}
def test_touch_then_save_then_reload(self, tmp_path):
path = tmp_path / "trending.json"
h = load_trending_history(str(path))
today = date(2026, 5, 17)
h.touch("https://github.com/a/b", today)
h.touch("https://github.com/c/d", today)
h.save()
h2 = load_trending_history(str(path))
assert h2.repos == {
"https://github.com/a/b": "2026-05-17",
"https://github.com/c/d": "2026-05-17",
}
def test_contains_returns_membership(self, tmp_path):
h = load_trending_history(str(tmp_path / "x.json"))
h.touch("https://github.com/a/b", date(2026, 5, 17))
assert "https://github.com/a/b" in h
assert "https://github.com/x/y" not in h
def test_cleanup_removes_expired_entries(self, tmp_path):
path = tmp_path / "trending.json"
path.write_text(
'{"repos": {'
'"https://github.com/old/repo": "2026-05-01", '
'"https://github.com/new/repo": "2026-05-15"'
'}, "updated_at": "2026-05-15T00:00:00+08:00"}',
encoding="utf-8",
)
h = load_trending_history(str(path))
h.cleanup(today=date(2026, 5, 17), keep_days=7)
assert "https://github.com/old/repo" not in h
assert "https://github.com/new/repo" in h
def test_cleanup_keeps_today_inclusive(self, tmp_path):
h = load_trending_history(str(tmp_path / "x.json"))
h.touch("https://github.com/a/b", date(2026, 5, 10))
# 2026-05-10 + 7 days = 2026-05-17 (last_seen 2026-05-10 仍在 keep 区间)
h.cleanup(today=date(2026, 5, 17), keep_days=7)
assert "https://github.com/a/b" in h
# 再过 1 天就出区间
h.cleanup(today=date(2026, 5, 18), keep_days=7)
assert "https://github.com/a/b" not in h
class TestSavePushFileProfile:
def test_default_profile_when_not_specified(self, tmp_path):
f = tmp_path / "push-x.md"
save_push_file(str(f), "body content", source_count=1, total_entries=1)
text = f.read_text(encoding="utf-8")
assert 'profile: "default"' in text
assert "body content" in text
def test_morning_profile(self, tmp_path):
f = tmp_path / "push-x.md"
save_push_file(
str(f), "body", source_count=2, total_entries=3, profile="morning"
)
text = f.read_text(encoding="utf-8")
assert 'profile: "morning"' in text
from storage import cleanup_old_files
import json as _j
class TestCleanupOldFilesTrendingHistory:
def test_prunes_trending_history_entries_not_file(self, tmp_path):
path = tmp_path / "trending-history.json"
old_date = (datetime.now().date() - timedelta(days=30)).isoformat()
fresh_date = datetime.now().date().isoformat()
path.write_text(
'{"repos": {'
f'"https://github.com/a/b": "{old_date}", '
f'"https://github.com/c/d": "{fresh_date}"'
'}, "updated_at": "..."}',
encoding="utf-8",
)
cleanup_old_files(days=7, data_dir=str(tmp_path))
# 文件应保留
assert path.exists()
# 过期条目应被剪枝
data = _j.loads(path.read_text(encoding="utf-8"))
assert "https://github.com/a/b" not in data["repos"]
assert "https://github.com/c/d" in data["repos"]
from storage import assemble_with_sentinels
class TestAssembleWithSentinels:
def test_assembles_all_sections_in_order(self):
out = assemble_with_sentinels(
{"rss": "R", "github": "G", "hackernews": "H", "insights": "I"}
)
assert out.index("SECTION:rss") < out.index("SECTION:github")
assert out.index("SECTION:github") < out.index("SECTION:hackernews")
assert out.index("SECTION:hackernews") < out.index("SECTION:insights")
assert "<!-- SECTION:rss BEGIN -->\nR\n<!-- SECTION:rss END -->" in out
def test_omits_empty_sections(self):
out = assemble_with_sentinels({"rss": "R", "github": "", "hackernews": "H", "insights": ""})
assert "SECTION:github" not in out
assert "SECTION:insights" not in out
assert "SECTION:rss" in out
assert "SECTION:hackernews" in out
def test_returns_empty_when_all_empty(self):
assert assemble_with_sentinels({"rss": "", "github": "", "hackernews": "", "insights": ""}) == ""

Some files were not shown because too many files have changed in this diff Show More