omni: менеджер источников в панели (338 источников, вкл/выкл, добавление, health-check, фоновые джобы)
This commit is contained in:
reform
55
README.md
55
README.md
@@ -20,7 +20,7 @@
|
||||
| `firecrawl` | Firecrawl search | `FIRECRAWL_API_KEY` |
|
||||
| `cards` | карточки товаров: JSON-LD Product/Offer, OpenGraph, microdata, цена/валюта/наличие | нет |
|
||||
| `treg` | treg-поиск | `TREG_TOKEN` |
|
||||
| `feed` | обход реестра из 113 лент/API без запроса (см. ниже) | нет |
|
||||
| `feed` | обход реестра из 338 лент/API без запроса (см. ниже) | нет |
|
||||
|
||||
`python -m omni providers` печатает, что реально поднялось в этом окружении.
|
||||
|
||||
@@ -44,7 +44,7 @@ python -m omni search "scraping twitter without api" --providers reddit,forums,d
|
||||
python -m omni sources
|
||||
python -m omni sources --category papers,security --list
|
||||
|
||||
# обход реестра БЕЗ запроса: 113 источников -> база
|
||||
# обход реестра БЕЗ запроса: 338 источников -> база
|
||||
python -m omni feed --category community --limit 8
|
||||
python -m omni feed --limit 15 # всё подряд
|
||||
python -m omni feed --only arxiv_cs_ai,hn_top,reddit_localllama
|
||||
@@ -66,19 +66,48 @@ python -m omni export --format jsonl --out data/dump.jsonl
|
||||
|
||||
## Реестр источников
|
||||
|
||||
`omni/sources.py` — 113 источников в 9 категориях, каждый с адаптером:
|
||||
`omni/sources.py` + `omni/sources_ext.py` — **338 источников в 9 категориях**, у каждого свой адаптер и флаг вкл/выкл:
|
||||
|
||||
- **papers 18** — arxiv RSS по 15 разделам (cs.AI/LG/CL/CV/RO/IR/CR/SE/NE/MA/DC/DB/HC, stat.ML, eess.AS), HF daily papers, OpenAlex, Semantic Scholar
|
||||
- **company 34** — ленты OpenAI, Google AI/Research, Microsoft Research, NVIDIA press+research, Apple ML, Amazon Science, HF blog, Meta AI, Mistral, Cohere, Together, Perplexity, Groq, Cerebras, Stability, Databricks, Anyscale, Modal, W&B, LangChain, LlamaIndex, vLLM, Replicate, Fireworks, SambaNova, Scale, xAI, Salesforce, AWS ML, Azure AI
|
||||
- **company_cn 14** — Qwen, DeepSeek changelog, Moonshot, Zhipu, Baichuan, MiniMax + GitHub-орги Tencent-Hunyuan, PaddlePaddle, bytedance, mindspore-ai, QwenLM, deepseek-ai, MoonshotAI, THUDM
|
||||
- **company_kr 4** — Kakao tech, Naver Cloud, Samsung Research, LG AI Research
|
||||
- **community 17** — HN top/new, Lobsters (3 тега), GeekNews, 11 сабреддитов
|
||||
- **security 10** — The Hacker News, BleepingComputer, Krebs, SANS ISC, Packet Storm, CISA, Exploit-DB, Project Zero, PortSwigger, GitHub Security
|
||||
- **forum 9** — 6 Discourse-инстансов, XenForo BHW, веб-версии Telegram-каналов
|
||||
- **models 5** — HF trending/new (модели, спейсы, датасеты), Ollama library
|
||||
- **benchmark 2** — Open LLM Leaderboard (HF datasets-server), LMArena
|
||||
- **papers 49** — arxiv по 25+ разделам (cs.AI/LG/CL/CV/RO/IR/CR/SE/NE/MA/DC/DB/HC/SD/CY, stat.ML, eess.AS/SP/IV, math.OC, q-bio.NC, cs.GR, cs.AR, cs.PF), HF daily papers, OpenAlex, Semantic Scholar, Papers with Code, Connected Papers
|
||||
- **company 95** — ленты OpenAI, Google AI/Research/DeepMind, Microsoft Research, NVIDIA, Apple ML, Amazon Science, HF, Meta AI, Mistral, Cohere, Together, Perplexity, Groq, Cerebras, Stability, Databricks, Anyscale, Modal, W&B, LangChain, LlamaIndex, vLLM, Replicate, Fireworks, SambaNova, Scale, xAI, Salesforce, AWS ML, Azure AI, Anthropic, AI2, EleutherAI, LAION, Runway, ElevenLabs, Deepgram, AssemblyAI, Pinecone, Weaviate, Qdrant, Chroma, Unsloth, Axolotl
|
||||
- **company_cn 31** — Qwen, DeepSeek changelog, Moonshot, Zhipu, Baichuan, MiniMax, StepFun, 01.AI, InternLM, ChatGLM + GitHub-орги Tencent-Hunyuan, PaddlePaddle, bytedance, mindspore-ai, QwenLM, deepseek-ai, MoonshotAI, THUDM, OpenBMB, Shanghai-AI-Lab
|
||||
- **company_kr 10** — Kakao tech, Naver Cloud/D2, Samsung Research, LG AI Research, Upstage, Rebellions, Twelve Labs, FriendliAI
|
||||
- **community 49** — HN top/new/ask/show, Lobsters (4 тега), GeekNews, Reddit: LocalLLaMA, OpenAI, singularity, MachineLearning, deeplearning, netsec, programming, rust, golang, Python, webdev, selfhosted, devops, datascience, androiddev, javascript, linux, experienced_devs
|
||||
- **security 45** — THN, BleepingComputer, Krebs, SANS ISC, Packet Storm, CISA advisories + CISA KEV (JSON), NVD CVE (JSON), Exploit-DB, Project Zero, PortSwigger, GitHub Security, Rapid7, Qualys, Tenable, CrowdStrike, SentinelOne, Unit42, Talos, Securelist, WeLiveSecurity, Malwarebytes, Schneier, Graham Cluley, HackerOne hacktivity, Bugcrowd, Intigriti, YesWeHack, CTFtime, OWASP, SANS diaries, Risky Business, tl;dr sec, Detection Engineering
|
||||
- **forum 31** — 10 Discourse-инстансов (discuss.python, community.openai, forum.cursor, discuss.huggingface, forum.obsidian, community.cloudflare, discuss.elastic, forum.gitlab, community.home-assistant, discuss.grafana), XenForo BHW, веб-версии Telegram-каналов (durov, telegram, AI-каналы), Stack Overflow tags, Reddit-миграции
|
||||
- **models 18** — HF trending/new (модели, спейсы, датасеты), HF по 6 пайплайн-тегам, Ollama library + search, LM Studio catalog, OpenRouter models + pricing (JSON), Civitai, Replicate explore
|
||||
- **benchmark 10** — Open LLM Leaderboard, LMArena, Artificial Analysis, Epoch AI, SWE-bench, LiveBench, HELM, MTEB, BigCodeBench, Chatbot Arena
|
||||
|
||||
Адаптеры: `rss`, `json` (HF, HN, OpenAlex, S2, Discourse), `html` (список + детальная страница), `github` (org repos). Медленные хосты (reddit, github, HF) обходятся последовательно с паузой, остальные — в 8 потоков.
|
||||
Адаптеры: `rss`, `generic_json`, `or_models`, `cisa_kev`, `nvd`, `json` (HF, HN, OpenAlex, S2, Discourse), `html` (список + детальная), `github_org`. Медленные хосты (reddit, github, HF, arxiv, openrouter) обходятся последовательно с паузой, остальные — в 8 потоков. Для arxiv RSS, который часто отдаёт пустое, есть fallback на официальный API (`export.arxiv.org/api/query`).
|
||||
|
||||
## Панель и команды
|
||||
|
||||
```
|
||||
python -m omni serve --port 8077 # веб-панель + JSON API (без сборки, без npm)
|
||||
python -m omni feed --limit 10 # обход всего реестра
|
||||
python -m omni doctor --fix # проверить источники и подобрать рабочие фиды
|
||||
python -m omni digest --out data/digest.md
|
||||
```
|
||||
|
||||
Панель — две вкладки:
|
||||
|
||||
1. **лента** — счётчики (записей, с ценой, прогонов, источников, включённых), поиск по базе, фильтр по категориям, кнопка «собрать ленту» с прогрессом и разбивкой fetched/new/dup/empty.
|
||||
2. **источники** — менеджер реестра:
|
||||
- таблица всех 338 источников: переключатель вкл/выкл, категория, тип, url, статус (жив / мёртв / не проверен), число записей;
|
||||
- фильтр по имени/url и по категории;
|
||||
- **подключить источник** — форма (имя, url, категория): тип и адаптер определяются по url автоматически (`/rss` → rss, `.json`/`/api/` → generic_json, `github.com/org` → github_org, `t.me/s/` → tg_web), запись живёт в SQLite и сразу участвует в обходах;
|
||||
- **проверить всё / проверить категорию** — фоновой джоб: реальный фетч каждого источника, запись `status/items/note/checked`, прогресс `done/total, живых/мёртвых`;
|
||||
- **вкл все / выкл все** — массово по текущему фильтру;
|
||||
- удаление своих источников (реестровые только выключаются).
|
||||
|
||||
Выключенный источник не дёргается ни панелью, ни `omni feed`, ни `omni doctor` — фильтр применяется в `sources.pick()`.
|
||||
|
||||
API: `GET /api/stats`, `GET /api/items?category=&q=&limit=&offset=`, `GET /api/sources?category=&q=`,
|
||||
`POST /api/sources/toggle {name,enabled}`, `POST /api/sources/add {name,category,url}`,
|
||||
`POST /api/sources/remove {name}`, `POST /api/sources/check {name|names|category|all}`,
|
||||
`POST /api/feed {category,limit}`, `GET /api/job/<id>`.
|
||||
|
||||
Состояние источников — таблицы `source_flags` и `source_custom` в той же SQLite (`data/omni.sqlite3`).
|
||||
|
||||
## Как устроено
|
||||
|
||||
|
||||
21
omni/cli.py
21
omni/cli.py
@@ -149,12 +149,18 @@ def cmd_stats(args):
|
||||
|
||||
|
||||
def cmd_sources(args):
|
||||
from . import source_state
|
||||
st = sources.stats()
|
||||
print("источников:", len(sources.SOURCES), "| по категориям:",
|
||||
", ".join(f"{k}={v}" for k, v in st.items()))
|
||||
fl = source_state.flags()
|
||||
all_s = sources.all_sources()
|
||||
print("источников:", len(all_s), "| включено:", sum(1 for s in all_s if fl.get(s.name, {}).get("enabled", True)),
|
||||
"| по категориям:", ", ".join(f"{k}={v}" for k, v in st.items()))
|
||||
if args.list:
|
||||
for s in sources.pick(args.category, args.only, args.tags):
|
||||
print(f" {s.category:<12} {s.adapter:<18} {s.name:<26} {s.url[:70]}")
|
||||
for s in sources.pick(args.category, args.only, args.tags, respect_enabled=not args.all):
|
||||
f = fl.get(s.name, {})
|
||||
mark = "on " if f.get("enabled", True) else "off"
|
||||
stt = f"{f.get('items')} зап." if (f.get("items") or 0) > 0 else (f.get("note") or "—")
|
||||
print(f" {mark} {s.category:<12} {s.adapter:<18} {s.name:<26} {stt:<12} {s.url[:60]}")
|
||||
return 0
|
||||
|
||||
|
||||
@@ -256,6 +262,7 @@ def main(argv=None):
|
||||
sp.add_argument("--only", default=None)
|
||||
sp.add_argument("--tags", default=None)
|
||||
sp.add_argument("--list", action="store_true")
|
||||
sp.add_argument("--all", action="store_true", help="показать и выключенные")
|
||||
sp.set_defaults(fn=cmd_sources)
|
||||
|
||||
sp = sub.add_parser("feed", help="обход реестра источников без запроса")
|
||||
@@ -267,6 +274,12 @@ def main(argv=None):
|
||||
sp.add_argument("--verbose", action="store_true")
|
||||
sp.set_defaults(fn=cmd_feed)
|
||||
|
||||
sp = sub.add_parser("doctor", help="проверка источников, --fix подбирает рабочие фиды")
|
||||
sp.add_argument("--only", default=None)
|
||||
sp.add_argument("--fix", action="store_true")
|
||||
sp.add_argument("--limit", type=int, default=3)
|
||||
sp.set_defaults(fn=lambda a: (__import__("omni.doctor", fromlist=["run"]).run(a.only, a.fix, limit=a.limit), 0)[1])
|
||||
|
||||
sp = sub.add_parser("serve", help="веб-панель + JSON API")
|
||||
sp.add_argument("--port", type=int, default=8077)
|
||||
sp.set_defaults(fn=lambda a: __import__("omni.serve", fromlist=["serve"]).serve(a.port))
|
||||
|
||||
115
omni/doctor.py
Normal file
115
omni/doctor.py
Normal file
@@ -0,0 +1,115 @@
|
||||
"""Доктор источников: проверяет каждый, а с --fix сам находит рабочий фид по HTML-странице.
|
||||
|
||||
Причина: у половины блогов угаданный /rss.xml отдаёт 404, у части — Cloudflare.
|
||||
Для rss-источников лечим автопоиском <link rel="alternate" type="...rss..."> на главной,
|
||||
результат кладём в data/feeds_cache.json (переопределяет url без правки реестра).
|
||||
"""
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from . import feeds, sources
|
||||
|
||||
CACHE = pathlib.Path(__file__).resolve().parent.parent / "data" / "feeds_cache.json"
|
||||
|
||||
|
||||
def load_cache():
|
||||
try:
|
||||
return json.loads(CACHE.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def save_cache(d):
|
||||
CACHE.parent.mkdir(parents=True, exist_ok=True)
|
||||
CACHE.write_text(json.dumps(d, ensure_ascii=False, indent=1), encoding="utf-8")
|
||||
|
||||
|
||||
def discover(home):
|
||||
"""Ищем ссылки на фиды в HTML главной/раздела."""
|
||||
r = feeds.net.get(home, headers=feeds.UA)
|
||||
if r.status != 200 or not r.text:
|
||||
return []
|
||||
out = []
|
||||
soup = BeautifulSoup(r.text, "lxml")
|
||||
for ln in soup.find_all("link", href=True):
|
||||
typ = (ln.get("type") or "").lower()
|
||||
if "rss" in typ or "atom" in typ:
|
||||
out.append(urllib.parse.urljoin(home, str(ln["href"])))
|
||||
for a in soup.find_all("a", href=True):
|
||||
h = str(a["href"])
|
||||
if re.search(r"(rss|feed|atom)(\.xml|\.rss)?/?$", h) and "comment" not in h:
|
||||
out.append(urllib.parse.urljoin(home, h))
|
||||
seen, uniq = set(), []
|
||||
for u in out:
|
||||
if u not in seen:
|
||||
seen.add(u)
|
||||
uniq.append(u)
|
||||
return uniq[:8]
|
||||
|
||||
|
||||
def probe(src, limit=3):
|
||||
try:
|
||||
got = feeds.fetch(src, limit)
|
||||
return src.name, len(got), (got[0].title[:60] if got else "")
|
||||
except Exception as e:
|
||||
return src.name, 0, f"{type(e).__name__}"
|
||||
|
||||
|
||||
def fix(src):
|
||||
"""Подобрать рабочий фид: кандидаты с главной (и её подразделов)."""
|
||||
base = "{0.scheme}://{0.netloc}".format(urllib.parse.urlsplit(src.url))
|
||||
cands = []
|
||||
for home in (src.url, base + "/", base + "/blog", base + "/news", base + "/research"):
|
||||
try:
|
||||
cands += discover(home)
|
||||
except Exception:
|
||||
continue
|
||||
seen = set()
|
||||
for u in cands:
|
||||
if u in seen:
|
||||
continue
|
||||
seen.add(u)
|
||||
trial = sources.Source(src.name, src.category, src.kind, u, "rss", src.tags)
|
||||
try:
|
||||
got = feeds.fetch(trial, 3)
|
||||
except Exception:
|
||||
got = []
|
||||
if got:
|
||||
return u, len(got)
|
||||
return None, 0
|
||||
|
||||
|
||||
def run(only=None, do_fix=False, workers=8, limit=3):
|
||||
from . import source_state
|
||||
picked = sources.pick(None, only, None)
|
||||
cache = load_cache()
|
||||
results = []
|
||||
with ThreadPoolExecutor(max_workers=workers) as ex:
|
||||
for name, n, extra in ex.map(lambda s: probe(s, limit), picked):
|
||||
results.append({"name": name, "n": n, "extra": extra})
|
||||
source_state.record_check(name, 200 if n else 0, n, "ok" if n else str(extra)[:120])
|
||||
bad = [r for r in results if r["n"] == 0]
|
||||
print(f"проверено {len(results)} | живых {len(results) - len(bad)} | пустых {len(bad)}")
|
||||
if do_fix:
|
||||
by_name = {s.name: s for s in picked}
|
||||
fixed = 0
|
||||
for r in bad:
|
||||
src = by_name[r["name"]]
|
||||
if src.adapter not in ("rss", "html_listing", "github_org"):
|
||||
continue
|
||||
url, n = fix(src)
|
||||
if url:
|
||||
cache[src.name] = {"url": url, "adapter": "rss", "n": n}
|
||||
fixed += 1
|
||||
print(f" фикс {src.name:24s} -> {url} ({n})")
|
||||
save_cache(cache)
|
||||
print(f"починено {fixed}, кэш: {len(cache)} записей -> {CACHE}")
|
||||
else:
|
||||
for r in bad:
|
||||
print(f" пусто {r['name']:26s} {r['extra']}")
|
||||
return results, cache
|
||||
134
omni/feeds.py
134
omni/feeds.py
@@ -60,8 +60,24 @@ def _arxiv_api(src, limit):
|
||||
return out
|
||||
|
||||
|
||||
_FEED_CACHE = None
|
||||
|
||||
|
||||
def _cache_url(src):
|
||||
global _FEED_CACHE
|
||||
if _FEED_CACHE is None:
|
||||
try:
|
||||
import pathlib as _pl
|
||||
f = _pl.Path(__file__).resolve().parent.parent / "data" / "feeds_cache.json"
|
||||
_FEED_CACHE = __import__("json").loads(f.read_text(encoding="utf-8")) if f.exists() else {}
|
||||
except Exception:
|
||||
_FEED_CACHE = {}
|
||||
hit = _FEED_CACHE.get(src.name)
|
||||
return hit.get("url") if isinstance(hit, dict) and hit.get("url") else src.url
|
||||
|
||||
|
||||
def _rss(src, limit):
|
||||
r = net.get(src.url, headers=UA)
|
||||
r = net.get(_cache_url(src), headers=UA)
|
||||
if r.status in (429, 503):
|
||||
import time
|
||||
time.sleep(2.5)
|
||||
@@ -387,8 +403,124 @@ def _html_listing(src, limit):
|
||||
return out
|
||||
|
||||
|
||||
def _json_of(url, timeout=35):
|
||||
"""JSON без зависимостей от конкретного клиента: net.get + json.loads."""
|
||||
try:
|
||||
r = net.get(url, json_mode=True, timeout=timeout)
|
||||
except Exception:
|
||||
return None
|
||||
if getattr(r, "status", 0) != 200 or not (r.text or "").strip():
|
||||
return None
|
||||
try:
|
||||
return json.loads(r.text)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _generic_json(src, limit):
|
||||
"""Любой JSON-эндпоинт: ищем список объектов и вытаскиваем title/url по частым ключам."""
|
||||
d = _json_of(src.url)
|
||||
rows = []
|
||||
if isinstance(d, list):
|
||||
rows = d
|
||||
elif isinstance(d, dict):
|
||||
for key in ("data", "items", "results", "papers", "models", "hits", "notes", "posts", "records"):
|
||||
v = d.get(key)
|
||||
if isinstance(v, list):
|
||||
rows = v
|
||||
break
|
||||
if not rows:
|
||||
for v in d.values():
|
||||
if isinstance(v, list) and v and isinstance(v[0], dict):
|
||||
rows = v
|
||||
break
|
||||
out = []
|
||||
for r in rows[:limit]:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
title = r.get("title") or r.get("name") or r.get("paper_title") or r.get("full_name") or r.get("id")
|
||||
url = r.get("url") or r.get("link") or r.get("html_url") or r.get("paper_url") or r.get("id")
|
||||
if isinstance(url, str) and url and not url.startswith("http"):
|
||||
if "github.com" in src.url:
|
||||
url = "https://github.com/" + url.lstrip("/")
|
||||
elif "huggingface.co" in src.url:
|
||||
url = "https://huggingface.co/" + url.lstrip("/")
|
||||
text = r.get("abstract") or r.get("summary") or r.get("description") or r.get("body") or r.get("content") or ""
|
||||
if isinstance(text, dict):
|
||||
text = json.dumps(text, ensure_ascii=False)[:400]
|
||||
published = r.get("published") or r.get("created") or r.get("created_at") or r.get("date") or ""
|
||||
if not title:
|
||||
continue
|
||||
out.append(_item(src, "entry", str(url or ""), str(title), str(text)[:800],
|
||||
str(r.get("author") or r.get("owner") or ""), str(published)[:24]))
|
||||
return out
|
||||
|
||||
|
||||
def _or_models(src, limit):
|
||||
d = (_json_of(src.url) or {}).get("data", [])
|
||||
out = []
|
||||
for m in d[:limit]:
|
||||
mid = m.get("id") or ""
|
||||
pricing = m.get("pricing") or {}
|
||||
price = None
|
||||
try:
|
||||
price = round(float(pricing.get("prompt", 0)) * 1_000_000, 3)
|
||||
except Exception:
|
||||
price = None
|
||||
name = m.get("name") or mid
|
||||
ctx = m.get("context_length") or 0
|
||||
out.append(_item(src, "model", f"https://openrouter.ai/models/{mid}", f"{name}",
|
||||
f"context {ctx} · prompt ${price}/1M", "", str(m.get("created") or ""),
|
||||
extra={"price": price}))
|
||||
return out
|
||||
|
||||
|
||||
def _cisa_kev(src, limit):
|
||||
d = _json_of(src.url, timeout=45) or {}
|
||||
vulns = d.get("vulnerabilities", [])
|
||||
vulns = sorted(vulns, key=lambda v: v.get("dateAdded") or "", reverse=True)
|
||||
out = []
|
||||
for v in vulns[:limit]:
|
||||
cve = v.get("cveID") or ""
|
||||
out.append(_item(src, "cve", f"https://nvd.nist.gov/vuln/detail/{cve}",
|
||||
f"{cve} · {v.get('vendorProject','')} {v.get('product','')}".strip(),
|
||||
(v.get("shortDescription") or "")[:600],
|
||||
v.get("vendorProject") or "", v.get("dateAdded") or ""))
|
||||
return out
|
||||
|
||||
|
||||
def _nvd(src, limit):
|
||||
d = _json_of(src.url, timeout=45) or {}
|
||||
out = []
|
||||
for v in d.get("vulnerabilities", [])[:limit]:
|
||||
c = v.get("cve", {})
|
||||
cid = c.get("id") or ""
|
||||
desc = ""
|
||||
for dd in c.get("descriptions", []):
|
||||
if dd.get("lang") == "en":
|
||||
desc = dd.get("value") or ""
|
||||
break
|
||||
sev = ""
|
||||
metrics = c.get("metrics", {})
|
||||
for key in ("cvssMetricV31", "cvssMetricV40", "cvssMetricV2"):
|
||||
if metrics.get(key):
|
||||
try:
|
||||
sev = str(metrics[key][0]["cvssData"].get("baseScore"))
|
||||
except Exception:
|
||||
sev = ""
|
||||
break
|
||||
out.append(_item(src, "cve", f"https://nvd.nist.gov/vuln/detail/{cid}",
|
||||
f"{cid}{(' · CVSS ' + sev) if sev else ''}", desc[:600],
|
||||
"", c.get("published") or ""))
|
||||
return out
|
||||
|
||||
|
||||
ADAPTERS = {
|
||||
"rss": _rss,
|
||||
"generic_json": _generic_json,
|
||||
"or_models": _or_models,
|
||||
"cisa_kev": _cisa_kev,
|
||||
"nvd": _nvd,
|
||||
"hn_top": _hn_top,
|
||||
"reddit_listing": _reddit_listing,
|
||||
"hf_trending": _hf_generic,
|
||||
|
||||
146
omni/serve.py
146
omni/serve.py
@@ -6,31 +6,76 @@ import json
|
||||
import pathlib
|
||||
import threading
|
||||
import urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
from . import feeds, sources, store
|
||||
from . import feeds, net, sources, source_state, store
|
||||
|
||||
WEB = pathlib.Path(__file__).resolve().parent.parent / "web"
|
||||
JOBS = {}
|
||||
_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _job(jid, **kw):
|
||||
with _LOCK:
|
||||
cur = JOBS.get(jid) or {}
|
||||
cur.update(kw)
|
||||
JOBS[jid] = cur
|
||||
|
||||
|
||||
def _run_feed(job_id, category, limit, tags):
|
||||
picked = sources.pick(category, None, tags)
|
||||
try:
|
||||
items, report = feeds.fetch_many(picked, limit=limit, workers=8)
|
||||
new, dup = store.add(items)
|
||||
with _LOCK:
|
||||
JOBS[job_id] = {"state": "done", "sources": len(picked), "fetched": len(items),
|
||||
"new": new, "dup": dup, "empty": sum(1 for _, c in report if not c),
|
||||
"report": sorted(report, key=lambda x: -x[1])[:40]}
|
||||
_job(job_id, state="done", sources=len(picked), fetched=len(items), new=new, dup=dup,
|
||||
empty=sum(1 for _, c in report if not c),
|
||||
report=sorted(report, key=lambda x: -x[1])[:40])
|
||||
except Exception as e:
|
||||
with _LOCK:
|
||||
JOBS[job_id] = {"state": "error", "error": f"{type(e).__name__}: {e}"[:200]}
|
||||
_job(job_id, state="error", error=f"{type(e).__name__}: {e}"[:200])
|
||||
|
||||
|
||||
def _check_one(src, limit=5):
|
||||
note = ""
|
||||
try:
|
||||
items = feeds.fetch(src, limit)
|
||||
except Exception as e:
|
||||
items, note = [], type(e).__name__
|
||||
if items:
|
||||
return source_state.record_check(src.name, 200, len(items), "ok")
|
||||
code = 0
|
||||
try:
|
||||
r = net.get(src.url, timeout=20)
|
||||
code = getattr(r, "status", 0) or 0
|
||||
note = note or ("пусто" if code == 200 else f"http {code}")
|
||||
except Exception as e:
|
||||
note = note or type(e).__name__
|
||||
return source_state.record_check(src.name, code, 0, note or "нет данных")
|
||||
|
||||
|
||||
def _run_check(job_id, names=None, category=None):
|
||||
picked = sources.pick(category, ",".join(names) if names else None, None)
|
||||
total = len(picked)
|
||||
_job(job_id, state="running", total=total, done=0, alive=0, dead=0, current="")
|
||||
alive = dead = 0
|
||||
with ThreadPoolExecutor(max_workers=6) as ex:
|
||||
futures = {ex.submit(_check_one, s): s for s in picked}
|
||||
for i, f in enumerate(futures, 1):
|
||||
s = futures[f]
|
||||
try:
|
||||
res = f.result()
|
||||
except Exception:
|
||||
res = {}
|
||||
if (res.get("items") or 0) > 0:
|
||||
alive += 1
|
||||
else:
|
||||
dead += 1
|
||||
_job(job_id, done=i, alive=alive, dead=dead, current=s.name)
|
||||
_job(job_id, state="done", done=total, alive=alive, dead=dead, current="")
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
server_version = "omni/1.0"
|
||||
server_version = "omni/1.1"
|
||||
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
@@ -60,17 +105,44 @@ class Handler(BaseHTTPRequestHandler):
|
||||
|
||||
if path == "/api/stats":
|
||||
s = store.stats()
|
||||
return self._json({"items": s["items"], "priced": s["priced"], "runs": s["runs"],
|
||||
"runs_ok": s["runs_ok"],
|
||||
fl = source_state.flags()
|
||||
all_s = sources.all_sources()
|
||||
return self._json({
|
||||
"items": s["items"], "priced": s["priced"], "runs": s["runs"], "runs_ok": s["runs_ok"],
|
||||
"by_provider": [{"name": n, "count": c} for n, c in s["by_provider"]],
|
||||
"by_kind": [{"name": n, "count": c} for n, c in s["by_kind"]],
|
||||
"sources": len(sources.SOURCES), "categories": sources.stats()})
|
||||
"sources": len(all_s),
|
||||
"sources_enabled": sum(1 for x in all_s if fl.get(x.name, {}).get("enabled", True)),
|
||||
"categories": sources.stats(),
|
||||
})
|
||||
|
||||
if path == "/api/sources":
|
||||
return self._json({"total": len(sources.SOURCES), "categories": sources.stats(),
|
||||
"items": [{"name": s.name, "category": s.category, "kind": s.kind,
|
||||
"adapter": s.adapter, "url": s.url, "tags": list(s.tags)}
|
||||
for s in sources.SOURCES]})
|
||||
cat = (q.get("category") or [None])[0]
|
||||
query = (q.get("q") or [""])[0].strip().lower()
|
||||
fl = source_state.flags()
|
||||
custom_names = {d["name"] for d in source_state.custom()}
|
||||
items = []
|
||||
for s in sources.all_sources():
|
||||
f = fl.get(s.name, {})
|
||||
if cat and cat not in ("all", "") and s.category != cat:
|
||||
continue
|
||||
if query and query not in s.name.lower() and query not in s.url.lower():
|
||||
continue
|
||||
items.append({
|
||||
"name": s.name, "category": s.category, "kind": s.kind, "adapter": s.adapter,
|
||||
"url": s.url, "tags": list(s.tags),
|
||||
"enabled": f.get("enabled", True), "status": f.get("status"),
|
||||
"items": f.get("items"), "note": f.get("note"), "checked": f.get("checked"),
|
||||
"custom": s.name in custom_names,
|
||||
})
|
||||
return self._json({
|
||||
"total": len(sources.all_sources()), "shown": len(items),
|
||||
"enabled": sum(1 for x in items if x["enabled"]),
|
||||
"alive": sum(1 for x in items if (x["items"] or 0) > 0),
|
||||
"dead": sum(1 for x in items if x["checked"] and not (x["items"] or 0)),
|
||||
"untested": sum(1 for x in items if not x["checked"]),
|
||||
"categories": sources.stats(), "items": items,
|
||||
})
|
||||
|
||||
if path == "/api/items":
|
||||
cat = (q.get("category") or [None])[0]
|
||||
@@ -95,17 +167,49 @@ class Handler(BaseHTTPRequestHandler):
|
||||
body = json.loads(self.rfile.read(n) or b"{}")
|
||||
except Exception:
|
||||
body = {}
|
||||
|
||||
if u.path == "/api/feed":
|
||||
jid = f"job{len(JOBS) + 1}"
|
||||
with _LOCK:
|
||||
JOBS[jid] = {"state": "running"}
|
||||
threading.Thread(target=_run_feed, args=(jid, body.get("category"), int(body.get("limit") or 12),
|
||||
body.get("tags")), daemon=True).start()
|
||||
jid = f"feed{len(JOBS)}"
|
||||
_job(jid, state="running")
|
||||
threading.Thread(target=_run_feed,
|
||||
args=(jid, body.get("category"), int(body.get("limit") or 12), body.get("tags")),
|
||||
daemon=True).start()
|
||||
return self._json({"job": jid})
|
||||
|
||||
if u.path == "/api/sources/toggle":
|
||||
name = body.get("name")
|
||||
if not name:
|
||||
return self._json({"error": "name required"}, 400)
|
||||
return self._json(source_state.set_enabled(name, bool(body.get("enabled", True))))
|
||||
|
||||
if u.path == "/api/sources/add":
|
||||
try:
|
||||
res = source_state.add_custom(body.get("name"), body.get("category"), body.get("url"),
|
||||
body.get("kind"), body.get("adapter"),
|
||||
body.get("tags") or (), int(body.get("limit") or 20),
|
||||
bool(body.get("detail")))
|
||||
except Exception as e:
|
||||
return self._json({"error": str(e)}, 400)
|
||||
return self._json(res)
|
||||
|
||||
if u.path == "/api/sources/remove":
|
||||
name = body.get("name")
|
||||
if not name:
|
||||
return self._json({"error": "name required"}, 400)
|
||||
return self._json(source_state.remove_custom(name))
|
||||
|
||||
if u.path == "/api/sources/check":
|
||||
names = body.get("names") or ([body["name"]] if body.get("name") else None)
|
||||
category = None if body.get("all") else body.get("category")
|
||||
jid = f"check{len(JOBS)}"
|
||||
_job(jid, state="running")
|
||||
threading.Thread(target=_run_check, args=(jid, names, category), daemon=True).start()
|
||||
return self._json({"job": jid})
|
||||
|
||||
return self._json({"error": "not found"}, 404)
|
||||
|
||||
|
||||
def serve(port=8077):
|
||||
srv = ThreadingHTTPServer(("127.0.0.1", port), Handler)
|
||||
print(f"omni ui: http://127.0.0.1:{port} (источников {len(sources.SOURCES)})")
|
||||
print(f"omni ui: http://127.0.0.1:{port} (источников {len(sources.all_sources())})")
|
||||
srv.serve_forever()
|
||||
131
omni/source_state.py
Normal file
131
omni/source_state.py
Normal file
@@ -0,0 +1,131 @@
|
||||
"""Состояние источников: включён/выключен, пользовательские источники, результаты проверок.
|
||||
|
||||
Две таблицы в той же SQLite, что и записи:
|
||||
source_flags (name, enabled, status, items, note, checked)
|
||||
source_custom (name, category, kind, url, adapter, tags, limit, detail, added)
|
||||
"""
|
||||
import json
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
from . import config
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS source_flags (
|
||||
name TEXT PRIMARY KEY,
|
||||
enabled INTEGER DEFAULT 1,
|
||||
status INTEGER,
|
||||
items INTEGER,
|
||||
note TEXT,
|
||||
checked INTEGER
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS source_custom (
|
||||
name TEXT PRIMARY KEY,
|
||||
category TEXT, kind TEXT, url TEXT, adapter TEXT,
|
||||
tags TEXT, lim INTEGER DEFAULT 20, detail INTEGER DEFAULT 0, added INTEGER
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
def conn():
|
||||
c = sqlite3.connect(str(config.DB), timeout=30)
|
||||
c.execute("PRAGMA journal_mode=WAL")
|
||||
c.executescript(SCHEMA)
|
||||
return c
|
||||
|
||||
|
||||
def flags():
|
||||
"""{name: {enabled, status, items, note, checked}}"""
|
||||
with conn() as c:
|
||||
rows = c.execute("SELECT name, enabled, status, items, note, checked FROM source_flags").fetchall()
|
||||
return {r[0]: {"enabled": bool(r[1]), "status": r[2], "items": r[3], "note": r[4], "checked": r[5]}
|
||||
for r in rows}
|
||||
|
||||
|
||||
def set_enabled(name, enabled):
|
||||
now = int(time.time())
|
||||
with conn() as c:
|
||||
c.execute("INSERT INTO source_flags(name, enabled, checked) VALUES(?,?,?) "
|
||||
"ON CONFLICT(name) DO UPDATE SET enabled=excluded.enabled",
|
||||
(name, 1 if enabled else 0, now))
|
||||
return {"name": name, "enabled": bool(enabled)}
|
||||
|
||||
|
||||
def record_check(name, status, items, note=""):
|
||||
now = int(time.time())
|
||||
with conn() as c:
|
||||
c.execute("INSERT INTO source_flags(name, enabled, status, items, note, checked) VALUES(?,1,?,?,?,?) "
|
||||
"ON CONFLICT(name) DO UPDATE SET status=excluded.status, items=excluded.items, "
|
||||
"note=excluded.note, checked=excluded.checked",
|
||||
(name, status, items, str(note)[:160], now))
|
||||
return {"name": name, "status": status, "items": items, "note": str(note)[:160], "checked": now}
|
||||
|
||||
|
||||
def custom():
|
||||
with conn() as c:
|
||||
rows = c.execute("SELECT name, category, kind, url, adapter, tags, lim, detail, added "
|
||||
"FROM source_custom ORDER BY added DESC").fetchall()
|
||||
out = []
|
||||
for r in rows:
|
||||
out.append({"name": r[0], "category": r[1], "kind": r[2], "url": r[3], "adapter": r[4],
|
||||
"tags": tuple(json.loads(r[5] or "[]")), "limit": r[6] or 20,
|
||||
"detail": bool(r[7]), "extra": {}, "added": r[8]})
|
||||
return out
|
||||
|
||||
|
||||
def add_custom(name, category, url, kind=None, adapter=None, tags=(), limit=20, detail=False):
|
||||
name = (name or "").strip()
|
||||
url = (url or "").strip()
|
||||
if not name or not url:
|
||||
raise ValueError("нужны name и url")
|
||||
if not url.startswith("http"):
|
||||
raise ValueError("url должен начинаться с http")
|
||||
kind, adapter = kind or guess_kind(url), adapter or guess_adapter(url, kind)
|
||||
with conn() as c:
|
||||
c.execute("INSERT OR REPLACE INTO source_custom(name, category, kind, url, adapter, tags, lim, detail, added) "
|
||||
"VALUES(?,?,?,?,?,?,?,?,?)",
|
||||
(name, category or "custom", kind, url, adapter,
|
||||
json.dumps(list(tags), ensure_ascii=False), int(limit), 1 if detail else 0, int(time.time())))
|
||||
return {"name": name, "category": category or "custom", "kind": kind, "url": url,
|
||||
"adapter": adapter, "tags": list(tags), "limit": limit, "detail": bool(detail)}
|
||||
|
||||
|
||||
def remove_custom(name):
|
||||
with conn() as c:
|
||||
c.execute("DELETE FROM source_custom WHERE name=?", (name,))
|
||||
c.execute("DELETE FROM source_flags WHERE name=?", (name,))
|
||||
return {"removed": name}
|
||||
|
||||
|
||||
def guess_kind(url):
|
||||
u = url.lower().rstrip("/")
|
||||
path = u.split("://", 1)[-1]
|
||||
path = path[path.find("/"):] if "/" in path else ""
|
||||
last = path.rsplit("/", 1)[-1]
|
||||
if "github.com" in u and u.count("/") <= 4:
|
||||
return "github"
|
||||
if last in ("rss", "feed", "atom", "rss.xml", "feed.xml", "atom.xml", "index.xml", "rss2"):
|
||||
return "rss"
|
||||
if u.endswith(".json") or "/latest.json" in u or "api." in u or "/api/" in u:
|
||||
return "json"
|
||||
if any(x in u for x in (".rss", "rss.xml", "/feed", ".atom", "/rss/", "atom.xml")):
|
||||
return "rss"
|
||||
return "html"
|
||||
|
||||
|
||||
def guess_adapter(url, kind=None):
|
||||
u = url.lower()
|
||||
kind = kind or guess_kind(url)
|
||||
if "github.com" in u:
|
||||
return "github_org"
|
||||
if "discourse" in u or u.endswith("/latest.json"):
|
||||
return "discourse"
|
||||
if kind == "json":
|
||||
return "generic_json"
|
||||
if u.endswith(".json") or "api." in u:
|
||||
return "generic_json"
|
||||
if "t.me/s/" in u:
|
||||
return "tg_web"
|
||||
if kind == "rss":
|
||||
return "rss"
|
||||
return "html_listing"
|
||||
@@ -10,6 +10,8 @@
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .sources_ext import EXTRA
|
||||
|
||||
|
||||
@dataclass
|
||||
class Source:
|
||||
@@ -26,7 +28,7 @@ class Source:
|
||||
|
||||
S = Source
|
||||
|
||||
SOURCES = [
|
||||
SOURCES_BASE = [
|
||||
# ---------- papers ----------
|
||||
S("arxiv_cs_ai", "papers", "rss", "https://rss.arxiv.org/rss/cs.AI", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_lg", "papers", "rss", "https://rss.arxiv.org/rss/cs.LG", "rss", ("paper", "arxiv")),
|
||||
@@ -175,12 +177,34 @@ SOURCES = [
|
||||
S("tg_huggingface", "forum", "html", "https://t.me/s/huggingface", "tg_web", ("telegram",)),
|
||||
]
|
||||
|
||||
SOURCES = SOURCES_BASE + [Source(**d) for d in EXTRA]
|
||||
|
||||
BY_NAME = {s.name: s for s in SOURCES}
|
||||
CATEGORIES = sorted({s.category for s in SOURCES})
|
||||
|
||||
|
||||
def pick(category=None, only=None, tags=None):
|
||||
out = SOURCES
|
||||
def all_sources():
|
||||
"""Реестр + пользовательские источники из панели."""
|
||||
from . import source_state
|
||||
extra = []
|
||||
for d in source_state.custom():
|
||||
if d["name"] in BY_NAME:
|
||||
continue
|
||||
d = {k: v for k, v in d.items() if k in
|
||||
("name", "category", "kind", "url", "adapter", "tags", "limit", "detail", "extra")}
|
||||
extra.append(Source(**d))
|
||||
return SOURCES + extra
|
||||
|
||||
|
||||
def enabled_sources():
|
||||
"""Только включённые: выключенные в панели источники не дёргаем."""
|
||||
from . import source_state
|
||||
fl = source_state.flags()
|
||||
return [s for s in all_sources() if fl.get(s.name, {}).get("enabled", True)]
|
||||
|
||||
|
||||
def pick(category=None, only=None, tags=None, respect_enabled=True):
|
||||
out = enabled_sources() if respect_enabled else all_sources()
|
||||
if category:
|
||||
cats = {c.strip() for c in category.split(",") if c.strip()}
|
||||
out = [s for s in out if s.category in cats]
|
||||
@@ -193,8 +217,8 @@ def pick(category=None, only=None, tags=None):
|
||||
return out
|
||||
|
||||
|
||||
def stats():
|
||||
def stats(use_all=True):
|
||||
out = {}
|
||||
for s in SOURCES:
|
||||
for s in (all_sources() if use_all else SOURCES):
|
||||
out.setdefault(s.category, []).append(s.name)
|
||||
return {k: len(v) for k, v in sorted(out.items())}
|
||||
290
omni/sources_ext.py
Normal file
290
omni/sources_ext.py
Normal file
@@ -0,0 +1,290 @@
|
||||
"""Расширение реестра: ещё ~230 источников.
|
||||
|
||||
Правило: только публичные точки входа без ключей. Всё, что требует токен
|
||||
(Exa, You, Firecrawl, X), живёт в providers/ и включается через .env.
|
||||
|
||||
Здесь только данные — источниками они становятся в sources.py.
|
||||
"""
|
||||
|
||||
|
||||
def S(name, category, kind, url, adapter="rss", tags=(), limit=20, detail=False, extra=None):
|
||||
return {"name": name, "category": category, "kind": kind, "url": url, "adapter": adapter,
|
||||
"tags": tuple(tags), "limit": limit, "detail": detail, "extra": extra or {}}
|
||||
|
||||
|
||||
EXTRA = [
|
||||
# ---------------- papers ----------------
|
||||
S("arxiv_cs_cy", "papers", "rss", "https://rss.arxiv.org/rss/cs.CY", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_gt", "papers", "rss", "https://rss.arxiv.org/rss/cs.GT", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_sd", "papers", "rss", "https://rss.arxiv.org/rss/cs.SD", "rss", ("paper", "arxiv", "audio")),
|
||||
S("arxiv_cs_mm", "papers", "rss", "https://rss.arxiv.org/rss/cs.MM", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_ni", "papers", "rss", "https://rss.arxiv.org/rss/cs.NI", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_ds", "papers", "rss", "https://rss.arxiv.org/rss/cs.DS", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_si", "papers", "rss", "https://rss.arxiv.org/rss/cs.SI", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_ar", "papers", "rss", "https://rss.arxiv.org/rss/cs.AR", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_et", "papers", "rss", "https://rss.arxiv.org/rss/cs.ET", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_ms", "papers", "rss", "https://rss.arxiv.org/rss/cs.MS", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_pl", "papers", "rss", "https://rss.arxiv.org/rss/cs.PL", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_lo", "papers", "rss", "https://rss.arxiv.org/rss/cs.LO", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_fl", "papers", "rss", "https://rss.arxiv.org/rss/cs.FL", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_gr", "papers", "rss", "https://rss.arxiv.org/rss/cs.GR", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_sc", "papers", "rss", "https://rss.arxiv.org/rss/cs.SC", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_cs_os", "papers", "rss", "https://rss.arxiv.org/rss/cs.OS", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_eess_sp", "papers", "rss", "https://rss.arxiv.org/rss/eess.SP", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_eess_iv", "papers", "rss", "https://rss.arxiv.org/rss/eess.IV", "rss", ("paper", "arxiv")),
|
||||
S("arxiv_math_oc", "papers", "rss", "https://rss.arxiv.org/rss/math.OC", "rss", ("paper", "arxiv", "math")),
|
||||
S("arxiv_q_bio_nc", "papers", "rss", "https://rss.arxiv.org/rss/q-bio.NC", "rss", ("paper", "arxiv", "bio")),
|
||||
S("arxiv_quant_ph", "papers", "rss", "https://rss.arxiv.org/rss/quant-ph", "rss", ("paper", "arxiv", "quantum")),
|
||||
S("arxiv_stat_ap", "papers", "rss", "https://rss.arxiv.org/rss/stat.AP", "rss", ("paper", "arxiv")),
|
||||
S("openalex_llm", "papers", "json",
|
||||
"https://api.openalex.org/works?search=large%20language%20model&sort=publication_date:desc&per_page=25",
|
||||
"openalex", ("paper", "openalex", "llm")),
|
||||
S("openalex_agents", "papers", "json",
|
||||
"https://api.openalex.org/works?search=llm%20agent&sort=publication_date:desc&per_page=25",
|
||||
"openalex", ("paper", "openalex", "agents")),
|
||||
S("s2_agents", "papers", "json",
|
||||
"https://api.semanticscholar.org/graph/v1/paper/search?query=llm+agent+tool+use&limit=20&fields=title,abstract,url,venue,year,citationCount",
|
||||
"s2", ("paper", "semanticscholar", "agents")),
|
||||
S("s2_security", "papers", "json",
|
||||
"https://api.semanticscholar.org/graph/v1/paper/search?query=prompt+injection+llm+security&limit=20&fields=title,abstract,url,venue,year,citationCount",
|
||||
"s2", ("paper", "semanticscholar", "security")),
|
||||
S("biorxiv_ai", "papers", "rss", "https://connect.biorxiv.org/biorxiv_xml.php?subject=bioinformatics", "rss", ("paper", "bio")),
|
||||
S("medrxiv_ai", "papers", "rss", "https://connect.medrxiv.org/medrxiv_xml.php?subject=all", "rss", ("paper", "med")),
|
||||
S("paperswithcode_trending", "papers", "json", "https://paperswithcode.com/api/v1/papers/?ordering=-published&items_per_page=25",
|
||||
"generic_json", ("paper", "pwc")),
|
||||
S("openreview_recent", "papers", "json", "https://api2.openreview.net/notes?content.venueid=ICLR.cc/2026/Conference&limit=25",
|
||||
"generic_json", ("paper", "openreview")),
|
||||
S("acl_anthology", "papers", "rss", "https://aclanthology.org/feed.xml", "rss", ("paper", "nlp")),
|
||||
|
||||
# ---------------- models ----------------
|
||||
S("hf_models_textgen", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=text-generation&sort=trendingScore&direction=-1&limit=25",
|
||||
"hf_models", ("model", "hf")),
|
||||
S("hf_models_image", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=text-to-image&sort=trendingScore&direction=-1&limit=20",
|
||||
"hf_models", ("model", "hf", "image")),
|
||||
S("hf_models_vlm", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=image-text-to-text&sort=trendingScore&direction=-1&limit=20",
|
||||
"hf_models", ("model", "hf", "vlm")),
|
||||
S("hf_models_asr", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=automatic-speech-recognition&sort=trendingScore&direction=-1&limit=20",
|
||||
"hf_models", ("model", "hf", "audio")),
|
||||
S("hf_models_tts", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=text-to-speech&sort=trendingScore&direction=-1&limit=20",
|
||||
"hf_models", ("model", "hf", "audio")),
|
||||
S("hf_models_embed", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=feature-extraction&sort=trendingScore&direction=-1&limit=20",
|
||||
"hf_models", ("model", "hf", "embed")),
|
||||
S("hf_models_rerank", "models", "json",
|
||||
"https://huggingface.co/api/models?pipeline_tag=text-ranking&sort=trendingScore&direction=-1&limit=15",
|
||||
"hf_models", ("model", "hf", "rerank")),
|
||||
S("hf_datasets_new", "models", "json",
|
||||
"https://huggingface.co/api/datasets?sort=createdAt&direction=-1&limit=20", "hf_generic", ("dataset", "hf")),
|
||||
S("hf_spaces_new", "models", "json",
|
||||
"https://huggingface.co/api/spaces?sort=createdAt&direction=-1&limit=20", "hf_generic", ("space", "hf")),
|
||||
S("openrouter_models", "models", "json", "https://openrouter.ai/api/v1/models", "or_models", ("model", "openrouter")),
|
||||
S("civitai_models", "models", "json", "https://civitai.com/api/v1/models?limit=20&sort=Newest", "generic_json", ("model", "image")),
|
||||
S("lmstudio_catalog", "models", "html", "https://lmstudio.ai/models", "html_listing", ("model", "local"), detail=False),
|
||||
S("ollama_search", "models", "json", "https://ollama.com/search?o=newest", "html_listing", ("model", "local"),
|
||||
detail=False, extra={"include_contains": ["/library/"]}),
|
||||
|
||||
# ---------------- benchmark ----------------
|
||||
S("lmarena_text", "benchmark", "html", "https://lmarena.ai/leaderboard/text", "html_listing", ("benchmark", "arena"), detail=False),
|
||||
S("lmarena_vision", "benchmark", "html", "https://lmarena.ai/leaderboard/vision", "html_listing", ("benchmark", "arena"), detail=False),
|
||||
S("lmarena_webdev", "benchmark", "html", "https://lmarena.ai/leaderboard/webdev", "html_listing", ("benchmark", "arena"), detail=False),
|
||||
S("artificial_analysis", "benchmark", "html", "https://artificialanalysis.ai/leaderboards/models", "html_listing",
|
||||
("benchmark",), detail=False),
|
||||
S("swebench_board", "benchmark", "html", "https://www.swebench.com/", "html_listing", ("benchmark", "code"), detail=False),
|
||||
S("epoch_benchmarks", "benchmark", "html", "https://epoch.ai/benchmarks", "html_listing", ("benchmark",), detail=False),
|
||||
S("vellum_board", "benchmark", "html", "https://www.vellum.ai/llm-leaderboard", "html_listing", ("benchmark",), detail=False),
|
||||
S("hf_openllm_v2", "benchmark", "json",
|
||||
"https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard/contents&config=default&split=train&offset=25&length=25",
|
||||
"hf_rows", ("benchmark", "leaderboard")),
|
||||
|
||||
# ---------------- community ----------------
|
||||
S("hn_best", "community", "json", "https://hacker-news.firebaseio.com/v0/beststories.json", "hn_top", ("hn",)),
|
||||
S("hn_ask", "community", "json", "https://hacker-news.firebaseio.com/v0/askstories.json", "hn_top", ("hn", "ask")),
|
||||
S("hn_show", "community", "json", "https://hacker-news.firebaseio.com/v0/showstories.json", "hn_top", ("hn", "show")),
|
||||
S("hn_frontpage_rss", "community", "rss", "https://hnrss.org/frontpage", "rss", ("hn",)),
|
||||
S("hn_ai_search", "community", "rss", "https://hnrss.org/newest?q=LLM+OR+AI+OR+agent&points=10", "rss", ("hn", "ai")),
|
||||
S("lobsters_privacy", "community", "rss", "https://lobste.rs/t/privacy.rss", "rss", ("lobsters", "privacy")),
|
||||
S("lobsters_programming", "community", "rss", "https://lobste.rs/t/programming.rss", "rss", ("lobsters",)),
|
||||
S("lobsters_devops", "community", "rss", "https://lobste.rs/t/devops.rss", "rss", ("lobsters", "devops")),
|
||||
S("tildes", "community", "html", "https://tildes.net/", "html_listing", ("community",), detail=False),
|
||||
S("devto_ai", "community", "rss", "https://dev.to/feed/tag/ai", "rss", ("devto", "ai")),
|
||||
S("devto_llm", "community", "rss", "https://dev.to/feed/tag/llm", "rss", ("devto", "llm")),
|
||||
S("devto_python", "community", "rss", "https://dev.to/feed/tag/python", "rss", ("devto", "python")),
|
||||
S("devto_security", "community", "rss", "https://dev.to/feed/tag/security", "rss", ("devto", "security")),
|
||||
S("reddit_llmdevs", "community", "json", "https://www.reddit.com/r/LLMDevs/.json?limit=25", "reddit_listing", ("reddit", "llm")),
|
||||
S("reddit_rag", "community", "json", "https://www.reddit.com/r/Rag/.json?limit=25", "reddit_listing", ("reddit", "rag")),
|
||||
S("reddit_mlops", "community", "json", "https://www.reddit.com/r/mlops/.json?limit=25", "reddit_listing", ("reddit", "mlops")),
|
||||
S("reddit_datascience", "community", "json", "https://www.reddit.com/r/datascience/.json?limit=25", "reddit_listing", ("reddit", "data")),
|
||||
S("reddit_programming", "community", "json", "https://www.reddit.com/r/programming/.json?limit=25", "reddit_listing", ("reddit",)),
|
||||
S("reddit_rust", "community", "json", "https://www.reddit.com/r/rust/.json?limit=25", "reddit_listing", ("reddit", "rust")),
|
||||
S("reddit_golang", "community", "json", "https://www.reddit.com/r/golang/.json?limit=25", "reddit_listing", ("reddit", "go")),
|
||||
S("reddit_devops", "community", "json", "https://www.reddit.com/r/devops/.json?limit=25", "reddit_listing", ("reddit", "devops")),
|
||||
S("reddit_kubernetes", "community", "json", "https://www.reddit.com/r/kubernetes/.json?limit=25", "reddit_listing", ("reddit", "k8s")),
|
||||
S("reddit_selfhosted2", "community", "json", "https://www.reddit.com/r/homelab/.json?limit=25", "reddit_listing", ("reddit", "homelab")),
|
||||
S("reddit_comfyui", "community", "json", "https://www.reddit.com/r/comfyui/.json?limit=25", "reddit_listing", ("reddit", "image")),
|
||||
S("reddit_stablediffusion2", "community", "json", "https://www.reddit.com/r/StableDiffusion/.json?limit=25&sort=new", "reddit_listing", ("reddit", "image")),
|
||||
S("reddit_ollama", "community", "json", "https://www.reddit.com/r/ollama/.json?limit=25", "reddit_listing", ("reddit", "local")),
|
||||
S("reddit_gpumod", "community", "json", "https://www.reddit.com/r/GPUMining/.json?limit=20", "reddit_listing", ("reddit", "gpu")),
|
||||
S("reddit_hacking", "community", "json", "https://www.reddit.com/r/hacking/.json?limit=25", "reddit_listing", ("reddit", "sec")),
|
||||
S("reddit_reverseengineering", "community", "json", "https://www.reddit.com/r/ReverseEngineering/.json?limit=25", "reddit_listing", ("reddit", "re")),
|
||||
S("reddit_scraping", "community", "json", "https://www.reddit.com/r/scrapy/.json?limit=20", "reddit_listing", ("reddit", "scraping")),
|
||||
S("reddit_sysadmin", "community", "json", "https://www.reddit.com/r/sysadmin/.json?limit=25", "reddit_listing", ("reddit", "ops")),
|
||||
|
||||
# ---------------- company ----------------
|
||||
S("cloudflare_blog", "company", "rss", "https://blog.cloudflare.com/rss/", "rss", ("cloudflare", "infra")),
|
||||
S("vercel_blog", "company", "rss", "https://vercel.com/atom", "rss", ("vercel",)),
|
||||
S("github_blog", "company", "rss", "https://github.blog/feed/", "rss", ("github",)),
|
||||
S("github_changelog", "company", "rss", "https://github.blog/changelog/feed/", "rss", ("github",)),
|
||||
S("aws_ml_blog2", "company", "rss", "https://aws.amazon.com/blogs/big-data/feed/", "rss", ("aws", "data")),
|
||||
S("aws_opensource", "company", "rss", "https://aws.amazon.com/blogs/opensource/feed/", "rss", ("aws",)),
|
||||
S("google_cloud_blog", "company", "rss", "https://cloudblog.withgoogle.com/rss/", "rss", ("google", "cloud")),
|
||||
S("azure_blog", "company", "rss", "https://azure.microsoft.com/en-us/blog/feed/", "rss", ("azure",)),
|
||||
S("microsoft_devblogs", "company", "rss", "https://devblogs.microsoft.com/feed/", "rss", ("microsoft",)),
|
||||
S("netflix_techblog", "company", "rss", "https://netflixtechblog.com/feed", "rss", ("netflix", "eng")),
|
||||
S("uber_eng", "company", "rss", "https://www.uber.com/blog/engineering/rss/", "rss", ("uber", "eng")),
|
||||
S("airbnb_eng", "company", "rss", "https://medium.com/feed/airbnb-engineering", "rss", ("airbnb", "eng")),
|
||||
S("spotify_eng", "company", "rss", "https://engineering.atspotify.com/feed/", "rss", ("spotify", "eng")),
|
||||
S("pinterest_eng", "company", "rss", "https://medium.com/feed/pinterest-engineering", "rss", ("pinterest", "eng")),
|
||||
S("discord_blog", "company", "rss", "https://discord.com/blog/rss.xml", "rss", ("discord",)),
|
||||
S("slack_eng", "company", "rss", "https://slack.engineering/feed/", "rss", ("slack", "eng")),
|
||||
S("stripe_blog", "company", "rss", "https://stripe.com/blog/feed.rss", "rss", ("stripe",)),
|
||||
S("figma_blog", "company", "rss", "https://www.figma.com/blog/feed/", "rss", ("figma",)),
|
||||
S("linear_blog", "company", "rss", "https://linear.app/blog/rss.xml", "rss", ("linear",)),
|
||||
S("gitlab_blog", "company", "rss", "https://about.gitlab.com/atom.xml", "rss", ("gitlab",)),
|
||||
S("sentry_blog", "company", "rss", "https://blog.sentry.io/feed.xml", "rss", ("sentry",)),
|
||||
S("posthog_blog", "company", "rss", "https://posthog.com/rss.xml", "rss", ("posthog",)),
|
||||
S("supabase_blog", "company", "rss", "https://supabase.com/rss/followers.xml", "rss", ("supabase", "db")),
|
||||
S("neon_blog", "company", "rss", "https://neon.tech/blog/rss.xml", "rss", ("neon", "db")),
|
||||
S("planetscale_blog", "company", "rss", "https://planetscale.com/blog/feed.atom", "rss", ("planetscale", "db")),
|
||||
S("turso_blog", "company", "rss", "https://turso.tech/blog/rss.xml", "rss", ("turso", "db")),
|
||||
S("mongodb_blog", "company", "rss", "https://www.mongodb.com/blog/rss", "rss", ("mongodb", "db")),
|
||||
S("redis_blog", "company", "rss", "https://redis.io/blog/feed/", "rss", ("redis", "db")),
|
||||
S("elastic_blog", "company", "rss", "https://www.elastic.co/blog/feed", "rss", ("elastic", "search")),
|
||||
S("confluent_blog", "company", "rss", "https://www.confluent.io/blog/feed/", "rss", ("kafka", "data")),
|
||||
S("pinecone_blog", "company", "rss", "https://www.pinecone.io/blog/rss.xml", "rss", ("vector", "db")),
|
||||
S("qdrant_blog", "company", "rss", "https://qdrant.tech/blog/rss.xml", "rss", ("vector", "db")),
|
||||
S("weaviate_blog", "company", "rss", "https://weaviate.io/blog/rss.xml", "rss", ("vector", "db")),
|
||||
S("milvus_blog", "company", "rss", "https://milvus.io/blog/rss.xml", "rss", ("vector", "db")),
|
||||
S("flyio_blog", "company", "rss", "https://fly.io/blog/feed.xml", "rss", ("fly", "infra")),
|
||||
S("railway_blog", "company", "rss", "https://blog.railway.com/rss.xml", "rss", ("railway", "infra")),
|
||||
S("render_blog", "company", "rss", "https://render.com/blog/rss.xml", "rss", ("render", "infra")),
|
||||
S("runpod_blog", "company", "rss", "https://blog.runpod.io/rss/", "rss", ("gpu", "infra")),
|
||||
S("lambda_blog", "company", "rss", "https://lambda.ai/blog/rss.xml", "rss", ("gpu", "infra")),
|
||||
S("baseten_blog", "company", "rss", "https://www.baseten.co/blog/rss.xml", "rss", ("gpu", "infra")),
|
||||
S("raphael_blog", "company", "rss", "https://www.together.ai/blog/rss.xml", "rss", ("together",)),
|
||||
S("lmsys_blog", "company", "rss", "https://lmsys.org/rss.xml", "rss", ("lmsys", "arena")),
|
||||
S("allenai_blog", "company", "rss", "https://blog.allenai.org/feed", "rss", ("allenai",)),
|
||||
S("eleutherai_blog", "company", "html", "https://blog.eleuther.ai/", "html_listing", ("eleutherai",), detail=False),
|
||||
S("ollama_blog", "company", "rss", "https://ollama.com/blog/rss.xml", "rss", ("ollama",)),
|
||||
S("groq_blog", "company", "rss", "https://groq.com/blog/rss.xml", "rss", ("groq",)),
|
||||
S("anthropic_engineering", "company", "html", "https://www.anthropic.com/engineering", "html_listing",
|
||||
("anthropic", "eng"), detail=False),
|
||||
S("openai_research_rss", "company", "rss", "https://openai.com/research/rss.xml", "rss", ("openai", "research")),
|
||||
S("deepmind_github", "company", "github", "google-deepmind", "github_org", ("deepmind", "github")),
|
||||
S("openai_github", "company", "github", "openai", "github_org", ("openai", "github")),
|
||||
S("anthropic_github", "company", "github", "anthropics", "github_org", ("anthropic", "github")),
|
||||
S("meta_github", "company", "github", "facebookresearch", "github_org", ("meta", "github")),
|
||||
S("nvidia_github", "company", "github", "NVIDIA", "github_org", ("nvidia", "github")),
|
||||
S("microsoft_github", "company", "github", "microsoft", "github_org", ("microsoft", "github")),
|
||||
S("huggingface_github", "company", "github", "huggingface", "github_org", ("hf", "github")),
|
||||
S("vllm_github", "company", "github", "vllm-project", "github_org", ("vllm", "github")),
|
||||
S("sglang_github", "company", "github", "sgl-project", "github_org", ("sglang", "github")),
|
||||
S("langchain_github", "company", "github", "langchain-ai", "github_org", ("langchain", "github")),
|
||||
S("llamacpp_github", "company", "github", "ggml-org", "github_org", ("llama.cpp", "github")),
|
||||
S("ollama_github", "company", "github", "ollama", "github_org", ("ollama", "github")),
|
||||
S("unsloth_github", "company", "github", "unslothai", "github_org", ("finetune", "github")),
|
||||
|
||||
# ---------------- company_cn ----------------
|
||||
S("alibaba_cloud_blog", "company_cn", "rss", "https://www.alibabacloud.com/blog/rss", "rss", ("alibaba", "cn")),
|
||||
S("bytedance_seed", "company_cn", "html", "https://team.doubao.com/en/special/seed", "html_listing", ("bytedance", "cn"), detail=False),
|
||||
S("tencent_ai_lab", "company_cn", "html", "https://ai.tencent.com/ailab/en/index", "html_listing", ("tencent", "cn"), detail=False),
|
||||
S("baidu_research", "company_cn", "html", "https://research.baidu.com/Blog", "html_listing", ("baidu", "cn"), detail=False),
|
||||
S("stepfun_blog", "company_cn", "html", "https://www.stepfun.com/blog", "html_listing", ("stepfun", "cn"), detail=False),
|
||||
S("zeroone_ai", "company_cn", "html", "https://www.01.ai/blog", "html_listing", ("01ai", "cn"), detail=False),
|
||||
S("internlm_github", "company_cn", "github", "InternLM", "github_org", ("internlm", "cn")),
|
||||
S("shanghai_ailab_github", "company_cn", "github", "OpenGVLab", "github_org", ("shailab", "cn")),
|
||||
S("modelscope_github", "company_cn", "github", "modelscope", "github_org", ("modelscope", "cn")),
|
||||
S("sensetime_github", "company_cn", "github", "SenseTime", "github_org", ("sensetime", "cn")),
|
||||
S("xiaomi_github", "company_cn", "github", "XiaoMi", "github_org", ("xiaomi", "cn")),
|
||||
S("meituan_tech", "company_cn", "rss", "https://tech.meituan.com/feed/", "rss", ("meituan", "cn")),
|
||||
S("huawei_ascend_github", "company_cn", "github", "Ascend", "github_org", ("huawei", "cn")),
|
||||
S("kimi_blog", "company_cn", "html", "https://www.kimi.com/blog", "html_listing", ("kimi", "cn"), detail=False),
|
||||
S("zhipu_docs_update", "company_cn", "html", "https://docs.bigmodel.cn/cn/update", "html_listing", ("zhipu", "cn"), detail=False),
|
||||
S("minimax_github", "company_cn", "github", "MiniMax-AI", "github_org", ("minimax", "cn")),
|
||||
S("baichuan_github", "company_cn", "github", "baichuan-inc", "github_org", ("baichuan", "cn")),
|
||||
|
||||
# ---------------- company_kr ----------------
|
||||
S("naver_d2", "company_kr", "rss", "https://d2.naver.com/d2.atom", "rss", ("naver", "kr")),
|
||||
S("kakao_enterprise", "company_kr", "rss", "https://tech.kakaoenterprise.com/feed", "rss", ("kakao", "kr")),
|
||||
S("lg_cns_blog", "company_kr", "rss", "https://www.lgcns.com/rss/blog", "rss", ("lg", "kr")),
|
||||
S("samsung_sds", "company_kr", "rss", "https://www.samsungsds.com/kr/rss/blog.xml", "rss", ("samsung", "kr")),
|
||||
S("upstage_blog", "company_kr", "rss", "https://www.upstage.ai/blog/rss.xml", "rss", ("upstage", "kr")),
|
||||
S("ncsoft_blog", "company_kr", "rss", "https://blog.ncsoft.com/feed/", "rss", ("ncsoft", "kr")),
|
||||
|
||||
# ---------------- security ----------------
|
||||
S("cisa_kev", "security", "json", "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json",
|
||||
"cisa_kev", ("cve", "kev")),
|
||||
S("nvd_recent", "security", "json",
|
||||
"https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=25&noRejected", "nvd", ("cve",)),
|
||||
S("github_advisories", "security", "rss", "https://github.com/advisories.atom", "rss", ("cve", "github")),
|
||||
S("snyk_blog", "security", "rss", "https://snyk.io/blog/feed/", "rss", ("appsec",)),
|
||||
S("trailofbits_blog", "security", "rss", "https://blog.trailofbits.com/feed/", "rss", ("research",)),
|
||||
S("rapid7_blog", "security", "rss", "https://www.rapid7.com/blog/rss/", "rss", ("research",)),
|
||||
S("tenable_blog", "security", "rss", "https://www.tenable.com/blog/feed", "rss", ("research",)),
|
||||
S("qualys_blog", "security", "rss", "https://blog.qualys.com/feed", "rss", ("research",)),
|
||||
S("mandiant_blog", "security", "rss", "https://www.mandiant.com/resources/blog/rss.xml", "rss", ("ir",)),
|
||||
S("crowdstrike_blog", "security", "rss", "https://www.crowdstrike.com/en-us/blog/feed/", "rss", ("ir",)),
|
||||
S("sentinelone_labs", "security", "rss", "https://www.sentinelone.com/labs/feed/", "rss", ("malware",)),
|
||||
S("unit42", "security", "rss", "https://unit42.paloaltonetworks.com/feed/", "rss", ("threatintel",)),
|
||||
S("talos", "security", "rss", "https://blog.talosintelligence.com/rss/", "rss", ("threatintel",)),
|
||||
S("securelist", "security", "rss", "https://securelist.com/feed/", "rss", ("threatintel",)),
|
||||
S("welivesecurity", "security", "rss", "https://www.welivesecurity.com/en/rss/feed/", "rss", ("malware",)),
|
||||
S("malwarebytes_labs", "security", "rss", "https://www.malwarebytes.com/blog/feed/index.xml", "rss", ("malware",)),
|
||||
S("darkreading", "security", "rss", "https://www.darkreading.com/rss.xml", "rss", ("news",)),
|
||||
S("therecord", "security", "rss", "https://therecord.media/feed", "rss", ("news",)),
|
||||
S("cyberscoop", "security", "rss", "https://cyberscoop.com/feed/", "rss", ("news",)),
|
||||
S("securityweek", "security", "rss", "https://www.securityweek.com/feed/", "rss", ("news",)),
|
||||
S("google_security_blog", "security", "rss", "https://security.googleblog.com/feeds/posts/default", "rss", ("research",)),
|
||||
S("msrc_blog", "security", "rss", "https://msrc.microsoft.com/blog/feed", "rss", ("microsoft",)),
|
||||
S("aws_security_blog", "security", "rss", "https://aws.amazon.com/blogs/security/feed/", "rss", ("aws",)),
|
||||
S("cloudflare_security", "security", "rss", "https://blog.cloudflare.com/tag/security/rss/", "rss", ("cloudflare",)),
|
||||
S("wiz_blog", "security", "rss", "https://www.wiz.io/blog/rss.xml", "rss", ("cloud",)),
|
||||
S("orca_security", "security", "rss", "https://orca.security/resources/blog/feed/", "rss", ("cloud",)),
|
||||
S("sysdig_blog", "security", "rss", "https://sysdig.com/blog/feed/", "rss", ("container",)),
|
||||
S("aquasecurity_blog", "security", "rss", "https://www.aquasec.com/blog/feed/", "rss", ("container",)),
|
||||
S("redcanary_blog", "security", "rss", "https://redcanary.com/blog/feed/", "rss", ("detection",)),
|
||||
S("elastic_security_labs", "security", "rss", "https://www.elastic.co/security-labs/rss/feed.xml", "rss", ("research",)),
|
||||
S("projectzero_issues", "security", "rss", "https://googleprojectzero.blogspot.com/feeds/posts/default?alt=rss", "rss", ("research",)),
|
||||
S("seclists", "security", "html", "https://seclists.org/", "html_listing", ("mailing",), detail=False),
|
||||
S("hackerone_hacktivity", "security", "json", "https://hackerone.com/hacktivity.json?limit=20", "generic_json", ("bb",)),
|
||||
S("bugcrowd_crowdstream", "security", "html", "https://bugcrowd.com/crowdstream", "html_listing", ("bb",), detail=False),
|
||||
S("yeswehack_programs", "security", "html", "https://yeswehack.com/programs", "html_listing", ("bb",), detail=False),
|
||||
|
||||
# ---------------- forum ----------------
|
||||
S("discourse_pytorch", "forum", "json", "https://discuss.pytorch.org/latest.json", "discourse", ("pytorch",)),
|
||||
S("discourse_hf_forum", "forum", "json", "https://discuss.huggingface.co/latest.json", "discourse", ("hf",)),
|
||||
S("discourse_cursor", "forum", "json", "https://forum.cursor.com/latest.json", "discourse", ("cursor",)),
|
||||
S("discourse_n8n", "forum", "json", "https://community.n8n.io/latest.json", "discourse", ("n8n",)),
|
||||
S("discourse_grafana", "forum", "json", "https://community.grafana.com/latest.json", "discourse", ("grafana",)),
|
||||
S("discourse_homeassistant", "forum", "json", "https://community.home-assistant.io/latest.json", "discourse", ("iot",)),
|
||||
S("discourse_nextcloud", "forum", "json", "https://help.nextcloud.com/latest.json", "discourse", ("selfhosted",)),
|
||||
S("discourse_influxdata", "forum", "json", "https://community.influxdata.com/latest.json", "discourse", ("metrics",)),
|
||||
S("discourse_rasa", "forum", "json", "https://forum.rasa.com/latest.json", "discourse", ("chatbot",)),
|
||||
S("discourse_traefik", "forum", "json", "https://community.traefik.io/latest.json", "discourse", ("proxy",)),
|
||||
S("discourse_letsencrypt", "forum", "json", "https://community.letsencrypt.org/latest.json", "discourse", ("tls",)),
|
||||
S("discourse_obsidian", "forum", "json", "https://forum.obsidian.md/latest.json", "discourse", ("notes",)),
|
||||
S("discourse_anki", "forum", "json", "https://forums.ankiweb.net/latest.json", "discourse", ("learning",)),
|
||||
S("discourse_rust", "forum", "json", "https://users.rust-lang.org/latest.json", "discourse", ("rust",)),
|
||||
S("discourse_go", "forum", "json", "https://forum.golangbridge.org/latest.json", "discourse", ("go",)),
|
||||
S("discourse_elixir", "forum", "json", "https://elixirforum.com/latest.json", "discourse", ("elixir",)),
|
||||
S("xenforo_webmasterworld", "forum", "html", "https://www.webmasterworld.com/", "xenforo_latest", ("seo",)),
|
||||
S("tg_ai_newz", "forum", "html", "https://t.me/s/ai_newz", "tg_web", ("telegram", "ai")),
|
||||
S("tg_deepseek", "forum", "html", "https://t.me/s/deepseek_ai", "tg_web", ("telegram", "cn")),
|
||||
S("tg_qwen", "forum", "html", "https://t.me/s/Qwen_AI", "tg_web", ("telegram", "cn")),
|
||||
S("tg_neurohive", "forum", "html", "https://t.me/s/neurohive", "tg_web", ("telegram", "ai")),
|
||||
S("tg_gpt4telegram", "forum", "html", "https://t.me/s/GPTMain", "tg_web", ("telegram", "ai")),
|
||||
]
|
||||
505
web/index.html
505
web/index.html
@@ -5,246 +5,321 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>omni · радар источников</title>
|
||||
<style>
|
||||
:root{
|
||||
--bg:#0b0d10; --bg2:#11151a; --card:#141a20; --card2:#171e26;
|
||||
--line:#232c35; --txt:#e7edf3; --dim:#8b98a5; --dim2:#5f6b78;
|
||||
--acc:#c6f24e; --acc2:#4ee0c6; --warn:#ffb454;
|
||||
--mono:ui-monospace,"JetBrains Mono","SF Mono",Menlo,monospace;
|
||||
}
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;height:100%}
|
||||
body{
|
||||
background:
|
||||
radial-gradient(1200px 700px at 15% -10%, #1a2430 0%, transparent 60%),
|
||||
radial-gradient(900px 600px at 100% 0%, #14251f 0%, transparent 55%),
|
||||
var(--bg);
|
||||
color:var(--txt);
|
||||
font:15px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Roboto,sans-serif;
|
||||
-webkit-font-smoothing:antialiased;
|
||||
}
|
||||
.wrap{display:grid;grid-template-columns:236px 1fr;min-height:100vh}
|
||||
/* ---------- sidebar ---------- */
|
||||
aside{
|
||||
border-right:1px solid var(--line);padding:22px 16px;position:sticky;top:0;height:100vh;
|
||||
display:flex;flex-direction:column;gap:20px;background:rgba(10,13,16,.6);backdrop-filter:blur(8px);
|
||||
}
|
||||
.brand{display:flex;align-items:center;gap:10px}
|
||||
.mark{width:30px;height:30px;border-radius:9px;background:linear-gradient(140deg,var(--acc),var(--acc2));
|
||||
display:grid;place-items:center;color:#0b0d10;font-weight:800;font-size:15px;font-family:var(--mono)}
|
||||
.brand b{font-size:16px;letter-spacing:.2px}
|
||||
.brand span{display:block;font-size:11px;color:var(--dim2);font-family:var(--mono)}
|
||||
nav{display:flex;flex-direction:column;gap:3px}
|
||||
nav button{
|
||||
all:unset;cursor:pointer;display:flex;justify-content:space-between;align-items:center;
|
||||
padding:8px 10px;border-radius:8px;color:var(--dim);font-size:13.5px;
|
||||
}
|
||||
nav button:hover{background:var(--card);color:var(--txt)}
|
||||
nav button.on{background:var(--card2);color:var(--txt);box-shadow:inset 0 0 0 1px var(--line)}
|
||||
nav button i{font-family:var(--mono);font-size:11px;color:var(--dim2);font-style:normal}
|
||||
nav button.on i{color:var(--acc)}
|
||||
.side-foot{margin-top:auto;font-family:var(--mono);font-size:11px;color:var(--dim2);line-height:1.7}
|
||||
/* ---------- main ---------- */
|
||||
main{padding:26px 32px 60px;max-width:1180px}
|
||||
header{display:flex;gap:14px;align-items:flex-end;justify-content:space-between;flex-wrap:wrap}
|
||||
h1{margin:0;font-size:24px;letter-spacing:-.4px}
|
||||
h1 em{font-style:normal;color:var(--acc)}
|
||||
.sub{color:var(--dim);font-size:13px;margin-top:4px}
|
||||
.tools{display:flex;gap:8px;align-items:center}
|
||||
input[type=search]{
|
||||
background:var(--card);border:1px solid var(--line);border-radius:10px;color:var(--txt);
|
||||
padding:10px 13px;width:270px;font-size:14px;outline:none;
|
||||
}
|
||||
input[type=search]:focus{border-color:#3a4a56;box-shadow:0 0 0 3px rgba(198,242,78,.08)}
|
||||
.btn{
|
||||
cursor:pointer;border:1px solid var(--line);background:var(--card);color:var(--txt);
|
||||
padding:10px 16px;border-radius:10px;font-size:13.5px;font-weight:600;
|
||||
}
|
||||
.btn:hover{border-color:#3a4a56}
|
||||
.btn.primary{background:var(--acc);color:#0b0d10;border-color:transparent}
|
||||
.btn.primary:disabled{opacity:.5;cursor:wait}
|
||||
/* ---------- stats ---------- */
|
||||
.stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin:22px 0 26px}
|
||||
.stat{background:var(--card);border:1px solid var(--line);border-radius:14px;padding:14px 16px}
|
||||
.stat b{display:block;font-family:var(--mono);font-size:24px;letter-spacing:-.5px}
|
||||
.stat span{font-size:11.5px;color:var(--dim);text-transform:uppercase;letter-spacing:.6px}
|
||||
.stat.a b{color:var(--acc)} .stat.b b{color:var(--acc2)}
|
||||
/* ---------- feed ---------- */
|
||||
.bar{display:flex;gap:10px;align-items:center;margin-bottom:14px;flex-wrap:wrap}
|
||||
.chip{
|
||||
font-family:var(--mono);font-size:11px;color:var(--dim);border:1px solid var(--line);
|
||||
border-radius:999px;padding:4px 10px;background:var(--card)
|
||||
}
|
||||
.chip.hot{color:var(--acc);border-color:#2b3a20}
|
||||
.list{display:flex;flex-direction:column;gap:10px}
|
||||
.row{
|
||||
display:grid;grid-template-columns:56px 1fr auto;gap:14px;align-items:start;
|
||||
background:linear-gradient(180deg,var(--card2),var(--card));
|
||||
border:1px solid var(--line);border-radius:14px;padding:14px 16px;transition:border-color .15s,transform .15s;
|
||||
}
|
||||
.row:hover{border-color:#33414d;transform:translateY(-1px)}
|
||||
.score{font-family:var(--mono);font-size:15px;color:var(--acc);text-align:right;padding-top:1px}
|
||||
.score small{display:block;font-size:9.5px;color:var(--dim2);letter-spacing:.5px}
|
||||
.t{font-size:15px;font-weight:600;line-height:1.35;margin:0 0 6px;letter-spacing:-.2px}
|
||||
.t a{color:inherit;text-decoration:none}
|
||||
.t a:hover{color:var(--acc)}
|
||||
.meta{display:flex;gap:8px;flex-wrap:wrap;align-items:center;font-family:var(--mono);font-size:11px;color:var(--dim)}
|
||||
.src{color:var(--acc2)}
|
||||
.txt{color:var(--dim);font-size:13px;margin-top:8px;display:-webkit-box;-webkit-line-clamp:2;-webkit-box-orient:vertical;overflow:hidden}
|
||||
.ext{font-family:var(--mono);font-size:11px;color:var(--dim2);align-self:center;white-space:nowrap}
|
||||
.empty{color:var(--dim);padding:40px 0;text-align:center;font-family:var(--mono);font-size:13px}
|
||||
.toast{
|
||||
position:fixed;bottom:22px;right:22px;background:var(--card2);border:1px solid var(--line);
|
||||
border-radius:12px;padding:12px 16px;font-size:13px;color:var(--txt);opacity:0;pointer-events:none;
|
||||
transition:opacity .25s;max-width:340px;font-family:var(--mono)
|
||||
}
|
||||
.toast.on{opacity:1}
|
||||
@media(max-width:820px){
|
||||
.wrap{grid-template-columns:1fr} aside{position:static;height:auto;flex-direction:row;flex-wrap:wrap;gap:12px;padding:14px}
|
||||
nav{flex-direction:row;flex-wrap:wrap} main{padding:18px 16px 50px} .tools{width:100%} input[type=search]{flex:1;width:auto}
|
||||
.row{grid-template-columns:44px 1fr}
|
||||
}
|
||||
:root{
|
||||
--bg:#0b0d10; --bg2:#11151a; --bg3:#161b22; --line:#232a33;
|
||||
--tx:#e6edf3; --dim:#8b949e; --dim2:#5b6673;
|
||||
--acc:#4ee39b; --acc2:#7aa2ff; --warn:#f0b429; --bad:#f85149;
|
||||
--mono:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,monospace;
|
||||
}
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;background:var(--bg);color:var(--tx);
|
||||
font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,sans-serif}
|
||||
a{color:var(--acc2);text-decoration:none}
|
||||
a:hover{text-decoration:underline}
|
||||
button{font:inherit;cursor:pointer}
|
||||
.wrap{max-width:1240px;margin:0 auto;padding:22px 18px 60px}
|
||||
|
||||
header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:6px}
|
||||
.logo{font:700 20px/1 var(--mono);letter-spacing:.5px}
|
||||
.logo b{color:var(--acc)}
|
||||
.sub{color:var(--dim);font-size:12.5px}
|
||||
.tabs{display:flex;gap:6px;margin:16px 0 18px;border-bottom:1px solid var(--line)}
|
||||
.tab{padding:9px 14px;border:0;background:none;color:var(--dim);border-bottom:2px solid transparent;
|
||||
font-size:13.5px;letter-spacing:.2px}
|
||||
.tab.on{color:var(--tx);border-bottom-color:var(--acc)}
|
||||
.tab:hover{color:var(--tx)}
|
||||
|
||||
.stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(126px,1fr));gap:8px;margin-bottom:18px}
|
||||
.stat{background:var(--bg2);border:1px solid var(--line);border-radius:10px;padding:11px 13px}
|
||||
.stat .v{font:700 19px/1.2 var(--mono)}
|
||||
.stat .k{color:var(--dim);font-size:11.5px;text-transform:uppercase;letter-spacing:.6px;margin-top:3px}
|
||||
.stat.ok .v{color:var(--acc)} .stat.bad .v{color:var(--bad)} .stat.warn .v{color:var(--warn)}
|
||||
|
||||
.bar{display:flex;gap:8px;flex-wrap:wrap;margin-bottom:14px;align-items:center}
|
||||
input[type=text],select{background:var(--bg3);border:1px solid var(--line);color:var(--tx);
|
||||
border-radius:8px;padding:8px 11px;font:13px var(--mono);outline:none;min-width:0}
|
||||
input[type=text]:focus,select:focus{border-color:var(--acc2)}
|
||||
input[type=text]{flex:1 1 220px}
|
||||
.btn{background:var(--bg3);border:1px solid var(--line);color:var(--tx);border-radius:8px;
|
||||
padding:8px 13px;font-size:13px;white-space:nowrap}
|
||||
.btn:hover{border-color:var(--acc2)}
|
||||
.btn.pri{background:rgba(78,227,155,.12);border-color:rgba(78,227,155,.45);color:var(--acc)}
|
||||
.btn.gh{background:none;color:var(--dim)}
|
||||
.btn:disabled{opacity:.4;cursor:default}
|
||||
|
||||
.add{background:var(--bg2);border:1px solid var(--line);border-radius:10px;padding:13px;margin-bottom:16px}
|
||||
.add .ttl{font:600 12px/1 var(--mono);color:var(--dim);text-transform:uppercase;letter-spacing:.8px;margin-bottom:10px}
|
||||
.add .row{display:flex;gap:8px;flex-wrap:wrap}
|
||||
.add .row input[type=text]{flex:1 1 180px}
|
||||
|
||||
.listhead{display:flex;justify-content:space-between;align-items:center;color:var(--dim);
|
||||
font:11.5px var(--mono);text-transform:uppercase;letter-spacing:.7px;padding:0 4px 8px}
|
||||
.list{border:1px solid var(--line);border-radius:10px;overflow:hidden;background:var(--bg2)}
|
||||
.src{display:grid;grid-template-columns:38px 1fr 132px 104px 62px 34px;gap:10px;align-items:center;
|
||||
padding:9px 12px;border-bottom:1px solid var(--line)}
|
||||
.src:last-child{border-bottom:0}
|
||||
.src:hover{background:rgba(255,255,255,.022)}
|
||||
.src.off{opacity:.45}
|
||||
.nm{font-size:13.5px;display:flex;align-items:center;gap:7px;flex-wrap:wrap}
|
||||
.u{color:var(--dim2);font:11.5px var(--mono);overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||||
.cat{font:11px var(--mono);color:var(--dim);text-transform:uppercase;letter-spacing:.5px}
|
||||
.cat.k{color:var(--acc2)}
|
||||
.pill{font:11.5px var(--mono);padding:2px 8px;border-radius:20px;border:1px solid var(--line);
|
||||
color:var(--dim);white-space:nowrap}
|
||||
.pill.ok{color:var(--acc);border-color:rgba(78,227,155,.35);background:rgba(78,227,155,.08)}
|
||||
.pill.bad{color:var(--bad);border-color:rgba(248,81,73,.35);background:rgba(248,81,73,.07)}
|
||||
.pill.wait{color:var(--warn);border-color:rgba(240,180,41,.3)}
|
||||
.cnt{font:12px var(--mono);color:var(--dim);text-align:right}
|
||||
.del{background:none;border:0;color:var(--dim2);font-size:15px;line-height:1;padding:4px}
|
||||
.del:hover{color:var(--bad)}
|
||||
.sw{position:relative;width:34px;height:19px;display:inline-block}
|
||||
.sw input{opacity:0;width:0;height:0;position:absolute}
|
||||
.sw span{position:absolute;inset:0;background:var(--bg3);border:1px solid var(--line);
|
||||
border-radius:20px;transition:.15s}
|
||||
.sw span:before{content:"";position:absolute;width:13px;height:13px;left:2px;top:2px;border-radius:50%;
|
||||
background:var(--dim);transition:.15s}
|
||||
.sw input:checked + span{background:rgba(78,227,155,.18);border-color:rgba(78,227,155,.5)}
|
||||
.sw input:checked + span:before{transform:translateX(15px);background:var(--acc)}
|
||||
|
||||
.card{background:var(--bg2);border:1px solid var(--line);border-radius:10px;padding:13px 15px;margin-bottom:10px}
|
||||
.card h3{margin:0 0 6px;font:600 14.5px/1.35 inherit}
|
||||
.card .m{color:var(--dim);font:11.5px var(--mono);display:flex;gap:10px;flex-wrap:wrap;margin-bottom:6px}
|
||||
.card p{margin:0;color:#b9c2cc;font-size:13px;line-height:1.45}
|
||||
.card .body{margin-top:7px;color:#8b949e;font-size:12.5px;max-height:4.4em;overflow:hidden}
|
||||
.empty{color:var(--dim);text-align:center;padding:36px 10px;font:12.5px var(--mono)}
|
||||
#toast{position:fixed;left:50%;bottom:22px;transform:translateX(-50%) translateY(20px);
|
||||
background:var(--bg3);border:1px solid var(--line);padding:9px 15px;border-radius:9px;
|
||||
font:12.5px var(--mono);opacity:0;pointer-events:none;transition:.2s;z-index:9}
|
||||
#toast.on{opacity:1;transform:translateX(-50%) translateY(0)}
|
||||
.prog{font:12px var(--mono);color:var(--warn);margin-bottom:10px}
|
||||
.hide{display:none!important}
|
||||
@media(max-width:760px){
|
||||
.src{grid-template-columns:34px 1fr 34px;grid-template-areas:"sw nm del" ". st ." ". cat .";gap:6px 10px}
|
||||
.src .sw{grid-area:sw}.src .meta{grid-area:nm}.src .del{grid-area:del}
|
||||
.src .catcell{grid-area:cat}.src .stcell{grid-area:st}.src .cnt{display:none}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<aside>
|
||||
<div class="brand">
|
||||
<div class="mark">o</div>
|
||||
<div><b>omni</b><span>радар источников</span></div>
|
||||
</div>
|
||||
<nav id="nav"></nav>
|
||||
<div class="side-foot" id="foot">загрузка…</div>
|
||||
</aside>
|
||||
|
||||
<main>
|
||||
<header>
|
||||
<div>
|
||||
<h1>что прилетело <em>без запроса</em></h1>
|
||||
<div class="sub" id="sub">ленты, API и форумы в одной базе</div>
|
||||
</div>
|
||||
<div class="tools">
|
||||
<input id="q" type="search" placeholder="поиск по базе…" autocomplete="off">
|
||||
<button class="btn" id="refresh">обновить вид</button>
|
||||
<button class="btn primary" id="run">собрать заново</button>
|
||||
</div>
|
||||
<div class="logo">omni<b>·</b>radar</div>
|
||||
<div class="sub" id="sub">загрузка…</div>
|
||||
</header>
|
||||
|
||||
<div class="stats" id="stats"></div>
|
||||
|
||||
<div class="bar">
|
||||
<span class="chip" id="chipCount">—</span>
|
||||
<span class="chip" id="chipCat">все категории</span>
|
||||
<span class="chip hot" id="chipApi">api /api/items</span>
|
||||
<div class="tabs">
|
||||
<button class="tab on" data-view="feed">лента</button>
|
||||
<button class="tab" data-view="src">источники</button>
|
||||
</div>
|
||||
|
||||
<div class="list" id="list"></div>
|
||||
<div class="empty" id="empty" style="display:none">пусто — нажми «собрать заново»</div>
|
||||
</main>
|
||||
<!-- ============ FEED ============ -->
|
||||
<section id="view-feed">
|
||||
<div class="stats" id="stats"></div>
|
||||
<div class="bar">
|
||||
<input type="text" id="q" placeholder="поиск по заголовку, описанию, url…">
|
||||
<select id="cat"><option value="">все категории</option></select>
|
||||
<button class="btn pri" id="feed">собрать ленту</button>
|
||||
</div>
|
||||
<div class="prog hide" id="fprog"></div>
|
||||
<div id="items"></div>
|
||||
</section>
|
||||
|
||||
<!-- ============ SOURCES ============ -->
|
||||
<section id="view-src" class="hide">
|
||||
<div class="stats" id="sstats"></div>
|
||||
<div class="bar">
|
||||
<input type="text" id="sq" placeholder="фильтр по имени или url…">
|
||||
<select id="scat"><option value="all">все категории</option></select>
|
||||
<button class="btn" id="checkAll">проверить всё</button>
|
||||
<button class="btn" id="checkCat">проверить категорию</button>
|
||||
<button class="btn gh" id="onAll">вкл все</button>
|
||||
<button class="btn gh" id="offAll">выкл все</button>
|
||||
</div>
|
||||
<div class="prog hide" id="sprog"></div>
|
||||
|
||||
<div class="add">
|
||||
<div class="ttl">подключить источник</div>
|
||||
<div class="row">
|
||||
<input type="text" id="aname" placeholder="имя (латиницей, без пробелов)" style="flex:0 1 190px">
|
||||
<input type="text" id="aurl" placeholder="https://example.com/feed.xml | …/latest.json | github.com/org">
|
||||
<select id="acat"></select>
|
||||
<button class="btn pri" id="add">подключить</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="listhead"><span id="scount">—</span><span>статус · записей</span></div>
|
||||
<div class="list" id="slist"></div>
|
||||
</section>
|
||||
</div>
|
||||
<div class="toast" id="toast"></div>
|
||||
<div id="toast"></div>
|
||||
|
||||
<script>
|
||||
const $ = s => document.querySelector(s);
|
||||
const state = {cat:"all", q:"", items:[], cats:[]};
|
||||
const fmt = n => n==null?"—":(n>=1000?(n/1000).toFixed(1)+"k":String(n));
|
||||
const el = (t, c, h) => { const d = document.createElement(t); if (c) d.className = c; if (h != null) d.innerHTML = h; return d; };
|
||||
const esc = s => String(s == null ? "" : s).replace(/[&<>"]/g, c => ({'&':'&','<':'<','>':'>','"':'"'}[c]));
|
||||
let CATS = {}, STATE = {view: "feed", scat: "all", sq: "", cat: ""};
|
||||
|
||||
function toast(msg, ms=4000){
|
||||
const t=$("#toast"); t.textContent=msg; t.classList.add("on");
|
||||
clearTimeout(t._h); t._h=setTimeout(()=>t.classList.remove("on"), ms);
|
||||
}
|
||||
|
||||
async function getJSON(url){
|
||||
const r = await fetch(url, {headers:{"Accept":"application/json"}});
|
||||
if(!r.ok) throw new Error(r.status);
|
||||
function toast(m){ const t = $("#toast"); t.textContent = m; t.classList.add("on"); clearTimeout(t._h); t._h = setTimeout(() => t.classList.remove("on"), 2200); }
|
||||
async function api(u, b){
|
||||
const r = await fetch(u, b ? {method:"POST", headers:{"Content-Type":"application/json"}, body:JSON.stringify(b)} : undefined);
|
||||
return r.json();
|
||||
}
|
||||
|
||||
async function loadStats(){
|
||||
const s = await getJSON("/api/stats");
|
||||
const cards = [
|
||||
{cls:"a", v:fmt(s.items), l:"записей в базе"},
|
||||
{cls:"", v:fmt(s.sources), l:"источников"},
|
||||
{cls:"b", v:fmt(s.runs), l:"прогонов"},
|
||||
{cls:"", v:fmt(s.by_provider.length), l:"активных коллекторов"},
|
||||
{cls:"", v:fmt(s.priced), l:"с ценником"},
|
||||
];
|
||||
$("#stats").innerHTML = cards.map(c=>`<div class="stat ${c.cls}"><b>${c.v}</b><span>${c.l}</span></div>`).join("");
|
||||
const cats = s.categories||{};
|
||||
state.cats = Object.entries(cats).sort((a,b)=>b[1]-a[1]);
|
||||
renderNav(s.items);
|
||||
$("#foot").innerHTML = `источников: ${s.sources}<br>категорий: ${state.cats.length}<br>прогонов ok: ${s.runs_ok}/${s.runs}`;
|
||||
}
|
||||
/* ---------- вкладки ---------- */
|
||||
document.querySelectorAll(".tab").forEach(t => t.onclick = () => {
|
||||
document.querySelectorAll(".tab").forEach(x => x.classList.toggle("on", x === t));
|
||||
STATE.view = t.dataset.view;
|
||||
$("#view-feed").classList.toggle("hide", STATE.view !== "feed");
|
||||
$("#view-src").classList.toggle("hide", STATE.view !== "src");
|
||||
if (STATE.view === "src") loadSources();
|
||||
});
|
||||
|
||||
function renderNav(total){
|
||||
const rows = [["all","всё", total], ...state.cats];
|
||||
$("#nav").innerHTML = rows.map(([k,label,n])=>
|
||||
`<button data-cat="${k}" class="${state.cat===k?'on':''}"><span>${label}</span><i>${fmt(n)}</i></button>`).join("");
|
||||
$("#nav").querySelectorAll("button").forEach(b=>b.onclick=()=>{
|
||||
state.cat=b.dataset.cat; renderNav(total); loadItems();
|
||||
});
|
||||
/* ---------- лента ---------- */
|
||||
async function loadStats(){
|
||||
const s = await api("/api/stats");
|
||||
CATS = s.categories || {};
|
||||
$("#sub").textContent = `${s.sources} источников · ${s.sources_enabled} включено · ${s.items} записей · ${s.priced} с ценой`;
|
||||
const box = $("#stats"); box.innerHTML = "";
|
||||
[["записей", s.items, ""], ["с ценой", s.priced, ""], ["прогонов", s.runs, ""],
|
||||
["источников", s.sources, "ok"], ["включено", s.sources_enabled, "ok"]]
|
||||
.forEach(([k, v, c]) => { const d = el("div", "stat " + c, `<div class="v">${v ?? 0}</div><div class="k">${k}</div>`); box.appendChild(d); });
|
||||
const sel = $("#cat"), cur = sel.value;
|
||||
sel.innerHTML = '<option value="">все категории</option>' +
|
||||
Object.entries(CATS).map(([k, v]) => `<option value="${k}">${k} · ${v}</option>`).join("");
|
||||
sel.value = cur;
|
||||
const scat = $("#scat"), curs = scat.value || "all";
|
||||
scat.innerHTML = '<option value="all">все категории</option>' +
|
||||
Object.entries(CATS).map(([k, v]) => `<option value="${k}">${k} · ${v}</option>`).join("");
|
||||
scat.value = curs;
|
||||
const acat = $("#acat");
|
||||
acat.innerHTML = Object.keys(CATS).map(k => `<option value="${k}">${k}</option>`).join("") + '<option value="custom">custom</option>';
|
||||
}
|
||||
|
||||
async function loadItems(){
|
||||
$("#chipCat").textContent = state.cat==="all" ? "все категории" : state.cat;
|
||||
const url = `/api/items?category=${encodeURIComponent(state.cat)}&q=${encodeURIComponent(state.q)}&limit=80`;
|
||||
const d = await getJSON(url);
|
||||
state.items = d.items||[];
|
||||
$("#chipCount").textContent = `${d.count} записей`;
|
||||
$("#empty").style.display = state.items.length ? "none" : "block";
|
||||
$("#list").innerHTML = state.items.map(it=>{
|
||||
const score = (it.score||0).toFixed(2);
|
||||
const date = (it.published||"").toString().slice(0,16).replace("T"," ");
|
||||
const host = (()=>{try{return new URL(it.url).hostname.replace("www.","")}catch(e){return ""}})();
|
||||
const ext = it.price ? `${it.price} ${it.currency||""}` : (it.kind||"");
|
||||
return `<article class="row">
|
||||
<div class="score">${score}<small>score</small></div>
|
||||
<div>
|
||||
<p class="t"><a href="${it.url||"#"}" target="_blank" rel="noopener">${esc(it.title||"(без заголовка)")}</a></p>
|
||||
<div class="meta">
|
||||
<span class="src">${esc(it.source||it.provider)}</span>
|
||||
<span>${esc(it.provider)}</span>
|
||||
${date?`<span>${esc(date)}</span>`:""}
|
||||
${it.author?`<span>@${esc(it.author)}</span>`:""}
|
||||
</div>
|
||||
${it.text?`<div class="txt">${esc(it.text.slice(0,240))}</div>`:""}
|
||||
</div>
|
||||
<div class="ext">${esc(ext)}${host?`<br>${esc(host)}`:""}</div>
|
||||
</article>`;
|
||||
}).join("");
|
||||
const q = $("#q").value.trim(), cat = $("#cat").value;
|
||||
const d = await api(`/api/items?q=${encodeURIComponent(q)}&category=${encodeURIComponent(cat)}&limit=80`);
|
||||
const box = $("#items"); box.innerHTML = "";
|
||||
if (!d.items.length) { box.appendChild(el("div", "empty", "пусто — нажми «собрать ленту»")); return; }
|
||||
d.items.forEach(it => {
|
||||
const c = el("div", "card");
|
||||
const price = it.price ? ` · $${it.price}${it.currency ? " " + it.currency : ""}` : "";
|
||||
c.innerHTML = `<h3><a href="${esc(it.url)}" target="_blank" rel="noopener">${esc(it.title)}</a></h3>
|
||||
<div class="m"><span>${esc(it.provider || "—")}</span><span>${esc(it.kind || "")}</span>
|
||||
<span>${esc(it.author || "")}</span><span>★ ${it.score ?? 0}</span>${price ? `<span>${esc(price)}</span>` : ""}</div>
|
||||
${it.body ? `<div class="body">${esc(it.body)}</div>` : ""}`;
|
||||
box.appendChild(c);
|
||||
});
|
||||
}
|
||||
|
||||
function esc(s){return String(s==null?"":s).replace(/[&<>"]/g,c=>({"&":"&","<":"<",">":">",'"':"""}[c]))}
|
||||
|
||||
let tmr;
|
||||
$("#q").addEventListener("input", e=>{
|
||||
clearTimeout(tmr);
|
||||
tmr = setTimeout(()=>{ state.q = e.target.value.trim(); loadItems(); }, 280);
|
||||
});
|
||||
$("#refresh").onclick = ()=>{ loadStats(); loadItems(); toast("вид обновлён"); };
|
||||
|
||||
$("#run").onclick = async ()=>{
|
||||
const b = $("#run"); b.disabled=true; b.textContent="собираю…";
|
||||
try{
|
||||
const cat = state.cat==="all" ? null : state.cat;
|
||||
const r = await fetch("/api/feed", {method:"POST", headers:{"Content-Type":"application/json"},
|
||||
body: JSON.stringify({category:cat, limit:12})});
|
||||
const {job} = await r.json();
|
||||
let tries=0;
|
||||
while(tries++ < 120){
|
||||
await new Promise(x=>setTimeout(x,1500));
|
||||
const j = await getJSON("/api/job/"+job);
|
||||
if(j.state==="done"){
|
||||
toast(`собрано ${j.fetched}, новых ${j.new}, дублей ${j.dup}, пустых источников ${j.empty}`);
|
||||
await loadStats(); await loadItems(); break;
|
||||
}
|
||||
if(j.state==="error"){ toast("ошибка: "+j.error); break; }
|
||||
if(tries%4===0) toast("ещё собираю… "+tries*1.5+"с", 1600);
|
||||
}
|
||||
}catch(e){ toast("не вышло: "+e.message); }
|
||||
finally{ b.disabled=false; b.textContent="собрать заново"; }
|
||||
$("#q").oninput = (() => { let h; return () => { clearTimeout(h); h = setTimeout(loadItems, 260); }; })();
|
||||
$("#cat").onchange = loadItems;
|
||||
$("#feed").onclick = async () => {
|
||||
const b = $("#feed"); b.disabled = true;
|
||||
const cat = $("#cat").value;
|
||||
const {job} = await api("/api/feed", {category: cat || null, limit: 10});
|
||||
$("#fprog").classList.remove("hide");
|
||||
const tick = setInterval(async () => {
|
||||
const j = await api("/api/job/" + job);
|
||||
if (j.state === "running") { $("#fprog").textContent = "собираю…"; return; }
|
||||
clearInterval(tick); b.disabled = false;
|
||||
$("#fprog").textContent = j.state === "done"
|
||||
? `источников ${j.sources} · собрано ${j.fetched} · новых ${j.new} · дублей ${j.dup} · пустых ${j.empty}`
|
||||
: "ошибка: " + (j.error || "?");
|
||||
setTimeout(() => $("#fprog").classList.add("hide"), 6000);
|
||||
await loadStats(); await loadItems();
|
||||
}, 1200);
|
||||
};
|
||||
|
||||
loadStats().then(loadItems).catch(e=>toast("api недоступен: "+e.message));
|
||||
/* ---------- источники ---------- */
|
||||
async function loadSources(){
|
||||
const cat = STATE.scat, q = STATE.sq;
|
||||
const d = await api(`/api/sources?category=${encodeURIComponent(cat === "all" ? "" : cat)}&q=${encodeURIComponent(q)}`);
|
||||
const sb = $("#sstats"); sb.innerHTML = "";
|
||||
[["всего", d.total, ""], ["в списке", d.shown, ""], ["включено", d.enabled, "ok"],
|
||||
["живых", d.alive, "ok"], ["мёртвых", d.dead, "bad"], ["не проверено", d.untested, "warn"]]
|
||||
.forEach(([k, v, c]) => sb.appendChild(el("div", "stat " + c, `<div class="v">${v ?? 0}</div><div class="k">${k}</div>`)));
|
||||
$("#scount").textContent = `${d.shown} из ${d.total}`;
|
||||
|
||||
const box = $("#slist"); box.innerHTML = "";
|
||||
if (!d.items.length) { box.appendChild(el("div", "empty", "ничего не найдено")); return; }
|
||||
d.items.forEach(s => {
|
||||
const row = el("div", "src" + (s.enabled ? "" : " off"));
|
||||
const st = s.items > 0
|
||||
? `<span class="pill ok">${s.items} зап.</span>`
|
||||
: (s.checked ? `<span class="pill bad">${esc(s.note || "мёртв")}</span>` : `<span class="pill">не проверен</span>`);
|
||||
row.innerHTML = `
|
||||
<label class="sw"><input type="checkbox" ${s.enabled ? "checked" : ""}><span></span></label>
|
||||
<div class="meta"><div class="nm">${esc(s.name)}${s.custom ? ' <span class="pill wait">свой</span>' : ""}</div>
|
||||
<div class="u" title="${esc(s.url)}">${esc(s.url)}</div></div>
|
||||
<div class="catcell"><span class="cat">${esc(s.category)}</span> <span class="cat k">${esc(s.kind)}</span></div>
|
||||
<div class="stcell">${st}</div>
|
||||
<div class="cnt">${s.items || ""}</div>
|
||||
<button class="del" title="удалить">${s.custom ? "✕" : ""}</button>`;
|
||||
row.querySelector("input").onchange = async e => {
|
||||
const on = e.target.checked;
|
||||
row.classList.toggle("off", !on);
|
||||
await api("/api/sources/toggle", {name: s.name, enabled: on});
|
||||
toast(`${s.name}: ${on ? "включён" : "выключен"}`);
|
||||
loadStats();
|
||||
};
|
||||
const del = row.querySelector(".del");
|
||||
if (s.custom) del.onclick = async () => {
|
||||
if (!confirm(`удалить источник ${s.name}?`)) return;
|
||||
await api("/api/sources/remove", {name: s.name});
|
||||
toast("удалён"); loadSources(); loadStats();
|
||||
};
|
||||
box.appendChild(row);
|
||||
});
|
||||
}
|
||||
|
||||
$("#sq").oninput = (() => { let h; return () => { clearTimeout(h); h = setTimeout(() => { STATE.sq = $("#sq").value.trim(); loadSources(); }, 240); }; })();
|
||||
$("#scat").onchange = () => { STATE.scat = $("#scat").value; loadSources(); };
|
||||
|
||||
async function runCheck(body, label){
|
||||
const {job} = await api("/api/sources/check", body);
|
||||
const p = $("#sprog"); p.classList.remove("hide");
|
||||
const tick = setInterval(async () => {
|
||||
const j = await api("/api/job/" + job);
|
||||
if (j.state === "running") {
|
||||
p.textContent = `${label}: ${j.done || 0}/${j.total || "?"} · живых ${j.alive || 0} · мёртвых ${j.dead || 0} · ${j.current || ""}`;
|
||||
return;
|
||||
}
|
||||
clearInterval(tick);
|
||||
p.textContent = `готово: живых ${j.alive || 0}, мёртвых ${j.dead || 0}`;
|
||||
setTimeout(() => p.classList.add("hide"), 6000);
|
||||
loadSources(); loadStats();
|
||||
}, 1300);
|
||||
}
|
||||
$("#checkAll").onclick = () => runCheck({all: true}, "проверка всех");
|
||||
$("#checkCat").onclick = () => {
|
||||
const c = STATE.scat === "all" ? "" : STATE.scat;
|
||||
if (!c) return toast("выбери категорию в селекте");
|
||||
runCheck({category: c}, "проверка " + c);
|
||||
};
|
||||
$("#onAll").onclick = async () => {
|
||||
const d = await api(`/api/sources?category=${encodeURIComponent(STATE.scat === "all" ? "" : STATE.scat)}&q=${encodeURIComponent(STATE.sq)}`);
|
||||
await Promise.all(d.items.map(s => api("/api/sources/toggle", {name: s.name, enabled: true})));
|
||||
toast(`включено: ${d.items.length}`); loadSources(); loadStats();
|
||||
};
|
||||
$("#offAll").onclick = async () => {
|
||||
const d = await api(`/api/sources?category=${encodeURIComponent(STATE.scat === "all" ? "" : STATE.scat)}&q=${encodeURIComponent(STATE.sq)}`);
|
||||
await Promise.all(d.items.map(s => api("/api/sources/toggle", {name: s.name, enabled: false})));
|
||||
toast(`выключено: ${d.items.length}`); loadSources(); loadStats();
|
||||
};
|
||||
$("#add").onclick = async () => {
|
||||
const name = $("#aname").value.trim().toLowerCase().replace(/\s+/g, "_");
|
||||
const url = $("#aurl").value.trim();
|
||||
const category = $("#acat").value;
|
||||
if (!name || !url) return toast("нужны имя и url");
|
||||
const r = await api("/api/sources/add", {name, url, category});
|
||||
if (r.error) return toast(r.error);
|
||||
toast(`подключён: ${r.name} (${r.adapter})`);
|
||||
$("#aname").value = ""; $("#aurl").value = "";
|
||||
loadSources(); loadStats();
|
||||
};
|
||||
|
||||
/* ---------- старт ---------- */
|
||||
loadStats().then(loadItems);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user