абузы: free-панели new-api на вкладке (роут /api/panels + грид), CSS вынесен из JS, панельный классификатор даёт freeapi-вес, тесты
This commit is contained in:
Vlad Gothbreach
174
omni/sources_quality.py
Normal file
174
omni/sources_quality.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""Quality-волна источников: официальные release-фиды проектов и security-адвайзори вендоров.
|
||||
|
||||
Зачем отдельный модуль: основной реестр (sources.py) и волны ext/fresh уже покрывают
|
||||
топовые репозитории, но провал в (1) runtime/инференсе второго эшелона и (2) security-
|
||||
продуктах, где релиз = факт CVE/патча. Здесь только официальные Atom/RSS-фиды
|
||||
(github.com/<repo>/releases.atom, linux-дистрибутивные и вендорные advisory-фиды),
|
||||
каждая запись проверена живым прогоном.
|
||||
|
||||
Модуль статический и НЕ импортирует .sources: sources.py уже импортирует волны,
|
||||
обратный импорт дал бы цикл. Дедуп по url/name делает all_sources().
|
||||
|
||||
Категории — только существующие в реестре, без новой схемы:
|
||||
models — релизы AI-runtime/инференса (diffusers, TensorRT-LLM, onnxruntime, ...);
|
||||
community — SDK/агентные фреймворки (MCP python-sdk, openai-agents, langgraph);
|
||||
security — продукты, где релиз = патч/CVE, контейнеры и advisory-фиды дистрибутивов;
|
||||
company — официальные технические блоги вендоров (NVIDIA, HashiCorp).
|
||||
Признак «это релиз» остаётся тегом `release` у всех 37 записей, kind=atom/rss.
|
||||
|
||||
Проверка (tools/probe_quality_sources.py, отчёт data/quality_sources_probe.json):
|
||||
* каждое имя и URL сверены с полным реестром all_sources() (≈3.5k источников) — ни одного
|
||||
дубля по имени, нормализованному URL и зеркалу репозитория (github.com/<o>/<r>);
|
||||
* первые записи не пересекаются по URL/заголовкам с другими кандидатами (>=50%/60%);
|
||||
* >= 2 записей с валидным http(s)-URL, без challenge/error-заголовков;
|
||||
* штатный адаптер feeds.fetch (rss, Atom поддерживается) отдаёт 10-20 записей.
|
||||
|
||||
Осознанно НЕ включено, хотя фиды живые (шум, не качество): pytorch/pytorch и
|
||||
pydantic/pydantic-ai (в releases.atom CI-теги viable/strict, clai2-bleeding),
|
||||
microsoft/DeepSpeed (nightly-last-green), keycloak отдаёт nightly-теги (оставлен —
|
||||
версии рядом), portainer/immich/netbox/nextcloud/gitea/elastic/opensearch/minio
|
||||
(низкий сигнал на релиз, не security), php.net (фид отдаёт 1 запись), а также
|
||||
зеркала и медиа-перепечатки (Cisco Talos, MSRC blog, PortSwigger, Cloudflare,
|
||||
Ubuntu USN, Rust/Go/Node/Python blogs) — они уже в реестре или дублируют его.
|
||||
"""
|
||||
|
||||
SPECS = [
|
||||
# ---------- AI runtime / inference (официальные релизы) ----------
|
||||
{"name": "gh_rel_diffusers", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/huggingface/diffusers/releases.atom",
|
||||
"tags": ("release", "github", "ai", "image"), "limit": 10},
|
||||
{"name": "gh_rel_peft", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/huggingface/peft/releases.atom",
|
||||
"tags": ("release", "github", "ai", "finetune"), "limit": 10},
|
||||
{"name": "gh_rel_accelerate", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/huggingface/accelerate/releases.atom",
|
||||
"tags": ("release", "github", "ai", "training"), "limit": 10},
|
||||
{"name": "gh_rel_textgenerationinference", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/huggingface/text-generation-inference/releases.atom",
|
||||
"tags": ("release", "github", "ai", "inference"), "limit": 10},
|
||||
{"name": "gh_rel_tokenizers", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/huggingface/tokenizers/releases.atom",
|
||||
"tags": ("release", "github", "ai", "nlp"), "limit": 10},
|
||||
{"name": "gh_rel_candle", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/huggingface/candle/releases.atom",
|
||||
"tags": ("release", "github", "ai", "rust"), "limit": 10},
|
||||
{"name": "gh_rel_tensorrtllm", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/NVIDIA/TensorRT-LLM/releases.atom",
|
||||
"tags": ("release", "github", "ai", "nvidia", "inference"), "limit": 10},
|
||||
{"name": "gh_rel_nvidiacontainertoolkit", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/NVIDIA/nvidia-container-toolkit/releases.atom",
|
||||
"tags": ("release", "github", "gpu", "container"), "limit": 10},
|
||||
{"name": "gh_rel_flashinfer", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/flashinfer-ai/flashinfer/releases.atom",
|
||||
"tags": ("release", "github", "ai", "inference"), "limit": 10},
|
||||
{"name": "gh_rel_onnxruntime", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/microsoft/onnxruntime/releases.atom",
|
||||
"tags": ("release", "github", "ai", "inference"), "limit": 10},
|
||||
{"name": "gh_rel_mlx", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/ml-explore/mlx/releases.atom",
|
||||
"tags": ("release", "github", "ai", "apple"), "limit": 10},
|
||||
{"name": "gh_rel_whispercpp", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/ggml-org/whisper.cpp/releases.atom",
|
||||
"tags": ("release", "github", "ai", "audio"), "limit": 10},
|
||||
{"name": "gh_rel_exo", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/exo-explore/exo/releases.atom",
|
||||
"tags": ("release", "github", "ai", "local"), "limit": 10},
|
||||
{"name": "gh_rel_vllmascend", "category": "models", "kind": "atom",
|
||||
"url": "https://github.com/vllm-project/vllm-ascend/releases.atom",
|
||||
"tags": ("release", "github", "ai", "inference"), "limit": 10},
|
||||
|
||||
# ---------- AI-агенты / SDK ----------
|
||||
{"name": "gh_rel_mcp_pythonsdk", "category": "community", "kind": "atom",
|
||||
"url": "https://github.com/modelcontextprotocol/python-sdk/releases.atom",
|
||||
"tags": ("release", "github", "ai", "mcp"), "limit": 10},
|
||||
{"name": "gh_rel_openaiagentspython", "category": "community", "kind": "atom",
|
||||
"url": "https://github.com/openai/openai-agents-python/releases.atom",
|
||||
"tags": ("release", "github", "ai", "agent"), "limit": 10},
|
||||
{"name": "gh_rel_langgraph", "category": "community", "kind": "atom",
|
||||
"url": "https://github.com/langchain-ai/langgraph/releases.atom",
|
||||
"tags": ("release", "github", "ai", "agent"), "limit": 10},
|
||||
|
||||
# ---------- security-продукты: релиз = патч/CVE ----------
|
||||
{"name": "gh_rel_caddy", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/caddyserver/caddy/releases.atom",
|
||||
"tags": ("security", "release", "github", "web"), "limit": 10},
|
||||
{"name": "gh_rel_keycloak", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/keycloak/keycloak/releases.atom",
|
||||
"tags": ("security", "release", "github", "iam"), "limit": 10},
|
||||
{"name": "gh_rel_grafana", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/grafana/grafana/releases.atom",
|
||||
"tags": ("security", "release", "github", "monitoring"), "limit": 10},
|
||||
{"name": "gh_rel_tailscale", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/tailscale/tailscale/releases.atom",
|
||||
"tags": ("security", "release", "github", "vpn"), "limit": 10},
|
||||
{"name": "gh_rel_envoy", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/envoyproxy/envoy/releases.atom",
|
||||
"tags": ("security", "release", "github", "proxy"), "limit": 10},
|
||||
{"name": "gh_rel_vault", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/hashicorp/vault/releases.atom",
|
||||
"tags": ("security", "release", "github", "secrets"), "limit": 10},
|
||||
{"name": "gh_rel_cosign", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/sigstore/cosign/releases.atom",
|
||||
"tags": ("security", "release", "github", "supply-chain"), "limit": 10},
|
||||
{"name": "gh_rel_trivy", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/aquasecurity/trivy/releases.atom",
|
||||
"tags": ("security", "release", "github", "scanner"), "limit": 10},
|
||||
{"name": "gh_rel_grype", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/anchore/grype/releases.atom",
|
||||
"tags": ("security", "release", "github", "scanner"), "limit": 10},
|
||||
{"name": "gh_rel_nuclei", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/projectdiscovery/nuclei/releases.atom",
|
||||
"tags": ("security", "release", "github", "scanner"), "limit": 10},
|
||||
{"name": "gh_rel_zaproxy", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/zaproxy/zaproxy/releases.atom",
|
||||
"tags": ("security", "release", "github", "web"), "limit": 10},
|
||||
{"name": "gh_rel_openssl", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/openssl/openssl/releases.atom",
|
||||
"tags": ("security", "release", "github", "crypto"), "limit": 10},
|
||||
{"name": "gh_rel_curl", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/curl/curl/releases.atom",
|
||||
"tags": ("security", "release", "github", "http"), "limit": 10},
|
||||
|
||||
# ---------- контейнеры/оркестрация: CVE-релевантные релизы ----------
|
||||
{"name": "gh_rel_runc", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/opencontainers/runc/releases.atom",
|
||||
"tags": ("security", "release", "github", "container"), "limit": 10},
|
||||
{"name": "gh_rel_containerd", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/containerd/containerd/releases.atom",
|
||||
"tags": ("security", "release", "github", "container"), "limit": 10},
|
||||
{"name": "gh_rel_kubernetes", "category": "security", "kind": "atom",
|
||||
"url": "https://github.com/kubernetes/kubernetes/releases.atom",
|
||||
"tags": ("security", "release", "github", "kubernetes"), "limit": 10},
|
||||
|
||||
# ---------- официальные advisory/technical фиды вендоров ----------
|
||||
{"name": "amazon_linux_alas", "category": "security", "kind": "rss",
|
||||
"url": "https://alas.aws.amazon.com/alas.rss",
|
||||
"tags": ("security", "release", "advisory", "linux", "aws"), "limit": 20},
|
||||
{"name": "aws_security_bulletins", "category": "security", "kind": "rss",
|
||||
"url": "https://aws.amazon.com/security/security-bulletins/rss/feed/",
|
||||
"tags": ("security", "release", "advisory", "aws"), "limit": 20},
|
||||
{"name": "nvidia_dev_blog", "category": "company", "kind": "rss",
|
||||
"url": "https://developer.nvidia.com/blog/feed/",
|
||||
"tags": ("release", "vendor", "nvidia", "ai"), "limit": 15},
|
||||
{"name": "hashicorp_blog", "category": "company", "kind": "rss",
|
||||
"url": "https://www.hashicorp.com/blog/feed.xml",
|
||||
"tags": ("release", "vendor", "hashicorp"), "limit": 15},
|
||||
]
|
||||
|
||||
NAMES = tuple(d["name"] for d in SPECS)
|
||||
|
||||
|
||||
def quality_sources(S, specs=None):
|
||||
"""Собрать Source-объекты quality-волны через переданный конструктор S.
|
||||
|
||||
S передаётся снаружи (обычно sources.Source), чтобы модуль не импортировал .sources —
|
||||
это дал бы циклический импорт. Вызов позиционный, как в sources_fresh.fresh: так
|
||||
работает и настоящий Source, и лёгкая заглушка `class _Fake: __init__(self, *a, **k)`,
|
||||
которой tools/run_fresh.py --wave quality собирает имена.
|
||||
Теги всегда кортежи; adapter по умолчанию rss — он же разбирает Atom.
|
||||
"""
|
||||
out = []
|
||||
for d in (SPECS if specs is None else specs):
|
||||
out.append(S(d["name"], d["category"], d["kind"], d["url"],
|
||||
d.get("adapter", "rss"), tuple(d.get("tags") or ()), d.get("limit", 10)))
|
||||
return out
|
||||
Reference in New Issue
Block a user