AIStoreHub Package Vault: Clips Studio
Package Slug: clips-studio
Archetype: tool
Upstream Repository: https://github.com/ColinGPT9/clips-studio
Workspace Files & Specifications (68 files)
- _clipskitty_script_host.py (13231 bytes)
"""Run a pipeline's Python script on the Python inside Clips Kitty. The installed app has no python.exe: its engine is Python frozen by PyInstaller into api.exe. A pipeline's manifest says `run: ["{python}", "src/main.py"]`, and in the installed app {python} is api.exe itself, started with CLIPSKITTY_SCRIPT_HOST=1 (sdk/python/clipskitty_sdk/host.py plugin_env). main.py sees that marker before it imports anything heavy and hands the command line to main() here, which behaves like a small `python`: api.exe [-u] [-B] ... script.py [args] run a script as __main__ api.exe -c "code" [args] run a string api.exe -m module [args] run a module as __main__ api.exe --multiprocessing-fork ... a multiprocessing child What this takes care of, because a frozen interpreter ignores PYTHONPATH, PYTHONUNBUFFERED and the rest of the PYTHON* variables: - sys.path: the script's folder, then each PYTHONPATH entry (the SDK and the pipeline's own folders), ahead of everything bundled with the app; - output is line-buffered UTF-8, so progress lines reach Clips Kitty at once; - Clips Kitty's own packages (core, plugins, video...) can't be imported from the bundle: they are not part of the plugin API, they change with every release, and keeping them out keeps a pipeline a separate program (D13). A pipeline's own folder of the same name still wins; - a module the app doesn't include stops the run with a plain error line, and so does a part of the SDK this app's copy doesn't have yet (the pipeline was made with a newer SDK); - multiprocessing works: a child started with --multiprocessing-fork loads the parent's script as __mp_main__ first, which CPython's spawn does not do for a frozen Windows program. It uses the standard library only and imports nothing from the engine. """ from __future__ import annotations import importlib.abc import importlib.machinery import json import os import runpy import sys import traceback import types MARKER = "CLIPSKITTY_SCRIPT_HOST" SCRIPT = "CLIPSKITTY_SCRIPT" # the running script, for multiprocessing children # Clips Kitty's own top-level packages and modules. tests/test_script_host.py # checks this against the repository, so a new engine package can't be missed. ENGINE_PACKAGES = frozenset({ "analysis", "campaigns", "connected_accounts", "core", "creator", "gaming", "llm", "longform", "main", "multilingual", "plugins", "publish", "remote_render", "server", "sources", "sports", "stream_workflows", "third_party", "transcription", "video", "video_editor", "_clipskitty_script_host", }) # Engine commands (main.py). With one of these first, the command line is the # engine's even when the marker is set, so a stray marker can never turn # `api.exe serve` into something else. ENGINE_COMMANDS = frozenset({ "process", "run", "status", "outro-backfill", "auth", "upload", "serve", "mcp", "channels", "models", "render-worker", "--config", "-h", "--help", }) # Interpreter options a tool may add when it starts sys.executable again # (subprocess, multiprocessing). They change nothing here; they are accepted # so the command line around them still works. _FLAGS = set("uBEIsSOqbdPRv") _WITH_VALUE = {"W", "X"} class UsageError(Exception): pass def wants_host(argv: list[str]) -> bool: """True when main.py should hand this command line to main().""" if os.environ.get(MARKER) != "1" or len(argv) < 2: return False return argv[1] not in ENGINE_COMMANDS def parse(argv: list[str]) -> tuple[str, str | None, list[str]]: """(mode, target, rest) from a python-style command line (without the program name). mode is "script", "code", "module" or "fork".""" i = 0 while i < len(argv): arg = argv[i] if arg == "--multiprocessing-fork": return "fork", None, argv[i:] if arg == "-c" or arg == "-m": if i + 1 >= len(argv): raise UsageError(f"{arg} needs a value") return ("code" if arg == "-c" else "module"), argv[i + 1], argv[i + 2:] if arg.startswith("-c") or arg.startswith("-m"): return ("code" if arg[1] == "c" else "module"), arg[2:], argv[i + 1:] if arg == "--": i += 1 continue if arg.startswith("-") and arg != "-": letters = arg[1:] j = 0 while j < len(letters): ch = letters[j] if ch in _WITH_VALUE: if j + 1 < len(letters): # -Xutf8 break i += 1 # -X utf8 if i >= len(argv): raise UsageError(f"-{ch} needs a value") break if ch in "cm": mode = "code" if ch == "c" else "module" value = letters[j + 1:] if value: return mode, value, argv[i + 1:] if i + 1 >= len(argv): raise UsageError(f"-{ch} needs a value") return mode, argv[i + 1], argv[i + 2:] if ch not in _FLAGS: raise UsageError(f"option {arg!r} isn't supported by Clips Kitty's Python") j += 1 i += 1 continue if arg == "-": raise UsageError("reading a script from standard input isn't supported by Clips Kitty's Python") return "script", arg, argv[i + 1:] raise UsageError("no script to run") def _python_path() -> list[str]: return [p for p in os.environ.get("PYTHONPATH", "").split(os.pathsep) if p] def _front(paths: list[str]) -> None: """Put `paths` at the front of sys.path, in order, without duplicates.""" for p in reversed(paths): while p in sys.path: sys.path.remove(p) sys.path.insert(0, p) def _engine_roots() -> list[str]: """Where the engine's own code lives: the bundle in the installed app, the checkout's root in a source checkout.""" roots = [os.path.dirname(os.path.abspath(__file__))] bundle = getattr(sys, "_MEIPASS", None) if bundle: roots.append(os.path.abspath(bundle)) return roots def _under(path: str, roots: list[str]) -> bool: full = os.path.abspath(path or os.curdir) return any(full == r or full.startswith(r.rstrip(os.sep) + os.sep) for r in roots) class EngineGuard(importlib.abc.MetaPathFinder): """Refuses Clips Kitty's own packages unless the pipeline's own folders have a module of that name, which then wins. The pipeline's folders are the ones it was started with plus any sys.path entry outside the engine's own (a pipeline may add a folder of its own at run time).""" def __init__(self, own_paths: list[str]): self.own_paths = list(own_paths) self.engine_roots = _engine_roots() def find_spec(self, fullname, path=None, target=None): top = fullname.partition(".")[0] if top not in ENGINE_PACKAGES or path is not None: return None # not ours, or a submodule of something already allowed search = self.own_paths + [p for p in sys.path if isinstance(p, str) and p not in self.own_paths and not _under(p, self.engine_roots)] spec = importlib.machinery.PathFinder.find_spec(fullname, search) if spec is not None: return spec raise ModuleNotFoundError( f"No module named {fullname!r}: Clips Kitty's own code isn't available to pipelines", name=fullname) def _setup_output() -> None: for name in ("stdout", "stderr"): stream = getattr(sys, name, None) if stream is None: setattr(sys, name, open(os.devnull, "w", encoding="utf-8")) continue try: stream.reconfigure(encoding="utf-8", errors="replace", line_buffering=True) except (AttributeError, ValueError, OSError): pass # not a text stream that can change (replaced, or a pipe that's gone): leave it as it is def _say_error(message: str) -> None: # The contract's error line (sdk/python/clipskitty_sdk/contract.py # error_line), written here so this module needs nothing from the SDK. try: print(json.dumps({"type": "error", "message": message}, ensure_ascii=False), flush=True) except Exception: pass # nowhere to report it: the exit code still says the run failed # The error line for a pipeline that imports a part of the SDK this app's # copy doesn't have. The SDK's run() says the same for an import inside # main() (sdk/python/clipskitty_sdk/job.py NEWER_VERSION); written here so # this module needs nothing from the SDK. NEWER_VERSION = ("This pipeline needs a newer version of Clips Kitty. Update Clips Kitty, or ask the pipeline's " "developer which version it needs.") def needs_newer_sdk(error: BaseException) -> bool: """An import of a module or name the bundled SDK doesn't have: `import clipskitty_sdk.media`, `from clipskitty_sdk import media` (name clipskitty_sdk) or `from clipskitty_sdk.media import Region` (name clipskitty_sdk.media). The SDK's job.needs_newer_sdk is the same rule.""" name = getattr(error, "name", None) or "" return isinstance(error, ImportError) and (name == "clipskitty_sdk" or name.startswith("clipskitty_sdk.")) def missing_module_message(error: ModuleNotFoundError) -> str: name = (error.name or "").partition(".")[0] or "a module" if name in ENGINE_PACKAGES: return (f"This pipeline tried to use Clips Kitty's own code ({name}), which pipelines can't use. " "Ask its developer to update it.") return (f"This pipeline needs {name}, which this version of Clips Kitty doesn't include. " "Ask its developer to update it.") def _install_guard(own_paths: list[str]) -> None: sys.meta_path.insert(0, EngineGuard(own_paths)) def _run_main_code(code: str, filename: str, argv0: str, rest: list[str]) -> None: module = types.ModuleType("__main__") module.__dict__["__builtins__"] = __builtins__ module.__file__ = filename if filename != "<string>" else None if module.__file__ is None: del module.__file__ sys.modules["__main__"] = module sys.argv = [argv0, *rest] exec(compile(code, filename, "exec"), module.__dict__) def _fork_child(rest: list[str]) -> None: """A multiprocessing child: load the parent's script as __mp_main__ (as CPython's spawn does for an ordinary Python), then let multiprocessing take over. freeze_support() runs the child and exits.""" script = os.environ.get(SCRIPT) if script and os.path.isfile(script): _front([os.path.dirname(script)]) content = runpy.run_path(script, run_name="__mp_main__") main_module = types.ModuleType("__mp_main__") main_module.__dict__.update(content) sys.modules["__mp_main__"] = sys.modules["__main__"] = main_module sys.argv = [sys.argv[0], *rest] import multiprocessing multiprocessing.freeze_support() raise UsageError("--multiprocessing-fork without a multiprocessing parent") def main(argv: list[str]) -> int: """Run the command line `argv` (without the program name). Returns the exit code; a SystemExit from the script passes through untouched.""" _setup_output() try: mode, target, rest = parse(argv) except UsageError as e: _say_error(f"Clips Kitty's Python: {e}") return 2 own = _python_path() script = os.path.abspath(target) if mode == "script" else "" if mode == "script": own = [os.path.dirname(script), *own] os.environ[SCRIPT] = script elif mode == "fork": parent = os.environ.get(SCRIPT) own = ([os.path.dirname(parent)] if parent else []) + own else: os.environ.pop(SCRIPT, None) # a -c or -m child must not reload some other script own = [os.getcwd(), *own] _front(own) _install_guard(own) try: if mode == "fork": _fork_child(rest) elif mode == "code": _run_main_code(target, "<string>", "-c", rest) elif mode == "module": sys.argv = [target, *rest] runpy.run_module(target, run_name="__main__", alter_sys=True) else: if not os.path.isfile(script): _say_error(f"Clips Kitty's Python: can't find the script {target!r}") return 2 sys.argv = [target, *rest] runpy.run_path(script, run_name="__main__") except SystemExit: raise except UsageError as e: _say_error(f"Clips Kitty's Python: {e}") return 2 except ImportError as e: traceback.print_exc() if needs_newer_sdk(e): _say_error(NEWER_VERSION) elif isinstance(e, ModuleNotFoundError): _say_error(missing_module_message(e)) return 1 # any other ImportError is a crash in the script, as below except Exception: # a crash in the script is exit code 1, as in python; Ctrl+C passes through traceback.print_exc() return 1 return 0 - aistorehub.json (832 bytes)
{ "name": "Clips Studio", "slug": "clips-studio", "version": "1.0.0", "type": "tool", "description": "Clips Kitty: Free open-source AI video clipper. Local Opus Clip alternative (also Klap & Vizard) that turns Twitch, Kick & YouTube streams into auto-captioned Shorts, Reels, and TikToks, and can auto-post them 24/7 from a mini PC. Local AI (Ollama, Whisper): no usage or subscription fees. Or OpenRouter: a 2-hour VOD cost 11 cents in my testing.", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/ColinGPT9/clips-studio", "tags": [ "clips", "creator-tools", "content-creator", "python", "opus-clip-alternative", "instagram-reels", "rea", "short-form-video", "clip-generator", "agent" ], "imported_at": "2026-11-21T23:56:27.228Z" } - analysis/__init__.py (0 bytes)
- analysis/audio_features.py (3654 bytes)
"""Audio excitement signals, fully local (FFmpeg decode -> numpy). Produces per-second feature arrays over the whole video: loudness - RMS energy per second spike - how loud this second is vs. the rolling 30s median (shouts, cheers, hype moments) burst - density of sudden energy onsets within the second (laughter, applause, rapid-fire excitement) noisiness - zero-crossing rate (broadband/noisy audio like laughter and crowd noise scores high; clean speech scores low) All arrays are raw values here; fusion.py normalizes them to per-video percentiles. No ML models involved — this is signal processing, which is exactly why it's fast (~seconds for a 30-minute video). """ from pathlib import Path import numpy as np from core import cancel from core.binaries import ffmpeg SAMPLE_RATE = 16000 FRAME = SAMPLE_RATE // 20 # 50 ms analysis frames def extract_audio_features(video_path: Path) -> dict[str, np.ndarray]: """Returns {feature_name: array indexed by second}. Empty dict if the video has no audio stream.""" pcm = _decode_mono_pcm(video_path) if pcm.size < SAMPLE_RATE: return {} # 50 ms frame energies are the working unit for everything below. n_frames = pcm.size // FRAME frames = pcm[: n_frames * FRAME].reshape(n_frames, FRAME).astype(np.float32) frame_rms = np.sqrt((frames**2).mean(axis=1)) frames_per_sec = SAMPLE_RATE // FRAME n_secs = n_frames // frames_per_sec if n_secs == 0: return {} per_sec = frame_rms[: n_secs * frames_per_sec].reshape(n_secs, frames_per_sec) loudness = per_sec.mean(axis=1) # Spike: this second vs. rolling 30s median loudness (clamped ratio). spike = loudness / np.maximum(_rolling_median(loudness, 30), 1e-6) spike = np.clip(spike, 0, 5) # Burst: count of frame-level onsets (energy jumping >2x over the # previous frame) per second — laughter and applause are onset-dense. onsets = frame_rms[1:] > 2.0 * np.maximum(frame_rms[:-1], 1e-6) onsets = np.concatenate([[False], onsets]) burst = ( onsets[: n_secs * frames_per_sec] .reshape(n_secs, frames_per_sec) .sum(axis=1) .astype(np.float32) ) # Noisiness: zero-crossing rate per second. signs = np.signbit(frames) frame_zcr = (signs[:, 1:] != signs[:, :-1]).mean(axis=1) noisiness = ( frame_zcr[: n_secs * frames_per_sec].reshape(n_secs, frames_per_sec).mean(axis=1) ) # Only count noisiness when something is actually audible. audible = loudness > max(loudness.mean() * 0.3, 1e-6) noisiness = noisiness * audible return { "loudness": loudness, "spike": spike, "burst": burst, "noisiness": noisiness.astype(np.float32), } def _decode_mono_pcm(video_path: Path) -> np.ndarray: cmd = [ ffmpeg(), "-v", "error", "-i", str(video_path), "-vn", "-ac", "1", "-ar", str(SAMPLE_RATE), "-f", "s16le", "-", ] result = cancel.run(cmd) # stops if the video is cancelled (#113) if result.returncode != 0: raise RuntimeError(f"ffmpeg audio decode failed:\n{result.stderr[-1000:].decode(errors='replace')}") return np.frombuffer(result.stdout, dtype=np.int16).astype(np.float32) / 32768.0 def _rolling_median(x: np.ndarray, window: int) -> np.ndarray: """Median of a centered window at each position (edges shrink).""" half = window // 2 out = np.empty_like(x) for i in range(x.size): lo, hi = max(0, i - half), min(x.size, i + half + 1) out[i] = np.median(x[lo:hi]) return out - analysis/chat_moments.py (9308 bytes)
"""What chat's reactions say happened, for the gaming profile. Chat replay says more than how many people were talking (analysis/hype.py's unique-chatter curve, which standard scoring keeps): a burst of messages far above the stream's own rate marks a moment, and what they say marks what kind of moment it was. A wall of POG / NO WAY is a big play, KEKW / LUL a funny one, monkaS a scare, F / L a fail, "clip it" viewers asking for exactly this. (Stream-highlight research and clip tools agree on both; docs/GAMING.md.) Chat reacts after the moment, by the stream's delay plus typing time, so a spike is dated config/gaming.yaml's chat_lag_seconds earlier. Everything is measured against this stream's own chat, so a small channel's burst of twelve messages counts like a big channel's burst of three hundred. """ import re from dataclasses import dataclass, field import numpy as np WINDOW = 5 # seconds a burst is counted over BASELINE = 45 # seconds of chat before it that set the normal rate SPIKE = 3.0 # a burst this many times the normal rate is a moment MIN_BURST = 4 # ...and at least this many messages # A burst with no reaction in it (no emote or word a class knows) is weaker # evidence, so it has to stand out more. PLAIN_SPIKE = 4.0 PLAIN_MIN_BURST = 6 MAX_EVENTS = 60 MERGE_SECONDS = 10 # bursts this close together are one moment @dataclass class ChatSignal: """curve: per second 0..1, how strongly chat marks a moment there (already moved earlier for chat's lag). events: (second, "CHAT: ...") for the model.""" curve: np.ndarray events: list = field(default_factory=list) messages: int = 0 _TOKEN = re.compile(r"[\w+':]+") _CHANNEL_EMOTE = re.compile(r"[a-z][a-z0-9]{1,24}([A-Z0-9][\w]*)") def _matchers(classes: dict) -> dict: """Per class: exact whole-word tokens, phrases / emoji found anywhere, channel-emote suffixes and regular expressions (config/gaming.yaml).""" out = {} for key, spec in (classes or {}).items(): tokens, phrases = set(), [] for w in spec.get("words") or []: w = str(w) if " " in w or not re.fullmatch(r"[\w+':]+", w) or not w.isascii(): phrases.append(w.lower()) # phrases, emoji and other scripts else: tokens.add(w) suffixes = {str(x) for x in spec.get("suffixes") or []} patterns = [re.compile(str(p), re.IGNORECASE) for p in spec.get("patterns") or []] whole = [re.compile(str(p), re.IGNORECASE) for p in spec.get("message_patterns") or []] out[key] = (tokens, phrases, spec.get("label", key), suffixes, patterns, whole) return out def classify(text: str, matchers: dict) -> dict: """The classes one message belongs to, with the words that put it there.""" found = {} tokens = set(_TOKEN.findall(text or "")) pieces = set((text or "").split()) | tokens low = (text or "").lower() emote_ends = {m.group(1): tok for tok in tokens if (m := _CHANNEL_EMOTE.fullmatch(tok))} message = (text or "").strip() for key, (words, phrases, _label, suffixes, patterns, whole) in matchers.items(): hits = [w for w in words if w in tokens] + [p for p in phrases if p in low] hits += [tok for end, tok in emote_ends.items() if end in suffixes] hits += [p for p in pieces for rx in patterns if rx.search(p)][:3] hits += [message for rx in whole if rx.search(message)][:1] if hits: found[key] = hits return found def _short(text: str) -> bool: """A reaction-length message (an emote, "www", "no way"), as opposed to someone writing a question or an opinion.""" t = (text or "").strip() return len(t) <= 14 or len(t.split()) <= 3 def _only(text: str, ignore: set) -> bool: """A message made only of hellos / good lucks (config/gaming.yaml chat_ignore).""" low = (text or "").strip().lower() if not low: return False if low in ignore: return True words = re.findall(r"[\w']+", low) return bool(words) and all(w in ignore for w in words) def chat_signal(messages: list, duration: float, classes: dict, lag: float = 6.0, ignore: list | None = None) -> ChatSignal | None: """messages: [(offset seconds, author, text)]. None when there is too little chat to tell anything from (under one message a minute). `ignore`: words a message made only of isn't a reaction (hellos, good lucks).""" n = int(duration) + 1 if n < WINDOW * 4 or not messages: return None if len(messages) < duration / 60: return None matchers = _matchers(classes) ignored = {str(w).lower() for w in (ignore or [])} rate = np.zeros(n, dtype=np.float32) short = np.zeros(n, dtype=np.float32) per_class = {k: np.zeros(n, dtype=np.float32) for k in matchers} words_at: dict[int, dict] = {} for t, _who, text in messages: s = int(t) if not 0 <= s < n: continue found = classify(text, matchers) if not found and ignored and _only(text, ignored): continue # a hello isn't a reaction to anything rate[s] += 1 if _short(text): short[s] += 1 for key, hits in found.items(): per_class[key][s] += 1 bucket = words_at.setdefault(s, {}) for h in hits: bucket[h] = bucket.get(h, 0) + 1 idx = np.arange(n) def window_sum(x: np.ndarray, width: int) -> np.ndarray: """Messages in the `width` seconds ending at t: looking back only, so a burst is dated where it starts, not seconds before it.""" c = np.concatenate([[0.0], np.cumsum(x, dtype=np.float64)]) return (c[idx + 1] - c[np.maximum(idx + 1 - width, 0)]).astype(np.float32) burst = window_sum(rate, WINDOW) # messages in (t-5, t] # The normal rate: messages per 5 s over the BASELINE seconds before that # window, floored so a near-silent chat doesn't turn three messages into # a spike. before = np.concatenate([[0.0], np.cumsum(rate, dtype=np.float64)]) hi = np.maximum(idx + 1 - WINDOW, 0) lo = np.maximum(hi - BASELINE, 0) span = np.maximum(hi - lo, 1) normal = ((before[hi] - before[lo]) / span * WINDOW).astype(np.float32) floor = max(float(np.median(burst)), 1.0) ratio = burst / np.maximum(normal, floor) reacting = sum(window_sum(v, WINDOW) for v in per_class.values()) share = np.clip(reacting / np.maximum(burst, 1.0), 0.0, 1.0) brief = np.clip(window_sum(short, WINDOW) / np.maximum(burst, 1.0), 0.0, 1.0) # 0 at the normal rate, 1 at twice the spike level. What chat says adds to # it (a burst of POGs is surer than a burst of anything), and so does how # it says it: a wall of short reactions is chat reacting to a moment, # long messages are chat discussing something. strength = (np.clip((ratio - 1.0) / (2 * SPIKE - 1.0), 0.0, 1.0) * (0.6 + 0.4 * share) * (0.5 + 0.5 * brief)) shift = max(0, int(round(lag))) curve = np.zeros(n, dtype=np.float32) curve[: n - shift] = strength[shift:] # Events: where a burst starts, dated back by the lag, named by what chat said. events = [] hot = (ratio >= SPIKE) & (burst >= MIN_BURST) s = 0 while s < n: if not hot[s]: s += 1 continue e = s while e + 1 < n and hot[e + 1]: e += 1 counts: dict = {} klass: dict = {} for sec in range(s, min(e + WINDOW, n)): for w, c in (words_at.get(sec) or {}).items(): counts[w] = counts.get(w, 0) + c for key in per_class: klass[key] = klass.get(key, 0) + int(per_class[key][sec]) top = sorted(counts.items(), key=lambda kv: -kv[1])[:3] kind = max(klass, key=klass.get) if klass and max(klass.values()) > 0 else None if kind is None and (ratio[s:e + 1].max() < PLAIN_SPIKE or burst[s:e + 1].max() < PLAIN_MIN_BURST or brief[s:e + 1].max() < 0.5): s = e + 1 # chat talking, not reacting: not a moment continue label = matchers[kind][2] if kind else "a burst of messages" said = ", ".join(f"{w} x{c}" for w, c in top) text = f"CHAT: {label}" + (f" ({said})" if said else f" ({int(burst[s:e + 1].max())} messages in {WINDOW}s)") events.append((max(0.0, float(s - shift)), text, float(ratio[s:e + 1].max()))) s = e + 1 # Bursts a few seconds apart are one moment chat kept reacting to: the # first names it, the strongest's words describe it. merged: list = [] for ev in sorted(events, key=lambda ev: ev[0]): if merged and ev[0] - merged[-1][3] <= MERGE_SECONDS: first = merged[-1] best = ev if ev[2] > first[2] else first merged[-1] = (first[0], best[1], max(first[2], ev[2]), ev[0]) else: merged.append((*ev, ev[0])) merged.sort(key=lambda ev: -ev[2]) events = sorted(((t, txt) for t, txt, _r, _last in merged[:MAX_EVENTS]), key=lambda ev: ev[0]) return ChatSignal(curve=curve, events=events, messages=len(messages)) - analysis/clip_edit.py (7739 bytes)
"""Conversational clip editing. Turns a plain-language request ("the crop is too far left", "make it 5 seconds longer", "the caption says 'gost', it should be 'ghost'") into the app's structured edit controls, using the same local LLM as the rest of the pipeline. Never raises into the request path — on any LLM trouble it returns a friendly no-op reply. """ import json import re from pathlib import Path from llm.base import LLMBackend PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "edit_clip.txt" _CROP_MODES = {"track", "center", "bias_left", "bias_right"} _POSITIONS = {"bottom", "middle", "top"} MIN_DURATION = 5.0 MAX_DURATION = 180.0 # matches the 60s+ monetization mode's upper bound def interpret_edit( message: str, *, clip_state: dict, caption_lines: list[dict], source_duration: float, llm: LLMBackend, ) -> dict: """Returns: { "reply": str, # what to show the user in chat "needs_render": bool, # whether a re-render job should be queued "start": float | None, # new clip start in the source (if changed) "end": float | None, # new clip end (if changed) "render_opts": dict, # crop / captions / caption_style / caption_lines } """ captions_text = "\n".join( f'{i}: [{l["start"]:.1f}-{l["end"]:.1f}] {l["text"]}' for i, l in enumerate(caption_lines) ) or "(no captions)" prompt = ( PROMPT_PATH.read_text(encoding="utf-8") .replace("{state}", json.dumps(clip_state, indent=2)) .replace("{captions}", captions_text) .replace("{source_duration}", f"{source_duration:.1f}") .replace("{message}", message.strip()) ) try: raw = llm.generate(prompt, json_mode=True) data = _parse(raw) except Exception: data = None if data is None: return { "reply": "Sorry, I couldn't work that out. Try things like " '"make it 5 seconds longer", "center the crop", "yellow captions", ' 'or "caption 3 should say ghost, not gost".', "needs_render": False, "start": None, "end": None, "render_opts": {}, } return _apply(data, clip_state, caption_lines, source_duration) def _apply( data: dict, clip_state: dict, caption_lines: list[dict], source_duration: float ) -> dict: render_opts: dict = {} start = float(clip_state.get("start", 0.0)) end = float(clip_state.get("end", 0.0)) new_start: float | None = None new_end: float | None = None # ---- timestamps ----------------------------------------------------- if _is_number(data.get("start")) or _is_number(data.get("end")): s = float(data["start"]) if _is_number(data.get("start")) else start e = float(data["end"]) if _is_number(data.get("end")) else end s, e = _clamp_window(s, e, source_duration) if abs(s - start) > 0.05: new_start = s if abs(e - end) > 0.05: new_end = e # ---- crop ----------------------------------------------------------- if isinstance(data.get("crop"), str) and data["crop"] in _CROP_MODES: render_opts["crop"] = data["crop"] # ---- color filter ---------------------------------------------------- filt = data.get("filter") if isinstance(filt, str): from video.filters import is_valid if is_valid(filt): render_opts["filter"] = filt # ---- manual picture adjustments --------------------------------------- adjust = data.get("adjust") if isinstance(adjust, dict): from video.filters import ADJUST_RANGES clean_adjust = {} for key, (lo, hi, _default) in ADJUST_RANGES.items(): if _is_number(adjust.get(key)): clean_adjust[key] = max(lo, min(hi, float(adjust[key]))) if clean_adjust: render_opts["adjust"] = clean_adjust # ---- captions on/off ------------------------------------------------ if isinstance(data.get("captions"), bool): render_opts["captions"] = data["captions"] # ---- caption style --------------------------------------------------- if isinstance(data.get("caption_style"), dict): clean = _clean_caption_style(data["caption_style"]) if clean: render_opts["caption_style"] = clean # ---- caption text corrections ---------------------------------------- fixes = data.get("caption_fixes") if isinstance(fixes, list) and fixes and caption_lines: fixed = [dict(l) for l in caption_lines] applied = False for fix in fixes: if not isinstance(fix, dict): continue idx = fix.get("line") text = fix.get("text") if _is_number(idx) and isinstance(text, str) and 0 <= int(idx) < len(fixed): fixed[int(idx)]["text"] = text.strip() applied = True if applied: # Timestamp edits would invalidate line indexes/times — caption # text fixes only apply when the window is unchanged. if new_start is None and new_end is None: render_opts["caption_lines"] = fixed needs_render = bool( data.get("needs_render", True) and (new_start is not None or new_end is not None or render_opts) ) reply = str(data.get("reply") or "").strip() if not reply: reply = "Done — re-rendering the clip now." if needs_render else "I didn't change anything." return { "reply": reply, "needs_render": needs_render, "start": new_start, "end": new_end, "render_opts": render_opts, } def _clean_caption_style(style: dict) -> dict: out: dict = {} if _is_number(style.get("font_size")): out["font_size"] = int(max(40, min(140, float(style["font_size"])))) color = style.get("color") if isinstance(color, str) and re.fullmatch(r"#?[0-9a-fA-F]{6}", color.strip()): out["color"] = "#" + color.strip().lstrip("#").upper() if isinstance(style.get("position"), str) and style["position"] in _POSITIONS: out["position"] = style["position"] font = style.get("font") if isinstance(font, str): from video.captions import FONTS match = next((f for f in FONTS if f.lower() == font.strip().lower()), None) if match: out["font"] = match if _is_number(style.get("words_per_caption")): out["words_per_caption"] = int(max(1, min(6, float(style["words_per_caption"])))) if isinstance(style.get("uppercase"), bool): out["uppercase"] = style["uppercase"] return out def _clamp_window(start: float, end: float, source_duration: float) -> tuple[float, float]: start = max(0.0, start) if source_duration > 0: end = min(source_duration, end) start = min(start, max(0.0, source_duration - MIN_DURATION)) if end - start < MIN_DURATION: end = start + MIN_DURATION if source_duration > 0: end = min(end, source_duration) if end - start > MAX_DURATION: end = start + MAX_DURATION return round(start, 2), round(end, 2) def _is_number(v) -> bool: return isinstance(v, (int, float)) and not isinstance(v, bool) def _parse(raw: str) -> dict | None: text = raw.strip() text = re.sub(r"^```(?:json)?\s*|\s*```$", "", text) s, e = text.find("{"), text.rfind("}") if s == -1 or e <= s: return None try: data = json.loads(text[s : e + 1]) except json.JSONDecodeError: return None return data if isinstance(data, dict) else None - analysis/fusion.py (54987 bytes)
"""Multimodal fusion: combines transcript, audio, visual, and reaction signals into one 0-100 clip score. final = w_text*text + w_visual*visual + w_reaction*reaction + w_audio*audio + w_engagement*engagement (weights in config) Candidates come from two pools: 1. transcript pool — Gemma's picks (analysis/highlights.py), now informed by an EVENTS timeline built from the signals 2. signal pool — windows where combined audio+visual excitement exceeds the configured percentile; their transcript text is scored by Gemma in one extra call so they compete on equal footing After dedup, the top finalists go through a RERANK pass: Gemma orders them against each other (relative judgment — far more reliable for small local models than absolute 0-100 scoring, which clusters). """ import json import re import time from pathlib import Path import numpy as np from analysis import highlights from analysis import intent as clip_intent from analysis.audio_features import extract_audio_features from analysis.visual_features import extract_visual_features, reaction_for_window from core import cancel, progress from core.models import ClipCandidate, Rejection, Segment from llm.base import LLMBackend, generate_json try: # Sports (sports/): used only when a job has a sport. Without the package # everything else here runs as it always has. from sports.core import clips as sport_clips from sports.core import detect as sport_detect from sports.core import select as sport_select except ImportError: sport_clips = sport_detect = sport_select = None # The shape rerank.txt asks for; held to it on a cloud model (llm.base.generate_json). ORDER_SCHEMA = { "type": "object", "properties": {"order": {"type": "array", "items": {"type": "integer"}}}, "required": ["order"], "additionalProperties": False, } RERANK_PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "rerank.txt" def find_clips( video_path: Path, segments: list[Segment], llm: LLMBackend, config: dict, signals: tuple[dict, dict] | None = None, creator_context=None, # creator.retrieval.CreatorContext | None weight_bias: dict | None = None, # per-channel multipliers from creator.learning audience: "np.ndarray | None" = None, # analysis.hype curve (chat/heatmap), 0..1 measure_reaction: bool = True, # False for a stream scored as gaming (core.modes.measures_reaction) gaming=None, # analysis.gaming.GamingProfile: score as a gaming stream chat=None, # analysis.chat_moments.ChatSignal: what chat's reactions mark sounds=None, # analysis.game_audio.GameSounds: what the game's own sound marks intent=None, # analysis.intent.ClipIntent: the person's direction; only ever adds sport=None, # sports.core.profile.SportProfile: a match, scored for its moments (sports/) ) -> tuple[list[ClipCandidate], list[Rejection]]: clips_cfg = config["clips"] analysis_cfg = config["analysis"] scoring_cfg = config.get("scoring", {}) weights = scoring_cfg.get( "weights", {"text": 0.30, "visual": 0.20, "reaction": 0.20, "audio": 0.20, "engagement": 0.10}, ) if sport is not None and gaming is None: # A match answers what the gaming path asks (sports/core/profile.py), # so its proven evidence (the voice jump, the crowd and whistle, # on-screen text, event windows, the capped bonus) runs with the # sport's data in it. gaming = sport # Clip selection (scoring.selection, analysis/highlights.py): 2 changes how # a standard run's candidates are found, scored and fitted. A gaming stream # and a match keep their own path, whatever it says. selection = highlights.selection_version(config) if gaming is None and sport is None else 1 if gaming is not None: # The standard weights (analysis/gaming.py): talk is judged as on any # stream, and the game adds on top (the game bonus below). weights = dict(gaming.weights) if weight_bias: # Learned from the user's own keep/edit/export behavior for THIS # creator — bounded (max 20% shift per channel) and renormalized. from creator.learning import apply_bias weights = apply_bias(weights, weight_bias) print(" Using learned scoring preferences for this creator") # ---- 1. extract + normalize signals -------------------------------- progress.emit(stage="signals") if signals is not None: # Precomputed by the pipeline in the background while Whisper was # transcribing — this stage costs nothing on that path. audio_raw, visual_raw = signals print(" Using audio/visual signals precomputed during transcription") else: print(" Extracting audio signals...") audio_raw = extract_audio_features(video_path) print(" Extracting visual signals...") visual_raw = extract_visual_features(video_path) audio_excitement = _combine([_pct(audio_raw[k]) for k in ("spike", "burst", "noisiness") if k in audio_raw]) visual_activity = _combine([_pct(visual_raw[k]) for k in ("motion", "scene_cut", "flash") if k in visual_raw]) combined = _combine([a for a in (audio_excitement, visual_activity) if a.size]) events = _build_events(audio_excitement, visual_activity, visual_raw) if events: print(f" {len(events)} notable audio/visual events detected") # ---- the gaming profile's evidence that something happened in the game -- # Only for a gaming stream; standard scoring never builds any of it. game_curve = np.zeros(0, dtype=np.float32) voice = np.zeros(0, dtype=np.float32) chat_curve = np.zeros(0, dtype=np.float32) game_sound = np.zeros(0, dtype=np.float32) people_sound = np.zeros(0, dtype=np.float32) screen = None guidance = "" events_title = None if gaming is not None: voice, voice_events = _voice_jump(audio_raw, segments) sources = [(voice, 0.6)] game_events = list(voice_events) if chat is not None: chat_curve = chat.curve sources.append((chat_curve, 0.9)) game_events += list(chat.events) if sounds is not None: # The game's sound (gunfire, a goal, a crash) is the game saying # so; a scream or laughter is mostly the people playing it. game_sound, people_sound = sounds.game, sounds.people sources += [(game_sound, 0.7), (people_sound, 0.5)] game_events += list(sounds.events) game_curve = _soft_or(sources) if scoring_cfg.get("read_screen", True) and game_curve.size and getattr(gaming, "reads_screen", True): # What the game writes on screen in those moments: an event's # banner, or a menu chat reacted to. screen = _read_screen(video_path, game_curve, segments, clips_cfg, gaming) if screen is not None: game_events += list(screen.events) game_events += [(s0, f"ON SCREEN: a menu, queue or settings screen ({words})") for s0, _e0, words in screen.menus] events = sorted(events + game_events, key=lambda ev: ev[0]) guidance = gaming.guidance("clips") # Per chunk when the stream changes game part way through. if len({str(g.get("name") or "") for g in gaming.games}) > 1: guidance = lambda start, end: gaming.guidance("clips", start, end) # noqa: E731 events_title = "GAME / CHAT / AUDIO EVENTS (from signal analysis):" print(f" {'Sports' if sport is not None else 'Gaming'}: scoring as a " f"{gaming.spec.get('label', 'game')}{'' if sport is not None else ' stream'}" + (f" ({gaming.game})" if gaming.game else "") + f": {len(chat.events) if chat is not None else 0} chat moment(s), " f"{len(sounds.events) if sounds is not None else 0} game sound(s), " f"{len(voice_events)} voice jump(s)") # ---- 2. candidate pools --------------------------------------------- candidates, _ = highlights.find_highlights( segments, llm, min_score=0, # fusion owns thresholding now max_clips=999, min_duration=clips_cfg["min_duration"], max_duration=clips_cfg["max_duration"], max_overlap=1.1, # fusion owns dedup too — keep all for scoring max_text_similarity=1.1, max_segment_reuse=1.1, chunk_seconds=analysis_cfg["chunk_seconds"], chunk_overlap_seconds=analysis_cfg["chunk_overlap_seconds"], long_video_threshold_seconds=analysis_cfg["long_video_threshold_seconds"], events=events, **({"guidance": guidance, "events_title": events_title} if gaming is not None else {}), **({"selection": 2} if selection == 2 else {}), ) # Candidate windows from signal peaks — detected PER MODALITY, not just # their mean. Averaging audio+visual hides content that is strong in only # ONE of them: a SILENT workout is visual-only, and the mean dilutes it # below the peak cutoff so it never becomes a candidate at all. Scanning # visual-alone and audio-alone peaks is what surfaces silent action (and # talk-free hype) for the scorer to judge. Dedup accumulates across all # three passes so the same moment isn't proposed twice. pk_pct = scoring_cfg.get("signal_peak_percentile", 90) peak_windows: list[tuple[float, float]] = [] seen = list(candidates) if selection == 2: # Only a pick that can make the cut covers its moment: one the model # scored near zero no longer hides a hot moment it overlaps. The picks # are already scored here; min_score is the run's own bar. seen = [c for c in candidates if c.score >= clips_cfg["min_score"]] # Audience hype (chat density / most-replayed) is a first-class peak # source: a moment chat exploded over is a candidate even if the # transcript and A/V signals missed it. peak_signals = [visual_activity, audio_excitement, combined] if audience is not None and audience.size: peak_signals.append(audience) if gaming is not None and game_curve.size: # Each in-game moment is a candidate of its own, from a few seconds # before it (the setup) to the reaction, even with nothing said: a # quiet streamer's best play has no transcript line to be found by. # First, so the moment's own window (with its setup) is the one kept. for win in _event_windows(game_curve, segments, clips_cfg["min_duration"], clips_cfg["max_duration"], existing=seen): peak_windows.append(win) seen.append(ClipCandidate(start=win[0], end=win[1], score=0)) for sig in peak_signals: for win in _signal_peak_windows( sig, segments, percentile=pk_pct, min_duration=clips_cfg["min_duration"], max_duration=clips_cfg["max_duration"], existing=seen, **({"video_end": max(segments[-1].end if segments else 0.0, float(sig.size))} if gaming is not None else {}), **({"strongest_first": True} if selection == 2 else {}), ): peak_windows.append(win) seen.append(ClipCandidate(start=win[0], end=win[1], score=0)) if peak_windows: print(f" {len(peak_windows)} signal-peak window(s) found beyond transcript picks " f"(per-modality: visual/audio/combined)") signal_cands = highlights.score_windows( segments, llm, peak_windows, events=events, **({"guidance": gaming.guidance("windows"), "events_title": events_title, "labels": {i: "game here: " + gaming.game_at(s0, e0)[0] for i, (s0, e0) in enumerate(peak_windows) if gaming.game_at(s0, e0)[0]}, "batch": 8} if gaming is not None else {}), **highlights.window_options(selection), ) # Signal peaks are seeded tight around the hot moment — grow them to a # full ~25s clip on sentence boundaries so action moments aren't tiny. if gaming is None: signal_cands = [ highlights._fit_to_segments( c, segments, clips_cfg["min_duration"], clips_cfg["max_duration"], target_duration=25.0, **highlights.start_rules(selection, c, lead_in=highlights.LEAD_IN_SECONDS), ) for c in signal_cands ] else: # One game moment makes a 15-35 s clip that starts before the play. # Snapping to sentences would move it to wherever the streamer # next spoke (often after the play), so it is only extended to # finish a line it would cut off. end_of_video = max(segments[-1].end if segments else 0.0, float(audio_excitement.size)) signal_cands = [ _frame_game_window(c, segments, clips_cfg["min_duration"], clips_cfg["max_duration"], end_of_video) for c in signal_cands ] candidates += signal_cands # ---- 2b. what the person asked for (analysis/intent.py) --------------- # Windows where an asked-for topic is said, or inside an asked-for range, # that no candidate covers yet: however many points a moment is offered, # it can only be picked if it is a candidate. Scored like signal peaks. if intent is not None: clip_intent.locate(intent, segments) end_of_video = max(segments[-1].end if segments else 0.0, float(audio_excitement.size)) wanted = clip_intent.windows(intent, candidates, end_of_video) if wanted: print(f" Clip direction: {len(wanted)} window(s) where what was asked for comes up") asked = highlights.score_windows(segments, llm, wanted, events=events, **highlights.window_options(selection)) for c in asked: c.source = "direction" candidates += [ highlights._fit_to_segments(c, segments, clips_cfg["min_duration"], clips_cfg["max_duration"], target_duration=30.0, **highlights.start_rules(selection, c)) for c in asked ] intent.added = len(asked) # ---- 2c. a match's moments (sports/) ---------------------------------- # Goals, saves, cards: found where the crowd, the commentary, the whistle, # the screen and the scoreboard agree, each given its own window (the # build-up, the moment, the reaction) as a candidate scored like any other. sport_moments: list = [] if sport is not None: highlights_choice = sport.option.get(" - analysis/game_audio.py (8412 bytes)
"""What the game sounds like, for the gaming profile. Chat and the streamer's voice say something happened; the game's own sound often says what: gunfire and explosions in a fight, a stadium crowd roaring at a goal, a crash in a race, a scream in a horror game. A quiet streamer's big play still sounds like one. A sound tagger (analysis/panns.py, trained on AudioSet's 527 everyday sounds) listens to every second, over a 2-second window, and its classes are pooled into groups (config/gaming.yaml's sound_groups). On a stream it mostly hears the streamer's voice and music, so a game sound is judged against its own level in this stream, the way chat_moments judges a chat burst: Rocket League's goal explosions score about 0.25, an Apex fight's gunfire 0.5-0.8, and both are the loudest of their kind in their stream. listen() is the expensive half (decode + tagging, run beside Whisper: about 600x realtime on a GPU, 120x on a CPU). sound_signal() turns what it heard into the game channel's evidence and events, weighted for the kind of game. """ import subprocess from dataclasses import dataclass, field from pathlib import Path import numpy as np from analysis import panns CHUNK_SECONDS = 60 # decoded at a time: a 3-hour VOD never sits in memory BASELINE = 90 # seconds either side that set a sound's usual level here MIN_PROB = 0.1 # below this the tagger isn't really hearing it MIN_REF = 0.2 # "as strong as it gets" is never set lower than this, so a # stream with no explosions doesn't turn its loudest bang into one EVENT = 0.5 # strength at which a sound is named to the model MERGE_SECONDS = 10 # a goal and its replay, a fight's bursts: one moment MAX_EVENTS = 60 DEFAULT_WEIGHT = 0.5 # a sound group a genre doesn't list @dataclass class GameSounds: """Per second 0..1: `game` for sounds the game makes (gunfire, a goal), `people` for the streamer's or friends' (a scream, laughter). events: (second, "GAME SOUND: ...") for the model.""" game: np.ndarray people: np.ndarray events: list = field(default_factory=list) def _pcm_chunks(path: Path, seconds: int): """Mono 32 kHz float32 audio, `seconds` at a time.""" from core.binaries import ffmpeg proc = subprocess.Popen( [ffmpeg(), "-v", "error", "-i", str(path), "-vn", "-ac", "1", "-ar", str(panns.SAMPLE_RATE), "-f", "f32le", "-"], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) size = seconds * panns.SAMPLE_RATE * 4 try: while True: buf = proc.stdout.read(size) if not buf: break yield np.frombuffer(buf[: len(buf) // 4 * 4], dtype=np.float32) finally: proc.stdout.close() proc.wait() def listen(path: Path, groups: dict, tag=None, chunks=None) -> dict[str, np.ndarray] | None: """{group: per-second probability} over the whole video: for second s, the strongest class of the group in the 2 s around it. None when there is no audio. `tag` and `chunks` stand in for the model and the decoder in tests.""" tag = tag or panns.tag names = {n: i for i, n in enumerate(panns.labels())} index = {g: [names[c] for c in spec.get("classes") or [] if c in names] for g, spec in groups.items()} index = {g: ix for g, ix in index.items() if ix} if not index: return None sr = panns.SAMPLE_RATE half, width = sr // 2, 2 * sr out: dict[str, list] = {g: [] for g in index} # The window for second s spans [s - 0.5, s + 1.5): half a second of # silence stands in before the start and after the end. buf = np.zeros(half, dtype=np.float32) start = -half # sample offset of buf[0] total = 0 s = 0 def run(windows: list) -> None: if windows: probs = tag(np.stack(windows)) for g, ix in index.items(): out[g].append(probs[:, ix].max(axis=1)) for chunk in chunks if chunks is not None else _pcm_chunks(path, CHUNK_SECONDS): total += chunk.size buf = np.concatenate([buf, chunk]) windows = [] while (s + 1) * sr + half <= start + buf.size: a = s * sr - half - start windows.append(buf[a:a + width]) s += 1 run(windows) keep = s * sr - half - start buf, start = buf[keep:], start + keep seconds = total // sr if seconds < 1: return None buf = np.concatenate([buf, np.zeros(width, dtype=np.float32)]) windows = [] while s < seconds: a = s * sr - half - start windows.append(buf[a:a + width]) s += 1 run(windows) return {g: np.concatenate(v)[:seconds].astype(np.float32) for g, v in out.items() if v} def _rolling_median(x: np.ndarray, half: int) -> np.ndarray: """Median of the `half` seconds either side of each second.""" from numpy.lib.stride_tricks import sliding_window_view padded = np.pad(x, half, mode="edge") return np.median(sliding_window_view(padded, 2 * half + 1), axis=1).astype(np.float32) def sound_signal(heard: dict[str, np.ndarray] | None, groups: dict, genres, genre_sounds: dict, max_events: int = MAX_EVENTS) -> GameSounds | None: """What listen() heard, as evidence of in-game moments. `genres`: the kind of game per second (or one for the whole video), which sets how much each sound means (config/gaming.yaml's genre_sounds).""" if not heard: return None n = max(v.size for v in heard.values()) if isinstance(genres, str): genres = [genres] * n kinds = sorted(set(genres)) at = {k: np.array([g == k for g in genres[:n]] + [False] * (n - len(genres[:n]))) for k in kinds} game = np.zeros(n, dtype=np.float32) people = np.zeros(n, dtype=np.float32) runs = [] for g, p in heard.items(): spec = groups.get(g) or {} p = np.asarray(p, dtype=np.float32)[:n] level = _rolling_median(p, BASELINE) # How strong this sound gets in this stream: its loudest moments # score 1, its usual level 0. ref = max(float(np.percentile(p, 99.5)), MIN_REF) usual = float(np.median(p)) strength = np.clip((p - level) / max(ref - usual, 0.05), 0.0, 1.0) * (p >= MIN_PROB) weight = np.zeros(n, dtype=np.float32) for k in kinds: weight[at[k][: p.size]] = float((genre_sounds.get(k) or {}).get(g, DEFAULT_WEIGHT)) strength = (strength * weight[: p.size]).astype(np.float32) side = people if spec.get("side") == "people" else game np.maximum(side[: p.size], strength, out=side[: p.size]) # A moment: where it is named to the model (short gaps bridged). sec = 0 while sec < strength.size: if strength[sec] < EVENT: sec += 1 continue end = sec while end + 1 < strength.size and (strength[end + 1] >= EVENT or strength[end + 2:end + 4].max(initial=0) >= EVENT): end += 1 runs.append((sec, end, g, float(strength[sec:end + 1].max()))) sec = end + 1 # Sounds close together are one moment (a fight's bursts, a goal and its # replay), named by what was heard, strongest first. moments: list[dict] = [] for lo, hi, g, peak in sorted(runs): if moments and lo - moments[-1]["end"] <= MERGE_SECONDS: m = moments[-1] m["end"] = max(m["end"], hi) m["peak"] = max(m["peak"], peak) m["groups"][g] = max(m["groups"].get(g, 0.0), peak) else: moments.append({"start": lo, "end": hi, "peak": peak, "groups": {g: peak}}) events = [] for m in sorted(moments, key=lambda m: -m["peak"])[:max_events]: heard_here = sorted(m["groups"], key=lambda g: -m["groups"][g]) label = " + ".join(str((groups.get(g) or {}).get("label", g)) for g in heard_here) own = all((groups.get(g) or {}).get("side") != "people" for g in heard_here) span = m["end"] - m["start"] + 1 events.append((float(m["start"]), ("GAME SOUND: " if own else "SOUND: ") + label + (f" ({span}s)" if span >= 3 else ""))) events.sort(key=lambda ev: ev[0]) return GameSounds(game=game, people=people, events=events) - analysis/game_text.py (8445 bytes)
"""What the game writes on screen, for the gaming profile. Games announce their moments in big letters: ELIMINATED, VICTORY ROYALE, PENTA KILL, "X SCORED", YOU DIED, ENEMY FELLED. They also fill the screen with text when nothing is happening: a settings page, a queue, a loading screen, which chat will happily react to. Reading the screen tells the two apart where the transcript, chat and the game's sound can't. OCR (RapidOCR: PaddleOCR's models on onnxruntime, Apache-2.0) costs about a second a frame on a CPU, so it only reads the game-moment windows, a frame every few seconds, strongest first, within a time budget. It reads Latin script and kanji (not kana), and config/gaming.yaml's screen_text lists each word in the languages games are commonly played in. - A banner (big text near the middle) with an event word becomes an "ON SCREEN:" event for the model and the clip's breakdown. It adds no points of its own: most videos never show one. - A window where most frames show menu words (a few different ones across it) is marked as a menu, and scores a little lower. """ import re import time import unicodedata from dataclasses import dataclass, field from pathlib import Path # Seconds between the frames read in a window: closer together over its # first seconds, where the moment is (a window starts 4 s before it), since # a banner is only up for two or three; further apart after. SAMPLE_EVERY = 4.0 SAMPLE_CLOSE = 2.0 CLOSE_FOR = 12.0 MAX_FRAMES = 160 TIME_BUDGET = 120.0 # seconds of reading per video, at most # The middle of the frame, read at this width: banners are there, and a # stylised one is only legible this large (read whole at 960 wide, a Rocket # League goal banner wasn't read at all). CENTRE = (0.12, 0.12, 0.88, 0.82) # left, top, right, bottom as shares of the frame WIDTH = 1280 BANNER_HEIGHT = 0.035 # a banner's letters are at least this share of that height; # a stream's chat overlay is 0.02-0.03 MIN_CONFIDENCE = 0.6 # A window is a menu, queue or settings screen when at least two thirds of # its frames show a menu word (a word in someone's captions or an overlay # now and then doesn't make one), and this many different ones appear across it (a # settings page shows a few per screen, and not always the same few). MENU_WORDS = 2 WHOLE_WORD = 5 # Latin words this short only match on their own # What the OCR often reads one letter as, folded together on both sides. _FOLD = str.maketrans({"Q": "O", "0": "O", "1": "I", "5": "S"}) _SEPARATORS = re.compile(r"[\s:!?.,|/\\\[\]()_\-、。,.・:!?「」『』()]+") _engine = None def available() -> bool: try: import rapidocr_onnxruntime # noqa: F401 except ImportError: return False return True @dataclass class ScreenText: """events: (second, "ON SCREEN: ...") for the model; menus: (start, end, what was read) for the windows that read as a menu, queue or loading screen; frames: how many were read.""" events: list = field(default_factory=list) menus: list = field(default_factory=list) frames: int = 0 def _norm(text: str) -> str: """Upper case, no accents, letters and digits only, the usual misreads folded: a stylised banner loses its spaces and accents, and "KIMOU11 A MARQUÉ" comes back as "KIMOU1TAMARQUE" or "...MAROUE".""" t = unicodedata.normalize("NFKD", str(text)) return "".join(ch for ch in t.upper() if ch.isalnum() and not unicodedata.combining(ch)).translate(_FOLD) class Lexicon: """Words to find in what the OCR read. Short Latin ones (ACE, GOAL, PLAY) only as a whole word, so ACE isn't found in PLACE or PLAY in PLAYER. Japanese and Chinese are written without spaces, so a kanji word (設定, 感度) is found anywhere in the text.""" def __init__(self, terms): self.terms = [(str(t), _norm(t)) for t in terms or [] if _norm(t)] def find(self, text: str) -> list[str]: compact = _norm(text) words = {_norm(w) for w in _SEPARATORS.split(str(text))} hits = [] for term, n in self.terms: whole = n.isascii() and len(n) <= WHOLE_WORD if (n in words or n == compact) if whole else n in compact: hits.append(term) return hits def _ocr(img): global _engine if _engine is None: from rapidocr_onnxruntime import RapidOCR _engine = RapidOCR() result, _elapsed = _engine(img, use_cls=False) return result or [] def read_frame(img, events: Lexicon, menu: Lexicon, ocr=None) -> tuple[list[str], set]: """(event words in a banner, menu words) in the middle of one frame.""" import cv2 h, w = img.shape[:2] left, top, right, bottom = CENTRE img = img[int(h * top):int(h * bottom), int(w * left):int(w * right)] h, w = img.shape[:2] # Linear, not area-averaged, even when shrinking: stylised letters kept # their shapes better (area turned "11 A MARQUÉ" into "LFASMAROUE"). img = cv2.resize(img, (WIDTH, max(2, round(h * WIDTH / w))), interpolation=cv2.INTER_LINEAR) height, width = img.shape[:2] found, menus = [], set() for box, text, conf in (ocr or _ocr)(img): if float(conf) < MIN_CONFIDENCE: continue menus.update(menu.find(text)) xs, ys = [p[0] for p in box], [p[1] for p in box] tall = (max(ys) - min(ys)) / height cx, cy = sum(xs) / len(xs) / width, sum(ys) / len(ys) / height # A stream's chat overlay and the HUD are small, or at the edges. if tall >= BANNER_HEIGHT and 0.05 <= cx <= 0.95 and 0.03 <= cy <= 0.97: hits = events.find(text) if hits: found.append(max(hits, key=len)) return found, menus def read_screen(path: Path, windows: list, genre_at, lexicon: dict, grab=None, read=None, budget: float = TIME_BUDGET, max_frames: int = MAX_FRAMES) -> ScreenText: """Read the screen in each window (strongest first, as given). genre_at: (start, end) -> the kind of game there, which picks its event words. `grab` (second -> frame) and `read` stand in for the video and the OCR in tests.""" if not windows: return ScreenText() if grab is not None: return _read_windows(windows, genre_at, lexicon, grab, read or read_frame, budget, max_frames) import cv2 from video.capture import video_capture with video_capture(path, required=False) as cap: if cap is None: return ScreenText() def grab_frame(t: float): cap.set(cv2.CAP_PROP_POS_MSEC, t * 1000.0) ok, img = cap.read() return img if ok else None return _read_windows(windows, genre_at, lexicon, grab_frame, read or read_frame, budget, max_frames) def _read_windows(windows, genre_at, lexicon: dict, grab, read, budget: float, max_frames: int) -> ScreenText: out = ScreenText() events_by_genre = lexicon.get("events") or {} menu = Lexicon(lexicon.get("menu")) t0 = time.monotonic() for start, end in windows: if out.frames >= max_frames or time.monotonic() - t0 > budget: break genre = genre_at(start, end) events = Lexicon([*(events_by_genre.get("generic") or []), *(events_by_genre.get(genre) or [] if genre != "generic" else [])]) read_here = menu_frames = 0 menu_seen: dict[str, int] = {} named = set() t = float(start) + 1.0 while t < end and out.frames < max_frames: img = grab(t) if img is None: break out.frames += 1 read_here += 1 found, menus = read(img, events, menu) if menus: menu_frames += 1 for m in menus: menu_seen[m] = menu_seen.get(m, 0) + 1 for term in found: if term not in named: named.add(term) out.events.append((t, f"ON SCREEN: {term}")) t += SAMPLE_CLOSE if t - start < CLOSE_FOR else SAMPLE_EVERY if read_here and menu_frames * 3 >= read_here * 2 and len(menu_seen) >= MENU_WORDS: words = sorted(menu_seen, key=lambda m: (-menu_seen[m], m))[:3] out.menus.append((float(start), float(end), ", ".join(words))) out.events.sort(key=lambda ev: ev[0]) return out - analysis/game_vision.py (7418 bytes)
"""The AI looks at the best candidates of a gaming stream. Everything else the gaming profile scores with is a stand-in for the picture: chat, the game's sound, a banner's words, the transcript. The last check before ranking shows the local model a few frames of each of the best candidates and asks what happens in them: a fight, a goal, a boss going down, or a menu, a loading screen, nothing at all. Only a local model that takes images does it (Gemma 3 and Gemma 4 through Ollama; gemma:7b can't). The video picture never leaves the PC, as the privacy page promises, so a cloud AI skips this step. Its verdict moves a clip by at most 10 points either way, within a time budget. It only takes points off as far as the clip is quiet: what is said counts as on any stream, so a just-chatting stretch or a reaction with no gameplay in the picture isn't marked down for that. """ import json import re import time from pathlib import Path PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "look_at_game.txt" MAX_CANDIDATES = 12 FRAMES = 4 # Longest side of each frame sent: Gemma's image encoder works at 896, and # at 512 gemma3:4b took two Apex fights for ordinary play. SIZE = 896 QUALITY = 85 TIME_BUDGET = 240.0 # seconds of looking per video, at most MAX_ADJUST = 10 def _parse(raw: str) -> dict | None: text = re.sub(r"^```(?:json)?\s*|\s*```$", "", (raw or "").strip()) s, e = text.find("{"), text.rfind("}") if s == -1 or e <= s: return None try: data = json.loads(text[s:e + 1]) except json.JSONDecodeError: return None return data if isinstance(data, dict) else None _can_see: dict[str, bool] = {} def can_see(llm) -> bool: """Whether the model really gets the images, asked once per model with a plain red square. A model can say it takes images and not receive them: gemma4:e4b on one Ollama release described "a dark, abstract background" for a settings page, and its verdicts were noise.""" name = getattr(llm, "name", "") if name not in _can_see: import cv2 import numpy as np red = np.zeros((96, 96, 3), dtype=np.uint8) red[:, :, 2] = 255 ok, buf = cv2.imencode(".jpg", red) try: answer = llm.look('What single colour fills this image? Respond with ONLY valid JSON: ' '{"colour": "<colour>"}', [buf.tobytes()]) if ok else "" except Exception: answer = "" _can_see[name] = "red" in str(answer).lower() return _can_see[name] def frame_times(start: float, end: float, count: int = FRAMES) -> list[float]: """Evenly through the clip: a game moment's window starts a few seconds before the play, so these cover the setup, the play and the reaction.""" span = max(0.0, end - start) return [start + (i + 0.5) * span / count for i in range(count)] def _jpeg(img) -> bytes: import cv2 h, w = img.shape[:2] scale = SIZE / max(h, w) if scale < 1: img = cv2.resize(img, (max(2, round(w * scale)), max(2, round(h * scale))), interpolation=cv2.INTER_AREA) ok, buf = cv2.imencode(".jpg", img, [cv2.IMWRITE_JPEG_QUALITY, QUALITY]) return buf.tobytes() if ok else b"" def verdict_delta(data: dict | None) -> tuple[int, str] | None: """(score change, what was seen) from the model's answer; None when it can't be read. Not gameplay: the full -10. Otherwise strength 5 (ordinary play) changes nothing, 10 adds 10, 0 takes 10 off.""" if not data: return None moment = " ".join(str(data.get("moment") or "").split())[:80] gameplay = data.get("gameplay") if gameplay is False or str(gameplay).lower() == "false": return -MAX_ADJUST, moment or "not gameplay" try: strength = float(data.get("strength")) except (TypeError, ValueError): return None strength = min(10.0, max(0.0, strength)) return max(-MAX_ADJUST, min(MAX_ADJUST, round((strength - 5) * 2))), moment def look_at(candidates: list, video_path, llm, gaming, segments, events, grab=None, budget: float = TIME_BUDGET, max_candidates: int = MAX_CANDIDATES, talk=None) -> int: """Adjust the best `candidates` (by score) by what the model sees in them; returns how many were looked at. `grab` (second -> frame) stands in for the video in tests. `talk` (candidate -> 0 silent .. 1 steady talking) scales a verdict that takes points off: none of it for a clip full of talk, all of it for a silent one.""" from core import cancel template = PROMPT_PATH.read_text(encoding="utf-8") ranked = sorted(candidates, key=lambda c: c.score, reverse=True)[:max_candidates] if not ranked: return 0 if grab is not None: return _look(ranked, template, llm, gaming, segments, events, grab, budget, cancel, talk) import cv2 from video.capture import video_capture with video_capture(video_path, required=False) as cap: if cap is None: return 0 def grab_frame(t: float): cap.set(cv2.CAP_PROP_POS_MSEC, t * 1000.0) ok, img = cap.read() return img if ok else None return _look(ranked, template, llm, gaming, segments, events, grab_frame, budget, cancel, talk) def _look(ranked, template, llm, gaming, segments, events, grab, budget, cancel, talk=None) -> int: from analysis.gaming import genre_spec t0 = time.monotonic() looked = 0 for c in ranked: if time.monotonic() - t0 > budget: break cancel.check_active() times = frame_times(c.start, c.end) frames = [grab(t) for t in times] images = [_jpeg(f) for f in frames if f is not None] images = [b for b in images if b] if len(images) < 2: continue game, genre = gaming.game_at(c.start, c.end) spec = genre_spec(genre) said = " ".join(s.text.strip() for s in segments if s.end > c.start and s.start < c.end) seen = [f"{sec - c.start:.0f}s: {desc}" for sec, desc in events if c.start - 1 <= sec <= c.end] prompt = (template .replace("{game}", f"{game} ({spec.get('label', 'game')})" if game else spec.get("label", "a game")) .replace("{highlights}", " ".join(str(spec.get("highlights", "")).split())) .replace("{count}", str(len(images))) .replace("{times}", ", ".join(f"{t - c.start:.0f}" for t in times)) .replace("{transcript}", said[:1200] or "(nothing)") .replace("{events}", "\n".join(seen[:8]) or "(nothing)")) try: result = verdict_delta(_parse(llm.look(prompt, images))) except Exception as e: print(f" (looking at a clip failed: {e})") result = None looked += 1 if result is None: continue delta, moment = result if delta < 0 and talk is not None: delta = round(delta * (1.0 - max(0.0, min(1.0, talk(c))))) c.score = max(0, min(100, c.score + delta)) c.subscores = c.subscores or {} c.subscores["seen"] = delta if moment: c.subscores["seen_what"] = moment why = c.subscores.get("game_why") c.subscores["game_why"] = f"SEEN: {moment}" + (f"; {why}" if why else "") return looked - analysis/gaming.py (8978 bytes)
"""Scoring a gaming stream as a gaming stream. The standard scorer judges talk: hooks, opinions, drama, quotable lines. On a game stream the moment is usually something that happened in the game (a kill streak, a boss going down, a goal) and the reaction to it, often with little said at all. This module is the gaming profile's knowledge: - which game is being played (the platform's category, else the title) and so which kind of game (config/gaming.yaml); - the guidance that tells the model what a highlight is in it; - the weights the fused score uses. It only applies when a job scores as a gaming stream (core.modes.gaming_scoring): Gaming / Reaction, or Vertical Live with its content set to Gaming / reaction. Pure Python: the per-second signals live in analysis/chat_moments.py and fusion. """ import math import re from dataclasses import dataclass, field from functools import lru_cache from pathlib import Path KNOWLEDGE_PATH = Path(__file__).resolve().parent.parent / "config" / "gaming.yaml" # The fused score's weights for a gaming stream: the standard ones. What is # said is judged exactly as on any stream (a reaction to BlizzCon is talk, # and weighting a game channel at 40% left a two-hour reaction with 13 # clips). The game adds value on top instead, like creator context: fusion's # game bonus, and a quiet stretch's share of the weight (fusion._fuse). STANDARD_WEIGHTS = {"text": 0.30, "visual": 0.20, "reaction": 0.20, "audio": 0.20, "engagement": 0.10} @lru_cache(maxsize=1) def knowledge() -> dict: import yaml return yaml.safe_load(KNOWLEDGE_PATH.read_text(encoding="utf-8")) or {} def _phrase(term: str) -> re.Pattern: """A game name matched as whole words, ignoring case ("rust" is not in "trust", "ark" is not in "dark").""" return re.compile(r"(?<![\w])" + re.escape(term.lower()) + r"(?![\w])") def genre_of(name: str) -> str | None: """The genre a game's name (or a title) points to, by the longest known name inside it; None when nothing matches.""" text = (name or "").lower() best, best_len = None, 0 for genre, names in (knowledge().get("games") or {}).items(): for term in names: t = str(term) if len(t) > best_len and _phrase(t).search(text): best, best_len = genre, len(t) return best # Twitch/Kick categories that are not a game: a stream's talking parts. NOT_GAMES = {"just chatting", "irl", "music", "art", "talk shows & podcasts", "special events", "sports", "travel & outdoors", "asmr", "pools, hot tubs, and beaches", "makers & crafting", "food & drink", "science & technology", "software and game development"} def _is_game(name: str) -> bool: return bool(name) and name.strip().lower() not in NOT_GAMES @dataclass class GamingProfile: genre: str = "generic" game: str = "" # the main game's name, when known games: list = field(default_factory=list) split_layout: bool = False # Gaming / Reaction (reaction not measured) weights: dict = field(default_factory=dict) @property def spec(self) -> dict: return genre_spec(self.genre) def game_at(self, start: float, end: float | None = None) -> tuple[str, str]: """(game, genre) being played over [start, end]: the platform's game for that part of the stream (a stream that goes from Just Chatting to Rust to Fortnite), else the stream's main game.""" end = start if end is None else end mid = (start + end) / 2 for g in self.games: name = str(g.get("name") or "").strip() if name and float(g.get("start") or 0) <= mid < float(g.get("end") or 0): return name, self._genre_for(name) return self.game, self.genre def _genre_for(self, name: str) -> str: if not _is_game(name): return "reaction" return genre_of(name) or ("generic" if name != self.game else self.genre) def genre_track(self, seconds: int) -> list[str]: """The kind of game at each second of the video, as game_at() has it.""" track = [self.genre] * seconds known: dict[str, str] = {} for g in reversed(self.games): # the first range listed wins, as in game_at name = str(g.get("name") or "").strip() if not name: continue if name not in known: known[name] = self._genre_for(name) lo = max(0, math.ceil(float(g.get("start") or 0))) hi = min(seconds, math.ceil(float(g.get("end") or 0))) track[lo:hi] = [known[name]] * max(0, hi - lo) return track def guidance(self, kind: str = "clips", start: float | None = None, end: float | None = None) -> str: """The block the scoring prompts carry (their {mode_guidance}), for the game played over [start, end] when given.""" game, genre = (self.game, self.genre) if start is None else self.game_at(start, end) spec = genre_spec(genre) label = spec.get("label", "game") highlights = " ".join(str(spec.get("highlights", "")).split()) callouts = ", ".join(f'"{c}"' for c in (spec.get("callouts") or [])[:8]) what = f"{game} ({label})" if game else f"a {label}" if genre == "reaction": what = f"a reaction / talking part of the stream ({game})" if game else "a reaction stream" lines = [ (f"THIS IS FROM A GAMING / REACTION STREAM: {what}. Judge what is said exactly as " "you would on any stream (a strong take, a funny line, a story, a heated moment), " "and ALSO count what happens in the game:"), f"- In-game moments are strong clips too: {highlights}.", (f"- The streamer's callouts show when something just happened ({callouts}), and the " "GAME / CHAT events listed with the transcript mark it too."), ("- A clear in-game moment is a strong clip even when little is said: a shout, a laugh, " "a scream or silence over a big play still counts. Do not mark a moment down just " "because there is little talking."), ("- For an in-game moment, include a few seconds of setup before it and end once the " "reaction lands; 15-35 seconds for one moment is ideal."), ("- Score low: menus, lobbies, queues and loading screens, reading out donations or " "subscriptions, and dead time between plays."), ] if genre == "reaction": lines.append("- When the streamer reacts to something they're watching, their " "strongest reactions and takes are the payoff (shock, laughter, a strong " "opinion), not the watched video on its own.") if kind == "rerank": lines = [lines[0], ("- Between clips that are otherwise as good, prefer the one with a " "real in-game moment or a big reaction.")] return "\n".join(lines) def genre_spec(genre: str) -> dict: genres = knowledge().get("genres") or {} return genres.get(genre) or genres.get("generic") or {} def games_summary(games: list | None) -> str: """The main game: the one played longest in total. Just Chatting and the like only count when nothing else was played.""" totals: dict[str, float] = {} for g in games or []: name = str(g.get("name") or "").strip() if name: span = max(0.0, float(g.get("end") or 0) - float(g.get("start") or 0)) totals[name] = totals.get(name, 0.0) + span if not totals: return "" real = {k: v for k, v in totals.items() if _is_game(k)} or totals return max(real, key=real.get) def profile_for(config: dict, games: list | None = None, title: str = "") -> GamingProfile: """The gaming profile for a job: the game (platform category first, then the title), its genre, and the weights, taking scoring.profiles.gaming from the settings when a user has set it.""" from core import modes game = games_summary(games) genre = None if game: genre = genre_of(game) if genre is None: # The title of a stream often names the game or says "speedrun"; a # YouTube video's tags often name it too. hints = " ".join(str(g.get("hint") or "") for g in games or []) genre = genre_of(title) or genre_of(hints) split = modes.is_gaming(config) scoring = config.get("scoring") or {} weights = dict((scoring.get("profiles") or {}).get("gaming", {}).get("weights") or scoring.get("weights") or STANDARD_WEIGHTS) weights.pop("game", None) # the game adds on top; it has no weight of its own return GamingProfile(genre=genre or "generic", game=game, games=list(games or []), split_layout=split, weights=weights) - analysis/highlights.py (35267 bytes)
"""LLM-based highlight detection with strict duplicate prevention. Scoring asks the model for 0-100 viral potential against an explicit framework (hooks, emotional peaks, opinion bombs, revelations, conflict, quotable lines, payoff structure), with long videos chunked and overlapped so a moment on a chunk boundary is never lost. Duplicate prevention is three independent checks, applied highest-score-first: 1. timestamp overlap — reject if >40% of the shorter clip overlaps a kept clip 2. transcript similarity — reject if the spoken text is >70% similar to a kept clip 3. segment reuse — reject if >40% of the candidate's transcript segments are already claimed by kept clips Every rejection carries a reason so the pipeline can log it for auditing. All robustness (JSON parsing, timestamp validation, duration enforcement) lives here in deterministic Python, NOT in the LLM backend — so swapping models never changes this module. """ import json import re from difflib import SequenceMatcher from pathlib import Path from core import cancel, progress from core.models import ClipCandidate, Rejection, Segment from llm.base import LLMBackend, generate_json PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "score_clips.txt" WINDOWS_PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "score_windows.txt" # ---- clip selection (scoring.selection) -------------------------------------- # # Two settings, read with defaults (config/settings.yaml doesn't list them): # # scoring.selection: 2 (the default) or 1. Selection 1 chooses clips as they were # chosen before selection 2; `selection: 1`, or a value it doesn't know, brings it back. # Selection 2 changes four things on a standard run. A run scored as # gaming (core.modes.gaming_scoring: Gaming / Reaction, or Vertical Live's # Gaming / reaction choice) and a Sports match keep selection 1 whatever # this says; any other run is a standard one, a gaming VOD on the standard # layout included (analysis/fusion.py decides). The four: # - the signal-peak and clip-direction windows are scored WINDOW_BATCH at # a time, each answer matched to its window by time, and a window left # at the neutral 50 is reported. One request for 36-84 windows ran past # a local model's output budget, so every window fell back to 50, and # a partial answer matched by position put hooks on the wrong clips; # - each signal pass's window budget goes to its strongest peaks, not to # its earliest, so a long video's late peaks can become candidates; # - only a transcript pick scored at or above min_score hides a signal # peak it overlaps; # - a clip's start never moves past the line it was picked on, or the # line its hook quotes; a start that would come in mid-sentence goes # back to the line the sentence starts on, when that is at most # SENTENCE_REACH_SECONDS earlier and within max_duration; and a signal # window gets up to LEAD_IN_SECONDS of complete sentences before it # (_fit_to_segments). End rules are unchanged. # scoring.longform_selection: the same choice for Longform's 16:9 clips, which # keep 1 until this says 2, whatever scoring.selection is (longform/process.py). WINDOW_BATCH = 8 LEAD_IN_SECONDS = 6.0 # A line runs on from the one before it when that one has no sentence end and # the pause between them is at most SENTENCE_PAUSE_SECONDS; a longer pause # ends the sentence, full stop or not (going back over it would put that # silence in the clip's first seconds). A start on a line that runs on goes # back to the line its sentence starts on, at most SENTENCE_REACH_SECONDS. On # a talking-head stream that reached 8 of the 11 picks whose kept line came in # mid-sentence; the other 3 sentences ran on over 4-8 lines without a sentence # end (8.5-16 s). # # A sentence end, for this rule (_ends_sentence_for_start), is one the end # rules read (_ends_sentence: . ! ? and the ellipsis, Chinese and Japanese # full-width ones, also inside closing quotes or brackets) or a mark they # don't read that a language in multilingual/languages.py ends sentences with # (_MORE_SENTENCE_ENDS): the danda and double danda (Hindi, Bengali), the # Arabic full stop (Urdu) and question mark (Arabic, Urdu), and the full-width # and half-width full stops (Chinese, Japanese). A transcript with none of # them at all (speech recognition that left the punctuation out, or Thai, # written without any) can't show where a sentence starts, so there the rule # moves no start. SENTENCE_REACH_SECONDS = 8.0 SENTENCE_PAUSE_SECONDS = 2.0 NEUTRAL_HOOK = "High-energy moment" # a window's hook when the model gave it none def selection_version(config) -> int: """scoring.selection from a run's settings: 2 when absent or 2; anything else (1, a blank) is 1.""" try: version = ((config or {}).get("scoring") or {}).get("selection", 2) return 2 if not isinstance(version, bool) and float(version) == 2 else 1 except (AttributeError, TypeError, ValueError): return 1 def window_options(selection: int) -> dict: """score_windows' keywords for a standard run's windows: none for selection 1 (one request for them all, as always); for selection 2, WINDOW_BATCH at a time, matched by time, with a warning for any window left at the neutral 50.""" return {"batch": WINDOW_BATCH, "warn": True} if selection == 2 else {} def start_rules(selection: int, c: ClipCandidate, lead_in: float = 0.0) -> dict: """_fit_to_segments' keywords for candidate `c`: none for selection 1; for selection 2, its start keeps the line it was picked on and the line its hook quotes, goes back to the start of its sentence when it would come in mid-sentence (up to SENTENCE_REACH_SECONDS), and up to `lead_in` seconds of setup may come before it.""" if selection != 2: return {} hook = c.hook if c.hook and c.hook != NEUTRAL_HOOK else "" return {"keep_own_start": True, "hook": hook, "lead_in": lead_in, "sentence_reach": SENTENCE_REACH_SECONDS} def find_highlights( segments: list[Segment], llm: LLMBackend, *, min_score: int = 60, max_clips: int = 3, min_duration: float = 10.0, max_duration: float = 60.0, max_overlap: float = 0.4, max_text_similarity: float = 0.7, max_segment_reuse: float = 0.4, chunk_seconds: float = 1200.0, chunk_overlap_seconds: float = 60.0, long_video_threshold_seconds: float = 1800.0, events: list[tuple[float, str]] | None = None, guidance="", # str, or (start, end) -> str for the part of the video a chunk covers events_title: str | None = None, selection: int = 1, # scoring.selection; 2: start_rules for the picks (analysis/fusion.py says) ) -> tuple[list[ClipCandidate], list[Rejection]]: """Returns (selected clips, rejected candidates with reasons). `events` is an optional multimodal timeline [(second, description)] shown to the model alongside each chunk's transcript. `guidance` is extra instruction for one kind of video (the gaming profile's, analysis/gaming.py); empty, the prompt is exactly the standard one. `selection` 2 fits each pick with start_rules; 1, the default, fits them as always.""" if not segments: return [], [] prompt_template = PROMPT_PATH.read_text(encoding="utf-8") video_end = segments[-1].end chunks = _chunk_segments(segments, chunk_seconds, chunk_overlap_seconds, long_video_threshold_seconds) print(f" Analyzing {len(chunks)} chunk(s) with {llm.name}...") candidates: list[ClipCandidate] = [] for i, chunk in enumerate(chunks, 1): # One LLM call per chunk, and a 12b model on a long stream makes each # of those minutes long. Checking here bounds how long Cancel takes to # land at roughly one chunk instead of the entire analysis stage. cancel.check_active() progress.emit(stage="analyze", current=i, total=len(chunks)) transcript_text = "\n".join(f"[{s.start:.1f} - {s.end:.1f}] {s.text}" for s in chunk) prompt = prompt_template.replace("{transcript}", transcript_text) # Guidance can depend on where in the video the chunk is (a stream that # changes game part way through). chunk_guidance = guidance(chunk[0].start, chunk[-1].end) if callable(guidance) else guidance prompt = prompt.replace("{mode_guidance}", _guidance_block(chunk_guidance)) prompt = prompt.replace("{events}", _events_block(events, chunk[0].start, chunk[-1].end, events_title)) prompt = prompt.replace("{min_duration}", str(int(min_duration))) prompt = prompt.replace("{max_duration}", str(int(max_duration))) raw = _generate_with_retry(llm, prompt) parsed = _parse_clips_json(raw) if parsed is None: snippet = " ".join(raw.split())[:150] print(f" Chunk {i}/{len(chunks)}: unparseable LLM output, skipping chunk (got: {snippet!r})") continue print(f" Chunk {i}/{len(chunks)}: {len(parsed)} candidate(s)") candidates.extend(parsed) candidates = [c for c in candidates if _valid_range(c, video_end)] candidates = [_fit_to_segments(c, segments, min_duration, max_duration, **start_rules(selection, c)) for c in candidates] candidates = [c for c in candidates if c.duration >= min_duration - 1] return _select_unique( candidates, segments, min_score=min_score, max_clips=max_clips, max_overlap=max_overlap, max_text_similarity=max_text_similarity, max_segment_reuse=max_segment_reuse, ) def score_windows( segments: list[Segment], llm: LLMBackend, windows: list[tuple[float, float]], events: list[tuple[float, str]] | None = None, guidance: str = "", events_title: str | None = None, labels: dict | None = None, # window index -> a note shown with it (the game there) batch: int = 0, warn: bool = False, ) -> list[ClipCandidate]: """Score specific time windows (signal peaks fusion found) in one LLM call, so signal candidates get real text/engagement scores and grounded hooks instead of placeholders. `batch` (the gaming profile, and selection 2's windows): at most this many windows a call, each answer matched to its window by time. One call for 45 windows ran past the model's output budget, and a cut-off answer left every window at a neutral 50. `warn` (selection 2): say so when a window is left at the neutral 50, so a fallback is never silent.""" if not windows: return [] # One call for all the windows, but on CPU that single call can take # minutes. Say it is happening rather than going quiet mid-analyze. print(f" Scoring {len(windows)} signal-peak window(s) with the model...") if batch and len(windows) > batch: results: list[ClipCandidate] = [] total = (len(windows) + batch - 1) // batch for b, i in enumerate(range(0, len(windows), batch), 1): progress.emit(stage="analyze", current=b, total=total) part = windows[i:i + batch] part_labels = {j - i: labels[j] for j in range(i, i + len(part)) if labels and labels.get(j)} results += _score_window_batch(segments, llm, part, events, guidance, events_title, part_labels, by_time=True, warn=warn) return results progress.emit(stage="analyze", current=1, total=1) return _score_window_batch(segments, llm, windows, events, guidance, events_title, labels, by_time=bool(batch), warn=warn) def _score_window_batch(segments, llm, windows, events, guidance, events_title, labels, by_time: bool, *, warn: bool = False) -> list[ClipCandidate]: template = WINDOWS_PROMPT_PATH.read_text(encoding="utf-8") blocks = [] for i, (start, end) in enumerate(windows): text = " ".join(s.text for s in segments if s.end > start and s.start < end) or "(no speech)" ev = _events_block(events, start, end, events_title) note = f" ({labels[i]})" if labels and labels.get(i) else "" blocks.append(f"WINDOW {i} [{start:.1f}s - {end:.1f}s]{note}:\n{text}\n{ev}".strip()) prompt = template.replace("{windows}", "\n\n".join(blocks)) prompt = prompt.replace("{mode_guidance}", _guidance_block(guidance)) raw = _generate_with_retry(llm, prompt, WINDOWS_SCHEMA) parsed = _parse_clips_json(raw) if warn and parsed is None: # Fusion hands the windows over signal by signal, not in time order. span = f"{min(s for s, _e in windows):.0f}-{max(e for _s, e in windows):.0f}s" print(f" Warning: the model's answer for {len(windows)} window(s) ({span}) couldn't be read; " "they stay at a neutral 50") if parsed is None: # Model failed — keep the windows anyway with neutral text scores; # their audio/visual signals still let strong moments compete. return [ ClipCandidate(start=s, end=e, score=50, hook="High-energy moment", source="signal") for s, e in windows ] results = [] by_index = dict(enumerate(parsed)) if by_time: # The model's own start for each window, when it copied one close # enough; its order only when it answered every window. by_index = {} for i, (start, _end) in enumerate(windows): near = [c for c in parsed if abs(c.start - start) <= 2.0] if near: by_index[i] = near[0] elif len(parsed) == len(windows): by_index[i] = parsed[i] for i, (start, end) in enumerate(windows): c = by_index.get(i) if c is not None: # Trust the model's score/hook but keep OUR window timestamps — # these came from the signals, not from the model. results.append(ClipCandidate(start=start, end=end, score=c.score, hook=c.hook, reason=c.reason, source="signal", engagement=c.engagement)) else: results.append(ClipCandidate(start=start, end=end, score=50, hook="High-energy moment", source="signal")) if warn: missed = [w for i, w in enumerate(windows) if by_index.get(i) is None] if missed: print(f" Warning: the model didn't answer {len(missed)} of {len(windows)} window(s) " f"({', '.join(f'{s:.0f}s' for s, _e in missed)}); they stay at a neutral 50") return results def _events_block(events: list[tuple[float, str]] | None, start: float, end: float, title: str | None = None) -> str: if not events: return "" lines - analysis/hype.py (20597 bytes)
"""Audience hype signals, all optional and failure-safe. Processing works identically without them: audience data only ever ADDS a small scoring bonus and can also seed candidate windows around audience peaks. Sources: - Twitch VODs: the public web GQL endpoint every chat-replay tool uses. The Client-ID is scraped from twitch.tv itself at call time, because Twitch rotates it (hardcoded IDs from 2023-era tools now 400). - YouTube, authenticated: the creator's organic audience-retention curve from YouTube Analytics, when a cached OAuth token includes yt-analytics.readonly. The raw retention curve is locally detrended before percentile ranking so ordinary retention decay does not make the start of every video look "hot". - YouTube, public fallback: the "most replayed" heatmap yt-dlp exposes for popular videos, then chat replay (live_chat) for finished live streams. - Kick: NOT possible — Kick deletes chat when a stream ends and has no replay endpoint (verified 2026-07); Kick hype comes from audio/visual signals only. Gift-sub / bits resistance: the Twitch curve counts UNIQUE CHATTERS per time bin, not messages. A gifted-sub bomb produces a burst of messages from few humans (plus bots), which barely moves unique-chatter density — so fake "hype" from sub trains doesn't outrank a genuinely popping chat. The bonus is also hard-capped in fusion so transcript/visual context stays dominant. """ import json import os import re import time import urllib.request from datetime import date from pathlib import Path import numpy as np BIN_SECONDS = 10 _GQL_URL = "https://gql.twitch.tv/gql" _UA = ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" " (KHTML, like Gecko) Chrome/126.0 Safari/537.36" ) _MAX_PAGES = 800 # ~48k messages — covers long VODs _TIME_BUDGET_S = 180 # never stall the pipeline on a slow chat fetch _YT_ANALYTICS_SCOPE = "https://www.googleapis.com/auth/yt-analytics.readonly" _RETENTION_BASELINE_RADIUS = 4 _RETENTION_IGNORE_START_RATIO = 0.05 # Clips Kitty consumes audience as a CLIP-level signal: fusion averages the # curve across each candidate and only bonuses windows whose mean is near the # top. Organic retention, however, arrives as narrow 1%-of-video samples. # Spread each local-retention peak into a short, gently-decaying interest zone # so a real audience moment can survive that clip-window averaging without # moving the peak or inventing new peaks. _RETENTION_CLIP_RADIUS_SECONDS = 12 _RETENTION_EDGE_WEIGHT = 0.925 def audience_curve(url: str, video_id: str, duration: float) -> np.ndarray | None: """Per-second 0..1 hype curve for this video, or None when the platform has no fetchable audience data. Never raises.""" return audience_signals(url, video_id, duration)[0] def audience_signals(url: str, video_id: str, duration: float) -> tuple[np.ndarray | None, list]: """(the hype curve, the chat messages it came from). The messages are [(offset seconds, author, text)] when the curve came from chat replay, for the gaming profile's reading of what chat said (analysis/chat_moments.py); empty for the other sources. Never raises.""" try: if video_id.startswith("tw_clip_"): return None, [] if video_id.startswith("tw_"): messages = _twitch_chat(video_id[3:], duration) return _chatters_to_curve(messages, duration), messages if video_id.startswith("kick_"): return None, [] # Kick retains no chat after the stream ends # YouTube: private organic retention is the best signal when the user # has already authorized it. No token / wrong channel / no data is a # normal condition, so each case falls through to the public sources. organic = _youtube_organic_retention(video_id, duration) if organic is not None: return organic, [] heat = _youtube_heatmap(url, duration) if heat is not None: return heat, [] messages = _youtube_live_chat(url, video_id, duration) return _chatters_to_curve(messages, duration), messages except Exception as e: print(f" (audience signal unavailable: {e})") return None, [] # ---- Twitch ---------------------------------------------------------------- def _scrape_client_id() -> str: req = urllib.request.Request("https://www.twitch.tv/", headers={"User-Agent": _UA}) html = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "replace") m = re.search(r'clientId\s*[=:]\s*"([a-z0-9]{20,40})"', html) if not m: raise RuntimeError("could not find Twitch web client id") return m.group(1) _COMMENTS_QUERY = """ query($videoID: ID!, $offset: Int) { video(id: $videoID) { comments(contentOffsetSeconds: $offset) { edges { node { contentOffsetSeconds commenter { id } message { fragments { text } } } } pageInfo { hasNextPage } } } }""" def _twitch_chat(vod_id: str, duration: float) -> list[tuple[float, str, str]]: """(offset_seconds, chatter_id, text) for the whole VOD's chat replay. The text is the message's fragments joined, emotes by their names. Paged by contentOffsetSeconds, NOT cursors: cursor pagination trips Twitch's client-integrity check (2026), while offset queries — 'give me the chat page at second X' — pass. Each page spans until its last message; stepping to last+1 walks the whole VOD. Overlaps are deduped. """ client_id = _scrape_client_id() seen: set[tuple[float, str, str]] = set() offset = 0.0 deadline = time.monotonic() + _TIME_BUDGET_S for _ in range(_MAX_PAGES): body = json.dumps( {"query": _COMMENTS_QUERY, "variables": {"videoID": vod_id, "offset": int(offset)}} ).encode() req = urllib.request.Request( _GQL_URL, data=body, headers={ "Client-ID": client_id, "Content-Type": "application/json", "User-Agent": _UA, }, ) with urllib.request.urlopen(req, timeout=20) as r: data = json.load(r) video = (data.get("data") or {}).get("video") if not video or not video.get("comments"): break edges = video["comments"]["edges"] if not edges: break last = 0.0 for e in edges: node = e["node"] commenter = node.get("commenter") or {} last = max(last, float(node["contentOffsetSeconds"])) # Skip deleted/anonymous accounts — also drops most bot noise. if commenter.get("id"): fragments = ((node.get("message") or {}).get("fragments") or []) text = "".join(str(f.get("text") or "") for f in fragments) seen.add((float(node["contentOffsetSeconds"]), commenter["id"], text)) if last >= duration or last < offset + 1: # done, or no forward progress break offset = last + 1 if time.monotonic() > deadline: print(" (chat fetch time budget hit — using the part fetched)") break return sorted(seen) # ---- YouTube ---------------------------------------------------------------- def _youtube_token_path() -> Path: """The token written by `python main.py auth`. Electron passes CLIPS_STUDIO_DATA_DIR for installed builds. A source checkout with default settings resolves to repo/data. A custom CLI data directory that is not also exported through the environment simply means this optional signal is unavailable, so the public fallback still works. """ override = os.environ.get("CLIPS_STUDIO_DATA_DIR") if override: return Path(override) / "youtube_token.json" from core.paths import resolve_data_dir return resolve_data_dir({}) / "youtube_token.json" def _youtube_organic_retention(video_id: str, duration: float) -> np.ndarray | None: """Creator-only organic audience retention via YouTube Analytics. This function is deliberately non-interactive. It never opens a browser: if the cached token is absent, lacks the Analytics scope, belongs to an account that cannot read this video, or the video has no retention yet, return None and let audience_curve() use the existing public path. """ token_path = _youtube_token_path() if not token_path.exists(): return None try: from google.auth.transport.requests import Request from google.oauth2.credentials import Credentials from googleapiclient.discovery import build creds = Credentials.from_authorized_user_file(str(token_path)) if not creds.has_scopes([_YT_ANALYTICS_SCOPE]): return None if creds.expired and creds.refresh_token: creds.refresh(Request()) token_path.write_text(creds.to_json(), encoding="utf-8") if not creds.valid: return None analytics = build( "youtubeAnalytics", "v2", credentials=creds, cache_discovery=False, ) report = analytics.reports().query( ids="channel==MINE", startDate="2005-04-23", endDate=date.today().isoformat(), metrics="audienceWatchRatio", dimensions="elapsedVideoTimeRatio", filters=f"video=={video_id};audienceType==ORGANIC", sort="elapsedVideoTimeRatio", ).execute() rows = report.get("rows") or [] curve = _retention_rows_to_curve(rows, duration) if curve is not None: print(f" YouTube organic retention loaded ({len(rows)} bins, OAuth)") return curve except Exception: # Expected fallbacks include: token revoked, Analytics API disabled, # someone else's video, and a new/low-view video with no retention. return None def _retention_rows_to_curve(rows: list, duration: float) -> np.ndarray | None: """Convert Analytics retention rows into Clips Kitty's per-second 0..1 signal. audienceWatchRatio naturally slopes down through almost every video. Using the raw values would therefore label the intro as the hottest part even when viewers never replayed or held there. Instead each 1%-of-video point is compared with the median on both sides (the same local-prominence idea validated against YouTube Studio), then those lifts are percentile-ranked within the video, matching the normalization convention used by the other Clips Kitty signals. The first 5% is deliberately suppressed: startup retention is dominated by autoplay, immediate exits and values over 100%, not a meaningful "moment". Finally, the per-second prominence curve is converted into a clip-scale interest envelope. Clips Kitty averages audience values over a whole clip; a genuine 1%-bin retention spike would otherwise be diluted almost to nothing inside a 25-60s candidate. The envelope keeps the maximum at the exact same second and lets its influence decay gently for 12s on either side. This is analogous to turning a point measurement into the short "hot zone" that the existing Most Replayed/chat curves already represent. """ if duration <= 0 or len(rows) < 5: return None points: list[tuple[float, float]] = [] for row in rows: try: ratio = float(row[0]) watch = float(row[1]) except (TypeError, ValueError, IndexError): continue if np.isfinite(ratio) and np.isfinite(watch) and 0.0 <= ratio <= 1.0: points.append((ratio, watch)) if len(points) < 5: return None points.sort(key=lambda p: p[0]) # Duplicate ratios are not expected, but collapsing them avoids np.interp # ambiguity if the API ever returns one. collapsed: list[tuple[float, float]] = [] for ratio, watch in points: if collapsed and abs(collapsed[-1][0] - ratio) < 1e-9: collapsed[-1] = (ratio, max(collapsed[-1][1], watch)) else: collapsed.append((ratio, watch)) ratios = np.asarray([p[0] for p in collapsed], dtype=np.float32) values = np.asarray([p[1] for p in collapsed], dtype=np.float32) n = values.size if n < 5: return None lift = np.empty(n, dtype=np.float32) for i, current in enumerate(values): left = values[max(0, i - _RETENTION_BASELINE_RADIUS):i] right = values[i + 1:min(n, i + _RETENTION_BASELINE_RADIUS + 1)] if left.size and right.size: # Using the higher side is conservative on a naturally declining # retention curve: only an actual local rise earns a strong lift. baseline = max(float(np.median(left)), float(np.median(right))) elif left.size: baseline = float(np.median(left)) elif right.size: baseline = float(np.median(right)) else: baseline = float(current) lift[i] = float(current) - baseline # The intro behaves unlike the rest of a retention graph; make it cold # before percentile ranking rather than letting 100%+ startup values win. intro = ratios < _RETENTION_IGNORE_START_RATIO if intro.any(): floor = float(lift.min()) - max(float(np.ptp(lift)), 1e-6) - 1.0 lift[intro] = floor # Same percentile-rank convention as analysis/fusion.py::_pct. order = lift.argsort(kind="stable").argsort(kind="stable").astype(np.float32) ranked = order / max(n - 1, 1) seconds = np.arange(max(int(duration) + 1, 2), dtype=np.float32) sample_seconds = np.clip(ratios * float(duration), 0.0, float(duration)) if sample_seconds[0] > 0: sample_seconds = np.insert(sample_seconds, 0, 0.0) ranked = np.insert(ranked, 0, ranked[0]) if sample_seconds[-1] < duration: sample_seconds = np.append(sample_seconds, float(duration)) ranked = np.append(ranked, ranked[-1]) point_curve = np.interp(seconds, sample_seconds, ranked).astype(np.float32) return _clip_scale_interest(point_curve) def _clip_scale_interest(curve: np.ndarray) -> np.ndarray: """Turn point-like retention prominence into short clip-scale hot zones. For every second, nearby high-retention values can raise its interest, with a linear decay from 1.00 at the measured peak to 0.925 at +/-12 seconds. This is a MAX envelope, not smoothing/averaging: measured peaks never move and weaker neighbours cannot reduce them. Values stay in 0..1. Why 12 seconds: Clips Kitty grows signal candidates to ~25 seconds. A +/-12s envelope therefore makes one genuine retention peak meaningful over approximately one candidate-sized window, while remaining much narrower than the 60s maximum clip length. """ if curve.size == 0: return curve radius = max(0, int(_RETENT - analysis/intent.py (33761 bytes)
"""A clip direction: what the person asked the clips to be about (#106). "Make sure there's a clip of when I died to the boss, prioritize funny moments, and more from the WoW part around 1:35." Typed into the app's bottom box (the assistant passes it as `focus` with the job) or sent on the API, it becomes a ClipIntent: - targets: a topic that gets talked about, or a moment that happened, each with the words someone would SAY when it comes up. The model writes those once, which is how "WoW" also finds "Azeroth" without a search index; - spans: time ranges ("1:35-1:55", "around 45 minutes", "near the end"), read by this code and never by the model; - styles: funny, laughing, hype, reactions, arguments, emotional; - counts: "at least two clips about ...". It only ever ADDS: points for clips that match, candidate windows where a target is talked about or inside an asked-for range, and a lift to the quality bar for a must-have that was actually found. Nothing is lowered or removed and the scoring prompts never see it, so a video queued without a direction is processed exactly as before (Colin, 2026-09-28: "I don't want it to interfere with current stuff if you just want to clip the stream"). "Avoid the intro" is understood and reported as not applied. Nothing is invented. A must-have whose words are never said is reported as not found, and no clip is made for it. Pure stdlib, so it tests on a CI runner without numpy. """ import json import re from dataclasses import dataclass, field MAX_CHARS = 600 # Points a clip gets, by how strongly it was asked for. Beside the other # additive layers (creator context +6, audience hype +8, the game +12) a # direction weighs more, because the person said so for this video. TOPIC_POINTS = {"must": 15, "strong": 15, "prefer": 8} RANGE_POINTS = {"must": 12, "strong": 12, "prefer": 6} STYLE_POINTS = {"strong": 10, "prefer": 6} # All of it together, for one clip: enough to tip close calls and lift what was # asked for, not so much that the scoring underneath stops meaning anything. CAP = 25 # Windows added for one target or range. A word said all stream long would # otherwise send the model hundreds of windows to score. WINDOWS_PER_TARGET = 12 WINDOW_SECONDS = 30.0 # "Around 1:35" covers five minutes either side of it. POINT_REACH = 300.0 # "The intro", "near the end": the first or last 15% of the video. EDGE_SHARE = 0.15 # A word said in more than a quarter of the transcript marks nothing. TOO_COMMON = 0.25 STYLES = ("funny", "laughing", "hype", "reactions", "arguments", "emotional") STRENGTHS = ("must", "strong", "prefer") STYLE_LABELS = {"funny": "funny moments", "laughing": "laughing", "hype": "hype moments", "reactions": "reactions", "arguments": "arguments", "emotional": "emotional moments"} # Said all the time in any stream, whatever it is about. "WoW" written with its # capitals still counts; "wow" the interjection does not. _FILLER = { "wow", "ok", "okay", "yeah", "yes", "no", "lol", "omg", "bro", "dude", "man", "like", "so", "game", "games", "play", "playing", "stream", "video", "chat", "guys", "thing", "stuff", } _STOP = { "a", "an", "the", "and", "or", "but", "of", "to", "in", "on", "at", "for", "from", "with", "about", "into", "around", "near", "this", "that", "these", "those", "it", "its", "is", "was", "are", "were", "be", "been", "i", "me", "my", "we", "our", "you", "your", "he", "she", "they", "them", "his", "her", "their", "what", "when", "where", "which", "who", "how", "there", "please", "make", "sure", "clip", "clips", "clipping", "want", "wanted", "would", "like", "create", "show", "pick", "choose", "grab", "cut", "some", "more", "most", "moments", "moment", "part", "parts", "segment", "section", "bit", "find", "give", "get", "include", "including", "focus", "prioritize", "prioritise", "priority", "weight", "higher", "lots", "lot", "few", "least", "much", "many", "really", "very", "also", "then", "than", "just", "only", "all", "any", "one", "two", "three", "four", "five", "talk", "talks", "talked", "talking", "said", "say", "says", "happened", "stream", "video", "vod", "minutes", "minute", "hours", "hour", "mark", "start", "end", "funny", "funniest", "laughing", "laugh", "laughs", "hype", "reactions", "reaction", "arguments", "argument", "emotional", "stuff", "things", "thing", "got", "did", "do", "does", "can", "could", "should", "will", "not", "don't", "dont", "avoid", "skip", "starts", "started", "starting", "long", "short", "good", "best", "boring", "big", "little", "whole", "found", "going", "back", "here", "able", "let", "lets", "let's", "try", "maybe", "use", "used", "using", "top", "great", "cool", "nice", "has", "have", "had", "being", "first", "last", "next", "ever", "every", "each", "other", "onto", "strong", "stronger", "weak", "huge", "discussion", "discussions", "conversation", "bits", "sections", "segments", "wants", "need", "needs", "put", "add", "keep", } # Ordinary words that are part of many names ("Ocarina of Time", "Call of # Duty"): never a search word on their own. _COMMON = { "time", "world", "war", "wars", "day", "days", "night", "life", "home", "way", "people", "year", "years", "new", "old", "great", "first", "last", "best", "high", "low", "hand", "head", "house", "city", "land", "king", "queen", "lord", "legend", "story", "call", "duty", "dark", "light", "star", "stars", "final", "fantasy", "grand", "auto", "theft", "league", "legends", "counter", "strike", "battle", "boss", "died", "death", "dead", "clip", "when", } _NUMBERS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9, "ten": 10, "a couple of": 2, "a couple": 2, "a few": 3} _MUST = re.compile(r"\b(make sure|must|need|needs|definitely|include|including|don'?t miss|" r"don'?t forget|has to|have to|guarantee)\b", re.I) _STRONG = re.compile(r"\b(focus|prioriti[sz]e|priority|weight|mostly|more clips|lots of|a lot|" r"really want|higher|heavily|especially)\b", re.I) # An exclusion. "Don't miss" and "don't forget" ask for something, not against it. _NEGATED = re.compile(r"\b(avoid|skip|without|exclude|leave out|never|no more|" r"don'?t (?!miss|forget)|do not (?!miss|forget)|not (?:the|any))\b", re.I) _STYLE_WORDS = { "funny": r"\b(funn(?:y|iest|ier)|hilarious|jokes?|joking|comed(?:y|ic)|humou?r)\b", "laughing": r"\b(laugh\w*|giggl\w*|cracking up)\b", "hype": r"\b(hype|hyped|crazy|insane|epic|intense|chat went|chat (?:going|goes) (?:crazy|wild)|" r"clutch|exciting)\b", "reactions": r"\b(react\w*|surprised?|shock\w*)\b", "arguments": r"\b(argu\w*|fights?|fighting|debat\w*|drama|beef|disagree\w*|heated)\b", "emotional": r"\b(emotional|sad|cry\w*|tears|wholesome|heartfelt|touching)\b", } # What a clip's own words show, for the styles no signal measures on its own. _LAUGH_TEXT = re.compile(r"\b(haha\w*|hehe\w*|lmao|lol|laugh\w*|hilarious)\b|\[laugh|\(laugh", re.I) _ARGUE_TEXT = re.compile(r"\b(no way|you'?re wrong|shut up|argu\w*|disagree\w*|that'?s not true|" r"are you serious|what are you talking about)\b", re.I) _EMOTION_TEXT = re.compile(r"\b(cry\w*|tears|love you|miss (?:you|him|her|them)|proud of|" r"thank you so much|emotional|means a lot)\b", re.I) _CLOCK = r"\d{1,2}:\d{2}(?::\d{2})?" # Each unit ends on a word boundary: "5 more clips" is not five minutes. _VERBAL = (r"\d+(?:\.\d+)?\s*(?:h|hr|hrs|hours?)\b(?:\s*(?:and\s*)?\d+\s*(?:m|mins?|minutes?)\b)?" r"|\d+\s*(?:m|mins?|minutes?)\b(?:\s*(?:and\s*)?\d+\s*(?:s|secs?|seconds?)\b)?") _TIME = rf"(?:{_CLOCK}|{_VERBAL})" _RANGE = re.compile(rf"({_TIME})\s*(?:-|–|—|to|until|till|through)\s*({_TIME})", re.I) _POINT = re.compile(rf"(?<![\d:])({_TIME})(?![\d:])", re.I) _FIRST_PART = re.compile(r"\b(?:the )?(?:intro|beginning|opening)\b|\b(?:at|near|from) the start\b|" r"\bfirst (?:part|few minutes|bit)\b", re.I) _LAST_PART = re.compile(r"\b(?:near|at|towards?) the end\b|\bthe (?:ending|outro|last part)\b|" r"\bend of the (?:stream|video)\b|\blast (?:part|few minutes|bit)\b", re.I) _COUNT = re.compile(r"\b(?:at least|minimum(?: of)?|min(?:imum)?\.?)?\s*" r"(\d+|one|two|three|four|five|six|seven|eight|nine|ten|a couple(?: of)?|a few)" r"\s+(?:more\s+|good\s+|funny\s+|short\s+)?clips?\b", re.I) SCHEMA = { "type": "object", "properties": { "targets": { "type": "array", "items": { "type": "object", "properties": { "what": {"type": "string"}, "kind": {"type": "string", "enum": ["topic", "moment"]}, "strength": {"type": "string", "enum": list(STRENGTHS)}, "terms": {"type": "array", "items": {"type": "string"}}, }, "required": ["what", "kind", "strength", "terms"], "additionalProperties": False, }, }, "styles": { "type": "array", "items": { "type": "object", "properties": { "style": {"type": "string", "enum": list(STYLES)}, "strength": {"type": "string", "enum": ["strong", "prefer"]}, }, "required": ["style", "strength"], "additionalProperties": False, }, }, "avoid": {"type": "array", "items": {"type": "string"}}, }, "required": ["targets", "styles", "avoid"], # Strict structured output (OpenAI, and OpenRouter models that take it) # refuses a schema without this on every object. A refusal falls back to # plain JSON and is remembered for the model, so the clip scoring's own # schema would be dropped with it for the rest of the session. "additionalProperties": False, } PROMPT = """Someone is using a video clipping app. They wrote what they want clipped from ONE video (a stream or recording of them). Turn it into JSON. Only use what they wrote: never add topics or moments of your own. - targets: each thing they want clips of. - what: a short name for it, in their words. - kind: "moment" for one thing that happened (a death, a win, a reveal), "topic" for a subject they talk about. - strength: "must" when they say make sure / include / need; "strong" for focus on / prioritize / more of; otherwise "prefer". - terms: 3 to 10 words or short phrases someone would SAY out loud in the video when this comes up: the name, its abbreviation, nicknames, places, characters and closely related words. (For example, "Minecraft" would get "Minecraft", "creeper", "Nether", "diamonds", "Ender Dragon", "villager"; a cake disaster would get "cake", "oven", "burnt", "collapsed", "frosting".) Keep an abbreviation's capital letters. - styles: which of funny, laughing, hype, reactions, arguments, emotional they asked for, each with strength "strong" or "prefer". - avoid: anything they asked to leave out or avoid, in their words. If they only gave a time or a style, "targets" is an empty list. Never copy the examples above into your answer. Leave times ("1:35", "around 45 minutes") and numbers of clips out of everything: those are read separately. Answer with only the JSON object: {"targets": [...], "styles": [...], "avoid": [...]} What they wrote: \"\"\"{text}\"\"\" """ @dataclass class Target: what: str kind: str = "topic" # topic | moment strength: str = "prefer" # must | strong | prefer terms: list[str] = field(default_factory=list) count: int = 0 # "at least N clips" of it; 0 when no number was given spots: list[float] = field(default_factory=list) # where it's said, set by locate() patterns: list = field(default_factory=list, repr=False) def label(self) -> str: return self.what @dataclass class Span: start: float end: float strength: str = "prefer" count: int = 0 words: str = "" # what they wrote for it def label(self) -> str: return f"{clock(self.start)}-{clock(self.end)}" def holds(self, c) -> bool: return self.start <= (c.start + c.end) / 2 <= self.end @dataclass class ClipIntent: text: str targets: list[Target] = field(default_factory=list) spans: list[Span] = field(default_factory=list) styles: dict[str, str] = field(default_factory=dict) # style -> strong | prefer not_applied: list[str] = field(default_factory=list) not_found: list[str] = field(default_factory=list) notes: list[str] = field(default_factory=list) fallback: bool = False # the model's answer was unusable; read from the words boosted: int = 0 added: int = 0 required: int = 0 def understood(self) -> list[str]: lines = [] for t in self.targets: head = "Must include" if t.strength == "must" or t.count else ( "Focus on" if t.strength == "strong" else "More of") lines.append(f"{head}: {t.what}{_at_least(t.count)}") for s in self.spans: head = "Must include" if s.strength == "must" or s.count else ( "Focus on" if s.strength == "strong" else "More from") lines.append(f"{head}: {s.label()}{_at_least(s.count)}") for style, strength in self.styles.items(): lines.append(f"{'Focus on' if strength == 'strong' else 'More'} {STYLE_LABELS[style]}") return lines def report(self) -> dict: """What was understood and done, for the video's outcome and the UI.""" return { "direction": self.text, "understood": self.understood(), "not_found": list(self.not_found), "not_applied": list(self.not_applied), "notes": list(self.notes), "boosted": self.boosted, "added_windows": self.added, "required": self.required, } def _at_least(n: int) -> str: # One is what a must-have already means, so only a real count is shown. return f" (at least {n} clips)" if n > 1 else "" def clock(seconds: float) -> str: s = max(0, int(round(seconds))) return f"{s // 3600}:{s % 3600 // 60:02d}:{s % 60:02d}" if s >= 3600 else f"{s // 60}:{s % 60:02d}" # ---- reading the direction --------------------------------------------------- def parse(text: str, llm, duration: float) -> ClipIntent | None: """The direction as a ClipIntent, or None when nothing was written. One model call, for what only a model can do (the targets and the words that mark them). Times, counts and exclusions are read here, and a model answer that comes back empty or broken falls back to the words themselves, so a direction always counts for something.""" tex - analysis/metadata.py (13457 bytes)
"""LLM-generated upload metadata: title, description, hashtags. Never fails the pipeline: any LLM misbehavior falls back to metadata derived from the clip's hook and the source video title. """ import json import re import unicodedata from dataclasses import dataclass, field from pathlib import Path from core.models import ClipCandidate, Segment from llm.base import LLMBackend, generate_json PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "metadata.txt" MAX_TITLE_LEN = 95 # leave headroom under YouTube's 100-char limit # The shapes metadata.txt and metadata_batch.txt ask for. A cloud model on the # user's key is held to them (llm.base.generate_json); _parse still checks. _METADATA_FIELDS = { "title": {"type": "string"}, "description": {"type": "string"}, "hashtags": {"type": "array", "items": {"type": "string"}}, } METADATA_SCHEMA = { "type": "object", "properties": _METADATA_FIELDS, "required": ["title", "description", "hashtags"], "additionalProperties": False, } BATCH_SCHEMA = { "type": "object", "properties": {"items": {"type": "array", "items": { "type": "object", "properties": {"index": {"type": "integer"}, **_METADATA_FIELDS}, "required": ["index", "title", "description", "hashtags"], "additionalProperties": False, }}}, "required": ["items"], "additionalProperties": False, } @dataclass class ClipMetadata: title: str description: str hashtags: list[str] = field(default_factory=list) def generate_metadata( candidate: ClipCandidate, segments: list[Segment], video_title: str, llm: LLMBackend, ) -> ClipMetadata: clip_text = " ".join( s.text for s in segments if s.end > candidate.start and s.start < candidate.end ) fallback = _fallback(candidate, video_title) if not clip_text: return fallback prompt = ( PROMPT_PATH.read_text(encoding="utf-8") .replace("{video_title}", video_title) .replace("{clip_text}", clip_text) ) try: raw = generate_json(llm, prompt, METADATA_SCHEMA) parsed = _parse(raw) except Exception: parsed = None if parsed is None: return fallback title = _clean_title(parsed.get("title", "")) or fallback.title description = str(parsed.get("description", "")).strip() or fallback.description hashtags = _clean_hashtags(parsed.get("hashtags", [])) or fallback.hashtags return ClipMetadata(title=title, description=description, hashtags=hashtags) BATCH_PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "metadata_batch.txt" def generate_metadata_batch( candidates: list[ClipCandidate], segments: list[Segment], video_title: str, llm: LLMBackend, batch_size: int = 8, creator_context: str = "", rules: str = "", ) -> list[ClipMetadata]: """Metadata for ALL clips in a few LLM calls instead of one per clip — on a long stream this cuts dozens of model calls from the analysis time. Any clip the model skips or mangles falls back to hook-based metadata. creator_context (optional): learned facts about the creator — series names, running jokes, collaborators — for more accurate titles/hashtags. rules (optional): a sport's own rules for its clips' titles (basketball: which player to name); "" for every other job, whose prompt is unchanged.""" results: list[ClipMetadata] = [_fallback(c, video_title) for c in candidates] template = BATCH_PROMPT_PATH.read_text(encoding="utf-8") if rules: template = template.replace("{clips}", "RULES FOR THESE CLIPS:\n" + rules + "\n\n{clips}") if creator_context: template = template.replace( "{clips}", "CREATOR CONTEXT (background knowledge — use for accuracy when relevant," " never invent beyond it):\n" + creator_context + "\n\n{clips}", ) for base in range(0, len(candidates), batch_size): batch = candidates[base : base + batch_size] blocks = [] for i, c in enumerate(batch): text = _clip_text(c, segments)[:900] blocks.append(f"CLIP {i}{_scoreboard_note(c)}{_plugin_notes(c)}:\n{text or '(no speech)'}") prompt = ( template.replace("{video_title}", video_title) .replace("{count}", str(len(batch))) .replace("{clips}", "\n\n".join(blocks)) ) try: data = _parse(generate_json(llm, prompt, BATCH_SCHEMA)) except Exception: data = None if not data or not isinstance(data.get("items"), list): continue # whole batch falls back for item in data["items"]: try: idx = int(item.get("index", -1)) except (TypeError, ValueError): continue if not 0 <= idx < len(batch): continue fallback = results[base + idx] results[base + idx] = ClipMetadata( title=_clean_title(item.get("title", "")) or fallback.title, description=str(item.get("description", "")).strip() or fallback.description, hashtags=_clean_hashtags(item.get("hashtags", [])) or fallback.hashtags, ) for c, m in zip(candidates, results): _earned(c, m) return results _PERIODS = {"Q1": "1st quarter", "Q2": "2nd quarter", "Q3": "3rd quarter", "Q4": "4th quarter", "H1": "1st half", "H2": "2nd half", "OT": "overtime"} _SCORE = re.compile(r"\bscore (\d+-\d+)") # The points a score change was worth, as ScoreChange.label() writes it ("score 81-79 (SAS), +2"). _POINTS = re.compile(r"\bscore \d+-\d+(?: \([^)]*\))?, \+(\d)\b") _CRUNCH = 120 # seconds left in the last quarter (or overtime) that make crunch time def _crunch(c: ClipCandidate) -> bool | None: """Whether the scoreboard says a clip's moment came in crunch time: the last two minutes of the 4th quarter (or 2nd half) or overtime. None for a clip with no game clock read (every sport but basketball).""" when = str((c.subscores or {}).get("sport_when") or "") if not when: return None period, _, left = when.partition(" ") try: minutes, seconds = (int(x) for x in left.split(":")) return period in ("Q4", "H2", "OT") and minutes * 60 + seconds <= _CRUNCH except ValueError: return period in ("Q4", "H2", "OT") def _scoreboard_note(c: ClipCandidate) -> str: """What the game's own scoreboard says about a clip's moment, for its title: the play, the team, the score in words (whose is whose and who leads), the quarter and the clock, and when it was not crunch time. On an NBA game the titles called a 3rd quarter put-back "Late-Game" and a shot with 11:30 left "Clutch". Only a sport that reads the game's clock (basketball) sets one; "" for every other clip, whose prompt is unchanged.""" crunch = _crunch(c) if crunch is None: return "" s = c.subscores or {} period, _, left = str(s["sport_when"]).partition(" ") play = str(s.get("sport_label") or "a play") team = str(s.get("sport_team") or "") context = str(s.get("sport_context") or "") why = str(s.get("sport_why") or "") score, points = _SCORE.search(why), _POINTS.search(why) # Its points: on an NBA game a "Step-back" two was described as a three. words = [play + (f" ({points.group(1)} points)" if points else "") + (f" by {team}" if team else "")] # The score only with whose is whose: "making it 97-86" alone had the # titles put the team the commentary named ahead, the wrong one. if score and not context and team: words[0] += f", making it {score.group(1)}" if context: words.append(context) if points: # Who scored, as the commentary says it (sports/basketball/commentary.py): # left to the model, a three went to the player who passed for it. player = str(s.get("sport_player") or "") words.append(f"the commentary says {player} scored it" if player else "the commentary doesn't say who scored it") words.append(_PERIODS.get(period, period) + (f" with {left} left" if left else "")) if not crunch: words.append("not crunch time, so not clutch or late-game") return " (the scoreboard: " + "; ".join(words) + ")" # A note as plugins/steps.py keeps it: the same rules as the SDK's # host.clean_note, here so analysis/ never imports plugins/. _NOTE_LEN = 160 _NOTES_LEN = 400 _LINK = re.compile(r"(?:https?://|\bwww\.)\S+", re.IGNORECASE) def _one_line(text: str) -> str: """A plugin's note on one line: control characters removed, links taken out, whitespace and newlines collapsed, cut to 160 characters.""" kept = "".join(ch for ch in str(text or "") if ch.isspace() or unicodedata.category(ch) != "Cc") return " ".join(_LINK.sub(" ", " ".join(kept.split())).split())[:_NOTE_LEN].rstrip() def _plugin_notes(c: ClipCandidate) -> str: """What Marketplace plugins said happens in a clip's moment (plugins/steps.py), for its title. "" for every clip without notes, whose prompt is unchanged.""" seen, parts = set(), [] for n in (c.subscores or {}).get("plugin_notes") or []: text = _one_line(str(n.get("text") or "")) if isinstance(n, dict) else "" if text and text not in seen: seen.add(text) parts.append(text) if not parts: return "" return (" (notes from Marketplace plugins about this moment, background only," " never invent beyond them: " + "; ".join(parts)[:_NOTES_LEN] + ")") def _clip_text(c: ClipCandidate, segments: list[Segment]) -> str: """What is said in a clip, for its title: the sentences it overlaps. A clip with a game clock read (basketball) gets only the words said inside it: a highlights package's commentary runs on from play to play, and on an NBA game the sentences around 12 s clips had the titles name players from the plays before and after them.""" if _crunch(c) is None: return " ".join(s.text for s in segments if s.end > c.start and s.start < c.end) words = [] for s in segments: if s.end <= c.start or s.start >= c.end: continue if not s.words: words.append(s.text) continue words += [w["word"] for w in s.words if c.start <= (w["start"] + w["end"]) / 2 <= c.end] return " ".join(words) # What only crunch time earns. On an NBA game, with the note saying it wasn't # crunch time, a 2nd-quarter basket was still titled "Clutch", one with 7:49 # left "Late-Game" and a three at +12 "Game-Changing". _CLUTCH = re.compile(r"\b(?:clutch|late[- ]game|crunch[- ]time|game[- ]changing)\b[ \t]*", re.IGNORECASE) def _earned(c: ClipCandidate, metadata: ClipMetadata) -> None: """A title and description that don't call a clip clutch (or late-game, crunch-time, game-changing) when the scoreboard says it wasn't crunch time. Every other clip as written.""" if _crunch(c) is not False: return title = re.sub(r"\s{2,}", " ", _CLUTCH.sub("", metadata.title)).strip() metadata.title = title if re.search(r"\w", title) else (_clean_title(c.hook) or metadata.title) metadata.description = re.sub(r"[ \t]{2,}", " ", _CLUTCH.sub("", metadata.description)).strip() def _fallback(candidate: ClipCandidate, video_title: str) -> ClipMetadata: """What a clip carries when the model could not write its metadata. No description and no hashtags. This used to be "Clip from: <video title>" with #clips, and every clip of a video whose metadata failed got that same caption: seven TikTok posts went out reading "Clip from: my brother exposes me… #clips" and were flagged as unoriginal content, which is exactly what a caption announcing a repost invites. The title still comes from the hook, and the creator's and required hashtags are added where they always were. """ title = _clean_title(candidate.hook) or _clean_title(video_title) or "Clip" return ClipMetadata(title=title, description="", hashtags=[]) def _parse(raw: str) -> dict | None: text = raw.strip() text = re.sub(r"^```(?:json)?\s*|\s*```$", "", text) start, end = text.find("{"), text.rfind("}") if start == -1 or end <= start: return None try: data = json.loads(text[start : end + 1]) except json.JSONDecodeError: return None return data if isinstance(data, dict) else None def _clean_title(title: str) -> str: title = re.sub(r"[<>]", "", str(title)).strip().strip('"') return title[:MAX_TITLE_LEN].strip() def _clean_hashtags(tags) -> list[str]: """One hashtag per entry, at most five. The model sometimes answers with several run together in one string, "#creatorname#drama#apology", and keeping that as one tag put it into posts as a single unreadable hashtag. Split on every # and every space. """ if not isinstance(tags, list): return [] cleaned: list[str] = [] for entry in tags: for part in re.split(r"[#\s]+", str(entry).lower()): word = re.sub(r"[^\w]", "", part) tag = f"#{word}" if word and tag != "#shorts" and tag not in cleaned: cleaned.append(tag) return cleaned[:5] - analysis/panns.py (6375 bytes)
"""PANNs MobileNetV1: an AudioSet sound tagger, for game sounds. The network is Kong et al.'s MobileNetV1 from "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition" (2020), adapted from qiuqiangkong/audioset_tagging_cnn, pytorch/models.py: Copyright (c) 2018-2020 Qiuqiang Kong. MIT License: permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. Not in third_party/: that folder holds verbatim copies, and upstream builds its front end on torchlibrosa and librosa, which this app doesn't ship. The layers and their names are upstream's, so the published checkpoint loads as-is; what changed is the front end (torch.stft, and the mel filterbank the checkpoint carries), and training-only parts (SpecAugment, mixup, dropout) are gone. The weights (Zenodo record 3987831, CC BY 4.0) are fetched by scripts/fetch_panns.py. """ import sys from pathlib import Path import numpy as np _ROOT = Path(__file__).resolve().parent.parent WEIGHTS = "panns_mobilenetv1.pth" LABELS = _ROOT / "config" / "audioset_labels.txt" # What the network was trained on; none of these are free parameters. SAMPLE_RATE = 32000 N_FFT = 1024 HOP = 320 MEL_BINS = 64 CLASSES = 527 _model = None def weights_path() -> Path: """Where the checkpoint lives, frozen build or checkout.""" if getattr(sys, "frozen", False): return Path(sys._MEIPASS) / WEIGHTS # type: ignore[attr-defined] return _ROOT / "models" / WEIGHTS def available() -> bool: """True when game sounds can be listened for. Without the weights the gaming profile goes on without this signal.""" try: import torch # noqa: F401 except ImportError: return False return weights_path().exists() def labels() -> list[str]: """The 527 AudioSet class names, in the network's output order.""" return LABELS.read_text(encoding="utf-8").splitlines() def _network(): import torch from torch import nn def conv_bn(inp, oup, stride): return nn.Sequential(nn.Conv2d(inp, oup, 3, 1, 1, bias=False), nn.AvgPool2d(stride), nn.BatchNorm2d(oup), nn.ReLU(inplace=True)) def conv_dw(inp, oup, stride): return nn.Sequential(nn.Conv2d(inp, inp, 3, 1, 1, groups=inp, bias=False), nn.AvgPool2d(stride), nn.BatchNorm2d(inp), nn.ReLU(inplace=True), nn.Conv2d(inp, oup, 1, 1, 0, bias=False), nn.BatchNorm2d(oup), nn.ReLU(inplace=True)) class MobileNetV1(nn.Module): def __init__(self): super().__init__() self.register_buffer("window", torch.hann_window(N_FFT, periodic=True)) self.register_buffer("melW", torch.zeros(N_FFT // 2 + 1, MEL_BINS)) self.bn0 = nn.BatchNorm2d(MEL_BINS) self.features = nn.Sequential( conv_bn(1, 32, 2), conv_dw(32, 64, 1), conv_dw(64, 128, 2), conv_dw(128, 128, 1), conv_dw(128, 256, 2), conv_dw(256, 256, 1), conv_dw(256, 512, 2), conv_dw(512, 512, 1), conv_dw(512, 512, 1), conv_dw(512, 512, 1), conv_dw(512, 512, 1), conv_dw(512, 512, 1), conv_dw(512, 1024, 2), conv_dw(1024, 1024, 1)) self.fc1 = nn.Linear(1024, 1024, bias=True) self.fc_audioset = nn.Linear(1024, CLASSES, bias=True) def forward(self, wave): """wave: (batch, samples) at 32 kHz -> (batch, 527) probabilities.""" spec = torch.stft(wave, N_FFT, HOP, N_FFT, self.window, center=True, pad_mode="reflect", return_complex=True) power = spec.real ** 2 + spec.imag ** 2 # (batch, freq, time) mel = torch.matmul(power.transpose(1, 2), self.melW) # (batch, time, mel) x = (10.0 * torch.log10(torch.clamp(mel, min=1e-10))).unsqueeze(1) x = self.bn0(x.transpose(1, 3)).transpose(1, 3) x = self.features(x) x = torch.mean(x, dim=3) x = torch.max(x, dim=2)[0] + torch.mean(x, dim=2) x = torch.relu(self.fc1(x)) return torch.sigmoid(self.fc_audioset(x)) return MobileNetV1() def load(): """The network with its weights, on the GPU when the build can use one. Cached: loading costs a second, a video only needs it once.""" global _model if _model is not None: return _model import torch # NOT torch.cuda.is_available(): see core/gpu.py. from core.gpu import torch_device device = torch_device() state = torch.load(weights_path(), map_location="cpu", weights_only=True) state = state.get("model", state) model = _network() # The checkpoint's STFT is a convolution with the window baked in # (torchlibrosa); torch.stft computes the same thing, so only its mel # filterbank is kept. state = {k: v for k, v in state.items() if not k.startswith("spectrogram_extractor.")} state["melW"] = state.pop("logmel_extractor.melW") state["window"] = model.window model.load_state_dict(state) model.eval().to(device) _model = model print(f" Game sounds (PANNs) loaded on {device}") return _model def tag(windows: np.ndarray, batch: int = 32) -> np.ndarray: """(n, samples) float32 audio at 32 kHz -> (n, 527) probabilities.""" import torch model = load() device = next(model.parameters()).device out = [] with torch.inference_mode(): for i in range(0, len(windows), batch): x = torch.from_numpy(np.ascontiguousarray(windows[i:i + batch], dtype=np.float32)).to(device) out.append(model(x).float().cpu().numpy()) return np.concatenate(out) if out else np.zeros((0, CLASSES), dtype=np.float32) - analysis/tighten.py (4856 bytes)
"""Propose cuts that tighten a clip: dead air and filler words. Retention on short-form is decided in the first seconds, and a clip that opens with two seconds of "uhh..." has spent them. This finds the pauses and the filler words and returns the ranges worth dropping. It PROPOSES only. The ranges go into the editor's normal edit list, drawn on the timeline like any hand-made cut, so they can be reviewed, adjusted or undone before anything is rendered — cutting someone's video silently on a guess is not a trade worth making. Silence comes from FFmpeg's silencedetect rather than from gaps in the transcript: a gap between words is not necessarily quiet, and cutting music, laughter or a held reaction would be worse than leaving the pause. """ import re import subprocess from pathlib import Path from core.binaries import ffmpeg # Sounds with no meaning to lose. Deliberately short: "like", "so" and # "you know" are frequently load-bearing in speech, and cutting them # mangles sentences. FILLERS = {"um", "uh", "umm", "uhh", "uhm", "erm", "ah", "eh", "hmm", "mm", "mmm"} MIN_GAP = 0.55 # a pause shorter than this is natural speech rhythm KEEP_BREATH = 0.15 # leave this much of the pause, so cuts don't clip breath MIN_CUT = 0.20 # not worth a cut below this MIN_KEEP = 0.30 # never leave an unwatchable sliver behind def _detect_silence(path: Path, noise_db: int = -32, min_gap: float = MIN_GAP) -> list[list[float]]: """Quiet stretches in the audio, as [start, end] seconds.""" cmd = [ ffmpeg(), "-hide_banner", "-nostats", "-i", str(path.resolve()), "-af", f"silencedetect=noise={noise_db}dB:d={min_gap:g}", "-f", "null", "-", ] try: r = subprocess.run( cmd, capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=300 ) except Exception: return [] out: list[list[float]] = [] start = None for m in re.finditer(r"silence_(start|end):\s*(-?[\d.]+)", r.stderr or ""): kind, value = m.group(1), float(m.group(2)) if kind == "start": start = value elif start is not None: out.append([max(0.0, start), value]) start = None if start is not None: # silence running to the end of the clip out.append([max(0.0, start), float("inf")]) return out def _filler_ranges(words: list[dict]) -> list[list[float]]: """Spans of standalone filler sounds, as [start, end] seconds.""" out = [] for w in words or []: token = re.sub(r"[^\w']", "", str(w.get("word", ""))).lower() if token in FILLERS: out.append([float(w["start"]), float(w["end"])]) return out def _merge(ranges: list[list[float]], duration: float) -> list[list[float]]: clean = sorted( [max(0.0, a), min(duration, b)] for a, b in ranges if b > a ) merged: list[list[float]] = [] for a, b in clean: if merged and a <= merged[-1][1] + 0.05: merged[-1][1] = max(merged[-1][1], b) else: merged.append([a, b]) return merged def propose( clip_path: Path, words: list[dict], duration: float, drop_silence: bool = True, drop_fillers: bool = True, ) -> dict: """Keep-ranges with the dead air and filler words taken out. Returns {keep, removed_seconds, cuts, new_duration}; `keep` is exactly the shape render_opts["edit"]["keep"] expects.""" cuts: list[list[float]] = [] if drop_silence: for a, b in _detect_silence(clip_path): b = min(b, duration) # Leave a breath at each end so speech doesn't start abruptly. a2, b2 = a + KEEP_BREATH, b - KEEP_BREATH # A pause at the very start or end can go entirely — there is no # speech on the outer side to protect. if a <= 0.05: a2 = 0.0 if b >= duration - 0.05: b2 = duration if b2 - a2 >= MIN_CUT: cuts.append([a2, b2]) if drop_fillers: cuts.extend(_filler_ranges(words)) cuts = _merge(cuts, duration) keep: list[list[float]] = [] cursor = 0.0 for a, b in cuts: if a - cursor >= MIN_KEEP: keep.append([round(cursor, 2), round(a, 2)]) cursor = max(cursor, b) if duration - cursor >= MIN_KEEP: keep.append([round(cursor, 2), round(duration, 2)]) if not keep: # everything looked droppable — clearly wrong, change nothing return {"keep": [[0.0, round(duration, 2)]], "removed_seconds": 0.0, "cuts": 0, "new_duration": round(duration, 2)} kept = sum(b - a for a, b in keep) return { "keep": keep, "removed_seconds": round(duration - kept, 2), "cuts": len(cuts), "new_duration": round(kept, 2), } - analysis/visual_features.py (5748 bytes)
"""Visual activity signals — no neural nets in the global pass. Global per-second features: motion - mean absolute frame difference (activity/action proxy) scene_cut - hard cuts (frame difference spiking far above its baseline) flash - sudden brightness jumps (gameplay events, explosions, effects) Decoding strategy matters here: OpenCV must decode *every* frame even when sampling (H.264 frames depend on previous frames), which costs ~10 minutes on a 30-minute video. Instead FFmpeg — multithreaded, SIMD — decodes and emits only the sampled frames, already downscaled and grayscale, piped straight into numpy. Same signals, ~20x faster. Reaction scoring (YOLOv8 presence + zoom-in deltas) is computed per candidate window in `reaction_for_window`, not globally — detector time is only spent inside likely clips. """ from pathlib import Path import cv2 import numpy as np from core import cancel from core.binaries import ffmpeg from video.capture import video_capture SAMPLE_FPS = 2.0 FRAME_W, FRAME_H = 160, 90 # analysis resolution; plenty for motion/cuts def extract_visual_features(video_path: Path) -> dict[str, np.ndarray]: frames = _decode_sampled_gray(video_path) if frames.shape[0] < 2: return {} # Per-sample raw signals. diffs = np.abs(np.diff(frames.astype(np.float32), axis=0)).mean(axis=(1, 2)) diffs = np.concatenate([[0.0], diffs]) brightness = frames.mean(axis=(1, 2), dtype=np.float32) flash_raw = np.abs(np.concatenate([[0.0], np.diff(brightness)])) # A hard cut is a frame difference far above the local baseline AND # large in absolute terms (avoids flagging noise in static videos). baseline = _rolling_median(diffs, int(SAMPLE_FPS * 30)) cuts_raw = (diffs > np.maximum(3.0 * baseline, 15.0)).astype(np.float32) # Bin per-sample arrays into per-second arrays. n_secs = int(frames.shape[0] / SAMPLE_FPS) if n_secs == 0: return {} def per_sec(x: np.ndarray, reducer) -> np.ndarray: k = int(SAMPLE_FPS) return reducer(x[: n_secs * k].reshape(n_secs, k), axis=1).astype(np.float32) return { "motion": per_sec(diffs, np.mean), "scene_cut": per_sec(cuts_raw, np.sum), "flash": per_sec(flash_raw, np.max), } def _decode_sampled_gray(video_path: Path) -> np.ndarray: from video.encoding import hwaccel_input_args cmd = [ ffmpeg(), "-v", "error", *hwaccel_input_args(), # NVDEC decode: this pass is decode-bound "-i", str(video_path), "-vf", f"fps={SAMPLE_FPS},scale={FRAME_W}:{FRAME_H}", "-f", "rawvideo", "-pix_fmt", "gray", "-", ] result = cancel.run(cmd) # stops if the video is cancelled (#113) if result.returncode != 0: raise RuntimeError(f"ffmpeg visual decode failed:\n{result.stderr[-1000:].decode(errors='replace')}") buf = np.frombuffer(result.stdout, dtype=np.uint8) n = buf.size // (FRAME_W * FRAME_H) return buf[: n * FRAME_W * FRAME_H].reshape(n, FRAME_H, FRAME_W) def _rolling_median(x: np.ndarray, window: int) -> np.ndarray: half = max(window // 2, 1) out = np.empty_like(x) for i in range(x.size): lo, hi = max(0, i - half), min(x.size, i + half + 1) out[i] = np.median(x[lo:hi]) return out def reaction_for_window( video_path: Path, start: float, end: float, audio_excitement: float, detector: str = "yolov8n.pt", sample_fps: float = 1.0, ) -> float: """Reaction proxy 0..1 for one candidate window: face/person presence x framing changes (zoom-ins / lean-ins) x synchronized audio excitement. Honest limitation: this is not facial-expression recognition (that needs a dedicated FER model, planned as a drop-in upgrade). It measures whether a human is on screen, being emphasized, while the audio is exciting — which correlates strongly with reaction moments in practice. """ from video.tracker import _get_model # reuse the cached YOLO instance model = _get_model(detector) with video_capture(video_path, required=False) as cap: if cap is None: return 0.0 fps = cap.get(cv2.CAP_PROP_FPS) or 30.0 times = np.arange(start, end, 1.0 / sample_fps) presence, areas = [], [] for t in times: cap.set(cv2.CAP_PROP_POS_FRAMES, int(t * fps)) ok, frame = cap.read() if not ok: continue from core.gpu import torch_device from video.tracker import _infer_lock with _infer_lock: # device= is not optional -- see core.gpu.torch_device. # This is the call that crashed a GTX 1060 at the # reactions stage: the model was on CPU, ultralytics put # the predictor back on cuda:0. results = model.predict(frame, classes=[0], conf=0.4, device=torch_device(), verbose=False) best_area = 0.0 for r in results: for b in r.boxes: x1, y1, x2, y2 = b.xyxy[0].tolist() best_area = max(best_area, (x2 - x1) * (y2 - y1) / (frame.shape[0] * frame.shape[1])) presence.append(1.0 if best_area > 0 else 0.0) areas.append(best_area) if not areas: return 0.0 presence_rate = float(np.mean(presence)) # Emphasis: how much the subject's size changes (zoom-ins, lean-ins). area_delta = float(np.abs(np.diff(areas)).sum() / max(np.mean(areas), 1e-6)) if len(areas) > 1 else 0.0 emphasis = min(area_delta / 3.0, 1.0) return presence_rate * (0.4 + 0.6 * emphasis) * (0.5 + 0.5 * audio_excitement) - analysis/voice_turns.py (28664 bytes)
"""Whose voice it is: where in a clip someone other than the video's main speaker is talking, so their captions can take another colour (#126). Nothing else here knows voices apart. The transcript is words and times, and the active-speaker model (video/asd.py) chooses between faces that are on screen together; a second voice is as often a caller or a friend on voice chat, with no face at all. So this goes by the sound of the voice, with two small models (fetched by scripts/fetch_voice_model.py): - a segmentation model (pyannote's segmentation-3.0, MIT) hears ten seconds at a time and marks, 60 times a second, which of up to three speakers is talking. It is what finds the turn: a question and its answer are a second apart with no pause between them. Its speakers are only "first, second, third in these ten seconds", though, not anybody in particular. - a speaker model (WeSpeaker's VoxCeleb ResNet34-LM, CC BY 4.0) turns a few seconds of one voice into a vector that sits close to other speech by the same person. It is what says who a turn belongs to. Once per video the voices in it are found from speech sampled across the whole video, and ranked by how much each talks: the one that talks most is the main speaker, in every clip of that video. In a clip, each speaker the segmentation hears is scored against that main voice, and the words of the ones who are clearly somebody else come back as turns. A solo clip coloured by mistake is the worst this can do, so a doubtful clip comes back with no turns at all: one voice, too little of a second, a clip the main speaker is barely in. Both models run on the onnxruntime the app already ships, one inference at a time on one core: a render is busy encoding on the others. """ import json import os import re import subprocess import sys import threading from pathlib import Path _ROOT = Path(__file__).resolve().parent.parent SPEAKER = "wespeaker_resnet34_lm.onnx" SEGMENTER = "pyannote_segmentation_3.onnx" MODEL = "pyannote-segmentation-3.0+wespeaker-voxceleb-resnet34-LM" VERSION = 1 # of the saved profile; raise it when how one is made changes # What the speaker model was trained on; none of these are free parameters. # The window matters as much as the rest: a Povey window in place of Hamming # gives a vector only 0.88 like the right one. SAMPLE_RATE = 16000 WIN = 400 # 25 ms HOP = 160 # 10 ms NFFT = 512 BINS = 80 DIM = 256 # The segmentation model's own geometry (its metadata says the same). CHUNK = 160000 # the 10 s it hears at a time CHUNK_SHIFT = 80000 # chunks overlap by half, so every moment is heard twice FRAME = 270 # samples from one of its frames to the next (17 ms) FRAME_SPAN = 991 # samples one frame looks at # Its seven answers per frame: nobody, one of three speakers, or two at once. _POWERSET = ((0, 0, 0), (1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 1, 0), (1, 0, 1), (0, 1, 1)) SOLO_SECONDS = 1.0 # speech a speaker needs, with nobody over it, to be known by EMBED_SECONDS = 6.0 # and more than this of it adds time, not certainty # The video's voices. STRETCHES = 24 # sampled across the video's speech STRETCH_SECONDS = 20.0 LINK = 0.60 # cosine distance two groups of voices merge within MERGE = 0.55 # two groups this alike are one voice heard twice MAIN_VOICES = 3 # heard fewer times than this, there is no main voice to speak of KEEP_VOICES = 6 BESIDE = 1 # chunks a voice has to share with the main one to be surely another # A clip. Scores are cosine similarity to the main voice: 1 the same, 0 unrelated. MAIN_LIKE = 0.55 # at or above: the main speaker OTHER_LIKE = 0.30 # at or below: somebody else, whoever else is in earshot APART = 0.15 # in between: somebody else when this much nearer another known voice MARGIN = 0.10 # an unsure one goes to the other voice only this much nearer to it # Words. The segmentation is sure of a turn to within a word or so; a # sentence is surer of where it ends. TURN_SECONDS = 1.0 # a turn shorter than this has to be a whole remark ("Yeah.") SAID_SECONDS = 1.5 # and a clip where the others say less than this, all told, stays plain BREATH = 0.3 # a pause this long is as good as a full stop REACH = 2 # words a turn's edge may move to meet the end of a sentence _SENTENCE_END = re.compile(r"[.!?…。!?]['\")\]]*$") # as transcription/cloud.py ends one _sessions: dict = {} _load_lock = threading.Lock() _run_lock = threading.Lock() # one inference at a time: this never takes a second core _told = {"missing": False} # the missing models are said once a session _profiles: dict[str, tuple] = {} # video id -> (source fingerprint, profile) _profile_locks: dict[str, threading.Lock] = {} _locks_lock = threading.Lock() # ---- the models ------------------------------------------------------------------ def weights_path(name: str = SPEAKER) -> Path: """Where a model lives, frozen build or checkout.""" if getattr(sys, "frozen", False): return Path(sys._MEIPASS) / name # type: ignore[attr-defined] return _ROOT / "models" / name def available() -> bool: """True when voices can be told apart. Without the models, captions stay one colour.""" try: import numpy # noqa: F401 import onnxruntime # noqa: F401 except ImportError: return False return weights_path(SPEAKER).exists() and weights_path(SEGMENTER).exists() def load(name: str = SPEAKER): """A model's session. Cached: loading costs a tenth of a second, a video's clips all use the one.""" with _load_lock: if name not in _sessions: import onnxruntime as ort opts = ort.SessionOptions() opts.intra_op_num_threads = 1 opts.inter_op_num_threads = 1 # From bytes, not a path: the install folder can hold any # character a Windows user name can. _sessions[name] = ort.InferenceSession( weights_path(name).read_bytes(), sess_options=opts, providers=["CPUExecutionProvider"] ) return _sessions[name] _banks = None def _mel(hz): import numpy as np return 1127.0 * np.log(1.0 + hz / 700.0) def _filterbank(): """Kaldi's triangular mel filters, 20 Hz to Nyquist: (BINS, NFFT/2).""" global _banks if _banks is None: import numpy as np low, high = _mel(20.0), _mel(SAMPLE_RATE / 2.0) step = (high - low) / (BINS + 1) left = low + np.arange(BINS, dtype=np.float64)[:, None] * step mel = _mel((SAMPLE_RATE / NFFT) * np.arange(NFFT // 2, dtype=np.float64))[None, :] _banks = np.maximum(0.0, np.minimum((mel - left) / step, (left + 2 * step - mel) / step)) return _banks def fbank(wave, normalize: bool = True): """Kaldi log-mel filterbank features, (frames, 80), as the speaker model was trained on: 25 ms Hamming frames every 10 ms, each with its mean removed and pre-emphasised, then each mel bin's mean over time taken off. `wave` is mono float32 in [-1, 1] at 16 kHz, at least one frame long.""" import numpy as np x = np.asarray(wave, dtype=np.float64) * 32768.0 frames = 1 + (x.size - WIN) // HOP f = x[np.arange(WIN)[None, :] + HOP * np.arange(frames)[:, None]] f = f - f.mean(axis=1, keepdims=True) f = f - 0.97 * np.concatenate([f[:, :1], f[:, :-1]], axis=1) power = np.abs(np.fft.rfft(f * np.hamming(WIN), n=NFFT, axis=1)) ** 2 feats = np.log(np.maximum(power[:, : NFFT // 2] @ _filterbank().T, np.finfo(np.float32).eps)) if normalize: feats = feats - feats.mean(axis=0, keepdims=True) return feats.astype(np.float32) def embed(waves: list): """One unit-length voice vector per wave, (n, 256). A wave too short to hold a frame gets zeros, which is like no voice at all.""" import numpy as np session = load(SPEAKER) name = session.get_inputs()[0].name out = np.zeros((len(waves), DIM), dtype=np.float32) for i, wave in enumerate(waves): if len(wave) < WIN + HOP: continue feats = fbank(wave)[None, :, :] with _run_lock: vector = session.run(None, {name: feats})[0][0] norm = float(np.linalg.norm(vector)) if norm > 0: out[i] = vector / norm return out def chunk_starts(samples: int) -> list[int]: """The first sample of each chunk over a sound `samples` long: half a chunk apart, the last one ending with the sound.""" starts = list(range(0, max(1, samples - CHUNK + 1), CHUNK_SHIFT)) if starts[-1] + CHUNK < samples: starts.append(samples - CHUNK) return starts def segment(wave) -> list[tuple]: """Who is talking when: [(first sample, marks)], one per 10 s chunk, the chunks overlapping by half. `marks` is (frames, 3) of 0/1: for each 17 ms frame, which of the chunk's three speakers are talking. A speaker's number means nothing outside its own chunk.""" import numpy as np session = load(SEGMENTER) name = session.get_inputs()[0].name powerset = np.asarray(_POWERSET, dtype=np.int8) wave = np.asarray(wave, dtype=np.float32) chunks = [] for first in chunk_starts(wave.size): piece = wave[first:first + CHUNK] if piece.size < CHUNK: piece = np.pad(piece, (0, CHUNK - piece.size)) with _run_lock: scores = session.run(None, {name: piece[None, None, :]})[0][0] chunks.append((first, powerset[np.argmax(scores, axis=-1)])) return chunks _models = (embed, segment) # the functions below take stand-ins for these, for tests def _pcm(source: Path, start: float, seconds: float): """`seconds` of the video's sound from `start`, mono float32 at 16 kHz. Empty when it has none.""" import numpy as np from core.binaries import ffmpeg out = subprocess.run( [ffmpeg(), "-v", "error", "-ss", f"{max(0.0, start):.3f}", "-t", f"{seconds:.3f}", "-i", str(source), "-vn", "-ac", "1", "-ar", str(SAMPLE_RATE), "-f", "s16le", "-"], capture_output=True, timeout=300, ) if out.returncode != 0 or not out.stdout: return np.zeros(0, dtype=np.float32) raw = out.stdout[: len(out.stdout) // 2 * 2] return np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0 # ---- the speakers of a stretch --------------------------------------------------- def _samples(frame: int) -> int: """The sample a frame's mark starts to apply at (the middle of what the frame looked at, less half a step).""" return frame * FRAME + (FRAME_SPAN - FRAME) // 2 def local_voices(wave, chunks: list[tuple], embed) -> list[dict]: """Each chunk's speakers that said enough to be known by: [{"chunk", "speaker", "seconds", "vector"}]. The vector is taken from what a speaker said with nobody talking over them.""" import numpy as np found, sounds = [], [] for c, (first, marks) in enumerate(chunks): alone = marks * (marks.sum(axis=1, keepdims=True) == 1) for k in range(marks.shape[1]): on = np.flatnonzero(alone[:, k]) if on.size * FRAME < SOLO_SECONDS * SAMPLE_RATE: continue # runs of consecutive frames, the longest first, up to the cap breaks = np.flatnonzero(np.diff(on) > 1) runs = sorted(zip(np.r_[on[0], on[breaks + 1]], np.r_[on[breaks], on[-1]] + 1), key=lambda r: r[0] - r[1]) pieces, kept = [], 0 for a, b in runs: piece = wave[first + _samples(int(a)): first + _samples(int(b))] pieces.append(piece) kept += piece.size if kept >= EMBED_SECONDS * SAMPLE_RATE: break found.append({"chunk": c, "speaker": k, "seconds": round(on.size * FRAME / SAMPLE_RATE, 2)}) sounds.append(np.concatenate(pieces)) for voice, vector in zip(found, embed(sounds) if sounds else []): voice["vector"] = vector return found def _unit(vectors, weights=None): """The mean direction of some voice vectors, unit length.""" import numpy as np mean = np.average(np.asarray(vectors, dtype=np.float64), axis=0, weights=weights) norm = float(np.linalg.norm(mean)) return mean / norm if norm > 0 else mean def cluster(E, distance: float) -> list[int]: """A group number for each row of E: average-linkage clustering on cosine distance, merging until the two closest groups are further apart than `distance`. In numpy because scipy.cluster is not in the installed app.""" import numpy as np n = len(E) if n == 0: return [] E = np.asarray(E, dtype=np.float64) D = 1.0 - E @ E.T np.fill_diagonal(D, np.inf) size = np.ones(n) members: list[list[int]] = [[i] for i in range(n)] while True: i, j = divmod(int(np.argmin(D)), n) if not np.isfinite(D[i, j]) or D[i, j] > distance: break merged = (size[i] * D[i] + size[j] * D[j]) / (size[i] + size[j]) D[i, :] = merged D[:, i] = merged D[i, i] = np.inf D[j, :] = np.inf D[:, j] = np.inf size[i] += size[j] members[i] += members[j] members[j] = [] labels = [0] * n for group, rows in enumerate(m for m in members if m): for row in rows: labels[row] = group return labels # ---- the video's voices ---------------------------------------------------------- def voices_of(E, seconds: list[float], where: list | None = None) -> list[dict]: """The voices among some voice vectors, the main speaker's first: [{"share", "heard", "spread", "beside", "centroid"}]. `seconds` is how much speech each vector stands for and `where` the (stretch, chunk) it was heard in. The main speaker is the voice heard in the most stretches, and only then the one that talks most: a streamer is there from start to finish, while what they watch and who they call come and go, and a video being reacted to often out-talks the person reacting. `beside` counts the chunks a voice shares with the main one. Two voices heard side by side are two people; a voice never heard beside the main one may be the main speaker again, shouting or on another microphone.""" import numpy as np E = np.asarray(E, dtype=np.float64) where = list(where) if where is not None else [(i, 0) for i in range(len(E))] heard = [i for i in range(len(E)) if float(np.linalg.norm(E[i])) > 0] if not heard: return [] groups: dict[int, list[int]] = {} for row, label in zip(heard, cluster(E[heard], LINK)): groups.setdefault(label, []).append(row) found = list(groups.values()) def centre(group): return _unit(E[group], [sec - ARCHITECTURE.md (69035 bytes)
# Architecture: Clips Kitty Clips Kitty is a local-first AI clipping engine with a desktop front end. It ingests a long video, finds the moments worth posting using an LLM plus multimodal signal analysis, renders speaker-tracked captioned vertical clips, and hands them to a review and editing studio. By default nothing is sent to a cloud AI service and there is no paid inference anywhere in the pipeline; cloud AI is an opt-in on the user's own account ([Cloud AI](#cloud-ai-opt-in)). Opt-in modes change the framing and the scoring for game streams, matches, lives that were already vertical, and 16:9 output. This document describes the system as it is built today. It is the only architecture document: design notes that used to live separately have been folded in here. **Contents** 1. [System overview](#1-system-overview) 2. [Repository structure](#2-repository-structure) 3. [The processing pipeline](#3-the-processing-pipeline) 4. [Clip scoring](#4-clip-scoring) 5. [Face tracking and framing](#5-face-tracking-and-framing) 6. [Creator intelligence](#6-creator-intelligence) 7. [Multilingual publishing](#7-multilingual-publishing) 8. [Long-form output](#8-long-form-output) 9. [The video editor](#9-the-video-editor) 10. [Local API service](#10-local-api-service) 11. [Desktop application](#11-desktop-application) 12. [State and storage](#12-state-and-storage) 13. [Configuration surface](#13-configuration-surface) 14. [Windows packaging](#14-windows-packaging) 15. [Design rules](#15-design-rules) --- ## 1. System overview ``` ┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐ │ Desktop app │ │ Watched channels, │ │ Assistants │ │ Electron + React + Vite │ │ streamer-tool hand-offs │ │ Ask Clips Kitty · MCP │ └────────────┬─────────────┘ └────────────┬─────────────┘ └────────────┬─────────────┘ └─────────────────────────────┼─────────────────────────────┘ │ HTTP + WebSocket (127.0.0.1:8765) ┌────────────────────────▼────────────────────────┐ │ FastAPI service (server/) │ │ queue · progress events · publisher · webhooks │ └────────────────────────┬────────────────────────┘ │ ┌──────────────────────────────────────────▼───────────────────────────────────────────┐ │ │ │ PIPELINE (core/pipeline.py) │ │ │ │ SOURCE ──► DOWNLOAD ──► TRANSCRIBE ──► ANALYZE ───► RENDER ───────► METADATA │ │ YouTube yt-dlp Whisper, signals + framing by mode titles, │ │ Twitch local, or the LLM, + captions, descriptions, │ │ Kick cloud on the local or here or on a hashtags │ │ local file user's account cloud render PC │ │ │ │ signals: audio · visual · reaction · text · engagement │ │ side channels: creator knowledge · chat replay · heatmaps · game metadata · │ │ score box (OCR) · crowd, whistle and game sounds (PANNs) │ │ framing by mode: speaker tracking (standard) · webcam + game (Gaming / Reaction) │ │ · the whole 9:16 frame (Vertical Live) · the ball (Sports) │ │ · 16:9 (Longform) · per shot (Podcast) │ │ │ └──────────────────────────────────────────┬───────────────────────────────────────────┘ │ ┌─────────────────────┬──────────┴──────────┬─────────────────────┐ ▼ ▼ ▼ ▼ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ SQLite state │ │ Rendered clips │ │ Creator KB │ │ Publishing │ │ core/state.py │ │ data/clips/ │ │ knowledge + │ │ YouTube · │ │ │ │ │ │ events │ │ WoopSocial · │ │ │ │ │ │ │ │ Upload-Post │ └───────────────────┘ └───────────────────┘ └───────────────────┘ └───────────────────┘ ``` By default everything in that diagram runs on the local machine. The only network calls a normal run makes are **yt-dlp fetching the source video** and, for Twitch VODs, an optional chat-replay request. Whisper, the LLM, YOLOv8, and FFmpeg are all local. Three things add network paths, and each is off until the user turns it on: **cloud AI** (the transcript, or the audio for cloud transcription, goes to the provider the user signed in to), **publishing** (finished clips go to the platforms, directly or through a publishing service) and **remote rendering** (one clip's stretch of video goes to the user's own render PC, [10.3](#103-remote-rendering)). Two plain file fetches run on their own, carry nothing about the user, and can each be switched off: the desktop app's update check and Clips Kitty's online list of pipelines (`plugins/registry.py` `ONLINE_URL`, once a day when the Marketplace opens or the app starts with a pipeline installed; `docs/platform/DECISIONS.md` D29). ### End to end The same flow as the README, from a watched channel to a published clip: ```mermaid flowchart TD W["Watched channel<br/>YouTube's RSS feed spots a new video"] --> I I["Video input<br/>YouTube · Twitch VOD · Kick VOD · local file"] --> T T["Transcription<br/>faster-whisper, word-level timestamps<br/>local, or OpenRouter, signed in with OAuth"] --> A A["AI video analysis<br/>audio · visual · reaction signals in 1-second bins"] --> S S["Clip detection and scoring<br/>an LLM scores the transcript with those signals<br/>local (Ollama) by default, or OpenRouter, signed in with OAuth"] --> E E["Video editing<br/>9:16 crop on whoever is speaking (YOLOv8 + TalkNet),<br/>on the ball for a match, webcam + game for Gaming / Reaction"] --> C C["Captions<br/>word-synced, burned in, fully editable"] --> M M["Multilingual<br/>translate, subtitle or dub into 19 languages"] --> X X["Export<br/>organized folders, clean filenames, metadata"] --> P P["Publishing<br/>YouTube, TikTok, Instagram: now, on a schedule,<br/>or hands-off for a watched channel"] ``` ### The AI stack | Job | Tool | Where it runs | |---|---|---| | Download | yt-dlp | local | | Transcription | faster-whisper, word-level timestamps | local, CUDA or CPU; or a cloud Whisper on the user's account | | Clip scoring, titles, translation, edit chat, assistant | any Ollama model (Gemma by default), or a cloud model | local via Ollama; or OpenRouter or a direct provider on the user's account | | Person/pose detection | YOLOv8 (`yolov8n-pose.pt`) + OpenCV | local, CUDA or CPU | | Text on screen (score box, game banners) | RapidOCR | local | | Sound events (crowd, whistle, game sounds) | PANNs (an AudioSet tagger) | local | | Who is talking (the second speaker's caption colour) | pyannote segmentation + WeSpeaker speaker model, on onnxruntime | local, CPU, one core | | Ball detection (Sports) | YOLOv8n, "sports ball" | local | | Rendering | FFmpeg with hardware encoding (NVENC / AMF / QSV) | local | | Dubbing voices | local TTS | local | ### Cloud AI (opt-in) The LLM and transcription can each run in the cloud instead, on the user's own account, chosen in Settings → AI ([docs/AI-BACKENDS.md](docs/AI-BACKENDS.md)). Local stays the default and the first choice. - `llm/providers/catalog.py` lists the providers: OpenRouter as the recommended cloud path, direct APIs (OpenAI, Anthropic, Gemini, xAI, Meta and others) as advanced. An OpenAI-compatible provider is one entry; a new wire format is one adapter in `llm/providers/adapters/`. - Every request goes through `llm/providers/http.py`: the key in a header, plain-English errors, bounded retries. A failing cloud model fails the job; it never falls back to another provider or to local AI behind the user's back. - OpenRouter can be signed in to instead of pasting a key: OAuth with PKCE (`llm/providers/oauth.py`), with the engine on localhost receiving the callback. The key it returns is the user's own and lives in the app's credential store (`llm/providers/keys.py`). - OpenRouter requests carry its app-attribution headers (`llm/providers/openrouter.py`), so the usage counts toward OpenRouter's public app rankings. - Cloud transcription (`transcription/cloud.py`) writes the same transcript cache as local Whisper, so every later stage is unchanged. --- ## 2. Repository structure ``` clips-studio/ ├── main.py # CLI entry: process, serve, models, status, channels ├── config/ │ ├── settings.yaml # quick setup at the top, advanced below │ ├── prompts/ # every LLM prompt as an editable text file │ ├── gaming.yaml # kinds of game and what a highlight is in each │ └── sports.yaml # each sport's moments, windows and commentary words ├── core/ │ ├── pipeline.py # stage orchestration for one video │ ├── state.py # SQLite schema + all queries │ ├── models.py # dataclasses shared between stages │ ├── progress.py # stage events → WebSocket │ ├── cancel.py # cooperative cancellation flags │ ├── modes.py # Vertical Live, Gaming / Reaction and Sports switches │ ├── preflight.py # can this install make a clip? (setup checks) │ ├── binaries.py # bundled FFmpeg, Ollama and model weights │ ├── queue.py # queue manager: order, pause, retry, estimate │ ├── prefetch.py # download-ahead for queued jobs │ ├── housekeeping.py # disk reclamation │ └── scheduler.py # (legacy CLI) `main.py run` poll loop; the app uses server/automation.py ├── sources/ # ── one file per platform ── │ ├── dispatch.py # URL → the right source module │ ├── youtube.py # yt-dlp download, H.264 selection, heatmap │ ├── twitch.py # VOD download + chat replay │ ├── kick.py # VOD download │ ├── channel_feed.py # what a watched channel posted (RSS, yt-dlp, Kick API) │ ├── vod_finder.py # the VOD a finished livestream left behind │ └── ytdlp_common.py # chunked, resumable download shared by all ├── transcription/ │ ├── transcriber.py # faster-whisper, word timestamps, GPU/CPU fallback │ └── cloud.py # transcription on the user's own provider account ├── llm/ # ── swappable model layer ── │ ├── base.py # LLMBackend interface │ ├── ollama_backend.py # anything Ollama serves │ ├── registry.py # config string → backend instance │ ├── manager.py # install/switch/recommend models │ ├── spec.py # which provider and model a backend spec names │ └── providers/ # cloud AI on the user's own account: catalog, adapters, │ # http, keys, OAuth sign-in, OpenRouter attribution ├── analysis/ # ── the clip engine ── │ ├── audio_features.py # loudness, spikes, burst density, laughter proxy │ ├── visual_features.py # scene cuts, motion, flashes, face metrics │ ├── hype.py # Twitch chat replay + YouTube most-replayed │ ├── gaming.py # scores a gaming stream as a gaming stream │ ├── game_audio.py # what the game sounds like │ ├── game_text.py # what the game writes on screen │ ├── game_vision.py # the AI looks at a gaming stream's best candidates │ ├── chat_moments.py # what chat's reactions say happened │ ├── panns.py # PANNs sound tagger: game sounds, crowd, whistle │ ├── voice_turns.py # whose voice it is: the turns of anyone but the main speaker │ ├── intent.py # clip direction: what the clips should be about │ ├── fusion.py # candidate generation, weighted scoring, rerank │ ├── highlights.py # LLM transcript scoring + duplicate prevention │ ├── metadata.py # titles, descriptions, hashtags │ ├── clip_edit.py # AI edit chat → validated edit operations │ └── tighten.py # silence and filler-word removal ├── creator/ # ── creator intelligence ── │ ├── identity.py # channel → creator profile resolution, merging │ ├── extractor.py # transcript → structured facts and events │ ├── retrieval.py # stored knowledge → scoring context │ ├── learning.py # preference learning from user actions │ └── models.py # knowledge types, thresholds ├── video/ # ── rendering ── │ ├── tracker.py # YOLOv8 pose tracking, speaker-aware │ ├── asd.py # TalkNet active-speaker detection │ ├── framing.py # crop-path smoothing and shot logic │ ├── cuts.py # frame-exact cuts for shot-aware framing (§5) │ ├── podcast.py # multi-cam shot detection, per-shot framing │ ├── cropper.py # 9:16 render │ ├── captions.py # word-synced ASS captions │ ├── cutter.py # accurate clip extraction │ ├── filters.py # colour presets and adjustments │ └── encoding.py # encoder detection and selection ├── video_editor/ # ── manual editing applied to a clip ── │ ├── timeline.py # non-destructive edit model │ ├── cuts.py # trims and internal cuts │ ├── audio.py # volume, fades, mutes, music bed │ ├── captions.py # caption edits │ ├── overlay.py # hook text │ ├── watermark.py # branding profiles │ └── export.py # final render ├── gaming/ # ── Gaming / Reaction (opt-in) ── │ ├── detect.py # who the streamer is, where the webcam sits │ ├── panels.py # chat boxes and timers the game crop keeps out │ ├── layout.py # where each part of a layout comes from and goes │ ├── framing.py # the eleven layouts (data in gaming/layouts.json) │ ├── compose.py # the FFmpeg render - campaigns/__init__.py (1219 bytes)
"""Clipping campaigns: submitting a published clip's public link to a campaign that pays for views, in Discord or on the campaign's own website. Clips Kitty never posts as the user. A campaign is submitted one of three ways: by hand in Discord with the command and values prepared (discord_assisted, the default), by a webhook a server admin authorised (discord_webhook), or by hand on the campaign's page (website). - model.py statuses, their labels and groups, platforms, the campaign rules - links.py Discord channel, message and webhook links - urls.py a submission's public link: where it comes from, and checking one - eligibility.py what a campaign asks of a post, checked, and the verdict - formatter.py the message, the command and the webhook payload - statemachine.py which status changes are allowed, applied compare-and-set - queue.py evaluating a submission once its post's link is known Standard library, `core.*`, `server.posts`, the publish and provider modules and the plugin store's id pattern only, so CI's light job tests all of it. Nothing here is ever passed to a plugin, and a pipeline can't import it (_clipskitty_script_host.py ENGINE_PACKAGES). """
- campaigns/discord_http.py (3962 bytes)
"""The one way Clips Kitty talks to Discord: a thin `requests` wrapper tests replace, the API's address, and what a failed request means. Tests never reach the real Discord. They hand the OAuth client and the webhook sender an object with the same `.request(method, url, **kw)` (a scripted fake), or point the whole engine at a fake server on this PC through CLIPS_KITTY_DISCORD_API / CLIPS_KITTY_DISCORD_AUTHORIZE. Those two are honoured only for a plain http://127.0.0.1:<port> address, so a stray variable can never send sign-ins or webhook tokens anywhere else. Nothing here logs a request: a webhook's URL is its token, and an OAuth request carries a code or a token in its body. """ import os import re import requests API_BASE = "https://discord.com/api/v10" AUTHORIZE_URL = "https://discord.com/oauth2/authorize" API_ENV = "CLIPS_KITTY_DISCORD_API" AUTHORIZE_ENV = "CLIPS_KITTY_DISCORD_AUTHORIZE" #: (connect, read) seconds for every Discord request. TIMEOUT = (5, 15) USER_AGENT = "ClipsKitty (https://github.com/ColinGPT9/clips-studio)" _LOOPBACK = re.compile(r"^http://127\.0\.0\.1:\d{1,5}(?:/[^\s?#]*)?$") class DiscordError(Exception): """A Discord call that didn't work, with what to show and a code for what to do about it: network, expired (sign in again), denied, bad_response, rate_limited, not_configured, not_connected, gone (a webhook that no longer exists).""" def __init__(self, code: str, message: str): super().__init__(message) self.code = code self.message = message def _override(name: str, default: str) -> str: value = str(os.environ.get(name) or "").strip().rstrip("/") return value if value and _LOOPBACK.match(value) else default def api_base() -> str: """Discord's API, or a fake one on this PC (tests, the scratch E2E).""" return _override(API_ENV, API_BASE) def authorize_base() -> str: """Discord's sign-in page, or a fake one on this PC.""" return _override(AUTHORIZE_ENV, AUTHORIZE_URL) class Http: """requests with Discord's timeouts and a User-Agent. Tests inject any object with the same request().""" def request(self, method: str, url: str, **kw): headers = {"User-Agent": USER_AGENT, **(kw.pop("headers", None) or {})} kw.setdefault("timeout", TIMEOUT) return requests.request(method, url, headers=headers, **kw) def body(response): """A response's JSON, or {} when it has none Clips Kitty can read.""" try: return response.json() except Exception: return {} def _causes(exc: BaseException) -> list[BaseException]: """The exception and everything it wraps: requests wraps urllib3, which wraps the socket error, in arguments as well as in causes.""" seen: list[BaseException] = [] stack: list = [exc] while stack: current = stack.pop() if not isinstance(current, BaseException) or any(current is s for s in seen): continue seen.append(current) stack.extend([getattr(current, "reason", None), current.__cause__, current.__context__, *getattr(current, "args", ())]) return seen def never_sent(exc: BaseException) -> bool: """Whether a failed request certainly never reached Discord: a connect timeout, a refused connection or a name that didn't resolve (DESIGN §4.3). Anything else (a read timeout, a connection dropped after it was made, a TLS error) may have been received, and a webhook has no way to tell, so it counts as maybe sent.""" import socket from urllib3 import exceptions as u3 # NameResolutionError is a NewConnectionError: the connection was never made. before = (requests.ConnectTimeout, u3.NewConnectionError, u3.ConnectTimeoutError, socket.gaierror, ConnectionRefusedError) if isinstance(exc, requests.ReadTimeout): return False return any(isinstance(e, before) for e in _causes(exc)) - campaigns/discord_oauth.py (9777 bytes)
"""Signing in with Discord: OAuth2 with PKCE for a Public Client. The app is a Public Client, so there is no client secret anywhere: the client id is public, and each sign-in proves itself with a fresh PKCE verifier (S256) that never leaves this PC until the exchange. Signing in asks for `identify` and `guilds` only, which lets Clips Kitty show who is connected and list their servers. It can't post as the user, read their messages or run another bot's commands; nothing in Discord's OAuth can. `webhook.incoming` is the one other scope: a server admin picks a channel on Discord's own page and Discord hands back a webhook for it. That flow is separate (purpose "webhook"), and its access token is revoked at once and never kept; only the webhook is. Every call takes the transport (`http`, campaigns.discord_http.Http or a test's fake) and the API address comes from discord_http.api_base(). """ import secrets import threading import time from dataclasses import dataclass, field from urllib.parse import quote, urlencode from campaigns import discord_http from campaigns.discord_http import DiscordError from core.scrub import redact_secrets from llm.providers.oauth import pkce_pair #: Clips Kitty's Discord application. Public by design (it is in every #: authorize link); Settings → Discord → Advanced can override it for testing. DISCORD_CLIENT_ID = "1563637373474971708" SIGN_IN_SCOPES = ("identify", "guilds") WEBHOOK_SCOPE = "webhook.incoming" CALLBACK_PATH = "/discord/callback" #: The port the engine runs on, and the only redirect registered with Discord. DEFAULT_PORT = 8765 #: How long a sign-in in the browser may take before its state is refused. SIGN_IN_SECONDS = 15 * 60 #: Server permissions that let someone add a webhook (Manage Webhooks, #: Administrator). MANAGE_WEBHOOKS = 1 << 29 ADMINISTRATOR = 1 << 3 def redirect_uri(port: int) -> str: """Where Discord sends the browser back. Discord accepts only the one registered for the app (port 8765); a fake server accepts any.""" return f"http://127.0.0.1:{int(port)}{CALLBACK_PATH}" def authorize_url(client_id: str, redirect: str, state: str, challenge: str, scopes) -> str: """The page the user's own browser opens to approve Clips Kitty.""" return discord_http.authorize_base() + "?" + urlencode({ "response_type": "code", "client_id": client_id, "scope": " ".join(scopes), "redirect_uri": redirect, "state": state, "code_challenge": challenge, "code_challenge_method": "S256", }, quote_via=quote) def _network() -> DiscordError: return DiscordError("network", "Couldn't reach Discord. Check your internet connection and try again.") def _token_call(http, path: str, form: dict): """POST a form to Discord's token endpoints. The form carries a code or a token, so no exception text (which can name the request) goes on.""" try: return http.request( "POST", discord_http.api_base() + path, data=form, headers={"Content-Type": "application/x-www-form-urlencoded"}, ) except Exception: raise _network() from None def _tokens(http, form: dict) -> dict: response = _token_call(http, "/oauth2/token", form) got = discord_http.body(response) got = got if isinstance(got, dict) else {} status = int(getattr(response, "status_code", 0) or 0) if status == 429: raise DiscordError("rate_limited", "Discord is busy. Wait a minute and try again.") if status >= 300: error = str(got.get("error") or "") if error == "invalid_grant": raise DiscordError("expired", "This Discord sign-in has expired or was already used. " "Connect Discord again.") if error in ("invalid_client", "unauthorized_client"): raise DiscordError("denied", "Discord didn't accept Clips Kitty's application ID. " "Check it under Settings → Discord → Advanced.") detail = redact_secrets(str(got.get("error_description") or error or ""))[:200] raise DiscordError("bad_response", f"Discord refused the sign-in ({status}{': ' + detail if detail else ''}).") if not str(got.get("access_token") or ""): raise DiscordError("bad_response", "Discord signed you in but sent no token back. Try again.") return got def exchange(http, client_id: str, code: str, verifier: str, redirect: str) -> dict: """The tokens for a one-time code. With `webhook.incoming` the answer may also hold the webhook Discord made ("webhook").""" return _tokens(http, { "grant_type": "authorization_code", "code": code, "redirect_uri": redirect, "client_id": client_id, "code_verifier": verifier, }) def refresh(http, client_id: str, refresh_token: str) -> dict: """New tokens for a refresh token. A Public Client sends its id, no secret.""" return _tokens(http, { "grant_type": "refresh_token", "refresh_token": refresh_token, "client_id": client_id, }) def revoke(http, client_id: str, token: str, hint: str = "refresh_token") -> bool: """Ask Discord to forget a token. False when it refused or couldn't be reached: the caller wipes its own copy either way.""" if not token: return False try: response = _token_call(http, "/oauth2/token/revoke", {"token": token, "token_type_hint": hint, "client_id": client_id}) except DiscordError: return False return 200 <= int(getattr(response, "status_code", 0) or 0) < 300 def _get(http, path: str, access_token: str): try: response = http.request("GET", discord_http.api_base() + path, headers={"Authorization": f"Bearer {access_token}"}) except Exception: raise _network() from None status = int(getattr(response, "status_code", 0) or 0) if status == 401: raise DiscordError("expired", "Your Discord sign-in has expired. Connect Discord again.") if status == 403: raise DiscordError("denied", "Discord didn't allow that. Connect Discord again.") if status == 429: raise DiscordError("rate_limited", "Discord is busy. Wait a minute and try again.") if status >= 300: raise DiscordError("bad_response", f"Discord answered {status}. Try again later.") return discord_http.body(response) def me(http, access_token: str) -> dict: """The signed-in user: {id, username, global_name}.""" got = _get(http, "/users/@me", access_token) if not isinstance(got, dict) or not str(got.get("id") or ""): raise DiscordError("bad_response", "Discord didn't say who signed in. Try again.") return {"id": str(got["id"]), "username": str(got.get("username") or ""), "global_name": str(got.get("global_name") or "")} def _can_manage_webhooks(guild: dict) -> bool: if guild.get("owner"): return True try: permissions = int(str(guild.get("permissions") or "0")) except ValueError: return False return bool(permissions & (MANAGE_WEBHOOKS | ADMINISTRATOR)) def guilds(http, access_token: str) -> list[dict]: """The user's servers: [{id, name, owner, can_manage_webhooks}].""" got = _get(http, "/users/@me/guilds", access_token) if not isinstance(got, list): raise DiscordError("bad_response", "Discord didn't send the list of your servers. Try again.") out = [] for guild in got: if not isinstance(guild, dict) or not str(guild.get("id") or ""): continue out.append({"id": str(guild["id"]), "name": str(guild.get("name") or ""), "owner": bool(guild.get("owner")), "can_manage_webhooks": _can_manage_webhooks(guild)}) return out # ---- sign-ins in progress --------------------------------------------------------- @dataclass class Pending: purpose: str # connect | webhook verifier: str redirect: str started: float extra: dict = field(default_factory=dict) class PendingSignIns: """Sign-ins waiting for the browser to come back, by their state. In memory: a sign-in outlives nothing but its browser tab. Each state is good once, for SIGN_IN_SECONDS.""" def __init__(self, ttl: float = SIGN_IN_SECONDS): self.ttl = ttl self._waiting: dict[str, Pending] = {} self._lock = threading.Lock() # routes run in a thread pool def _prune(self, now: float) -> None: """Drop sign-ins too old to finish. The caller holds the lock.""" for stale in [s for s, p in self._waiting.items() if now - p.started > self.ttl]: self._waiting.pop(stale, None) def start(self, purpose: str, redirect: str, *, now: float | None = None, extra: dict | None = None) -> tuple[str, str]: """A new sign-in: (state, S256 challenge) for the authorize link.""" now = time.time() if now is None else float(now) verifier, challenge = pkce_pair() state = secrets.token_urlsafe(18) with self._lock: self._prune(now) self._waiting[state] = Pending(purpose, verifier, redirect, now, dict(extra or {})) return state, challenge def take(self, state: str, *, now: float | None = None) -> Pending | None: """The sign-in for `state`, once; None when unknown, used or too old.""" now = time.time() if now is None else float(now) with self._lock: found = self._waiting.pop(str(state or ""), None) self._prune(now) if found is None or now - found.started > self.ttl: return None return found - campaigns/discord_webhook.py (7901 bytes)
"""Posting a message through a Discord webhook, and checking a webhook. A webhook is the one way Clips Kitty posts to Discord by itself: a server admin made it (or approved it through `webhook.incoming`) for one channel, and its URL carries the token that lets anyone holding it post there. So the token goes to Discord in the request path only, and never into a result, a message or a log line; a requests exception names the URL it failed on, so none is ever passed on either. Only a message is posted, with mentions switched off. A webhook can't run another bot's slash command. """ import re from dataclasses import dataclass from urllib.parse import quote from campaigns import discord_http from campaigns.discord_http import DiscordError from core.scrub import redact_secrets #: What Test posts (Settings → Discord → Destinations), after a confirm. TEST_MESSAGE = "Clips Kitty test message. You can delete it." #: A forum channel takes a message only as the first post of a new thread. TEST_THREAD_NAME = "Clips Kitty test" #: Discord's error codes: Unknown Webhook, Invalid Webhook Token, Missing #: Permissions, a forum channel's message needing a thread. UNKNOWN_WEBHOOK, INVALID_TOKEN, MISSING_PERMISSIONS, FORUM_NEEDS_THREAD = 10015, 50027, 50013, 220001 @dataclass class SendResult: # ok | rate_limited | gone | denied | forum_needs_thread | bad_request | server_error | unreachable | ambiguous kind: str message_id: str = "" channel_id: str = "" retry_after: float = 0.0 is_global: bool = False status: int = 0 code: int = 0 message: str = "" # already scrubbed _ID_SHAPE = re.compile(r"\d{17,20}") _TOKEN_SHAPE = re.compile(r"[\w-]{1,200}") NOT_A_WEBHOOK = "The stored webhook link isn't a Discord webhook's. Paste its link again." def _url(webhook_id: str, token: str) -> str: """The webhook's address. Its id and token must have a webhook's shape (a slash or a dot would walk off /webhooks/), and are quoted anyway. Raises ValueError otherwise.""" webhook_id, token = str(webhook_id or ""), str(token or "") if not _ID_SHAPE.fullmatch(webhook_id) or not _TOKEN_SHAPE.fullmatch(token): raise ValueError(NOT_A_WEBHOOK) return f"{discord_http.api_base()}/webhooks/{quote(webhook_id, safe='')}/{quote(token, safe='')}" def _clean(text: str, token: str) -> str: """Discord's words, safe to show and store: no token, nothing key-shaped.""" text = str(text or "") if token: text = text.replace(token, "[redacted]") return redact_secrets(text)[:300] def _code(got) -> int: try: return int(got.get("code") or 0) if isinstance(got, dict) else 0 except (TypeError, ValueError): return 0 def _retry_after(response, got) -> float: for value in ((got or {}).get("retry_after") if isinstance(got, dict) else None, (getattr(response, "headers", None) or {}).get("Retry-After")): try: seconds = float(value) except (TypeError, ValueError): continue if seconds >= 0: return seconds return 0.0 def _is_global(response, got) -> bool: headers = getattr(response, "headers", None) or {} return bool(isinstance(got, dict) and got.get("global")) \ or str(headers.get("X-RateLimit-Global") or "").lower() == "true" \ or str(headers.get("X-RateLimit-Scope") or "").lower() == "global" def classify(response, token: str = "") -> SendResult: """What Discord's answer to a webhook request means (DESIGN §5.5).""" status = int(getattr(response, "status_code", 0) or 0) got = discord_http.body(response) code = _code(got) said = _clean(got.get("message") if isinstance(got, dict) else "", token) if 200 <= status < 300: message = got if isinstance(got, dict) else {} return SendResult("ok", message_id=str(message.get("id") or ""), channel_id=str(message.get("channel_id") or ""), status=status) if status == 429: return SendResult("rate_limited", retry_after=_retry_after(response, got), is_global=_is_global(response, got), status=status, code=code, message="Discord asked Clips Kitty to slow down") detail = f": {said}" if said else "" if status == 404 or code in (UNKNOWN_WEBHOOK, INVALID_TOKEN): return SendResult("gone", status=status, code=code, message=f"The webhook doesn't exist any more{detail}") if status in (401, 403) or code == MISSING_PERMISSIONS: return SendResult("denied", status=status, code=code, message=f"The webhook isn't allowed to post there{detail}") if status == 400 and code == FORUM_NEEDS_THREAD: return SendResult("forum_needs_thread", status=status, code=code, message="This channel is a forum: set Forum on the destination") if 400 <= status < 500: return SendResult("bad_request", status=status, code=code, message=f"Discord refused the message ({status}){detail}") return SendResult("server_error", status=status, code=code, message=f"Discord had a problem ({status}). Clips Kitty will try again.") def failed(exc: BaseException) -> SendResult: """A request that raised: never sent, or maybe sent. Nothing of the exception's text, which names the URL and so the token.""" if discord_http.never_sent(exc): return SendResult("unreachable", message="Couldn't reach Discord") return SendResult("ambiguous", message="Discord didn't answer in time: check the channel, " "it may have been posted") def send(http, webhook_id: str, token: str, *, content: str, thread_name: str = "", thread_id: str = "") -> SendResult: """Post `content` through the webhook and wait for Discord's answer.""" params = {"wait": "true"} if thread_id: params["thread_id"] = str(thread_id) payload: dict = {"content": content, "allowed_mentions": {"parse": []}} if thread_name and not thread_id: payload["thread_name"] = thread_name try: url = _url(webhook_id, token) except ValueError: return SendResult("gone", message=NOT_A_WEBHOOK) try: response = http.request("POST", url, params=params, json=payload) except Exception as e: return failed(e) return classify(response, token) def inspect(http, webhook_id: str, token: str) -> dict: """What a webhook is: {id, guild_id, channel_id, name}. Raises DiscordError ("gone" for a webhook that doesn't exist or whose token is wrong) with a message safe to show.""" try: url = _url(webhook_id, token) except ValueError: raise DiscordError("gone", NOT_A_WEBHOOK) from None try: response = http.request("GET", url) except Exception: raise DiscordError("network", "Couldn't reach Discord to check the webhook. Try again.") from None result = classify(response, token) if result.kind == "ok": got = discord_http.body(response) got = got if isinstance(got, dict) else {} return {"id": str(got.get("id") or webhook_id), "guild_id": str(got.get("guild_id") or ""), "channel_id": str(got.get("channel_id") or ""), "name": str(got.get("name") or "")} if result.kind == "gone": raise DiscordError("gone", "Discord doesn't know that webhook: it was deleted or its link " "was reset. Copy its link again in Discord.") if result.kind == "rate_limited": raise DiscordError("rate_limited", "Discord is busy. Wait a minute and try again.") if result.kind == "denied": raise DiscordError("denied", result.message) raise DiscordError("bad_response", result.message) - campaigns/eligibility.py (13050 bytes)
"""What a campaign asks of a post, checked, and what to do about the answer. precheck runs before the post exists (the editor's picker: platform, deadline against the scheduled time, duration, and the caption being typed). check runs once the post has a link and adds what only then can be known: that it is public, which account posted it, where the clip came from, and whether the way of submitting is usable. verdict turns the checks into the next step. Each check passes, fails, or is unknown (Clips Kitty can't see). An unknown holds up only an automatic send: doing a step by hand is the review. """ import re from campaigns import urls from campaigns.model import ( FAIL, PASS, UNKNOWN, Check, cell, json_field, parse_utc, platform_label, ) from server import posts _HASHTAG = re.compile(r"#([\w]+)") _MENTION = re.compile(r"@([\w.]+)") # ---- where a clip came from ------------------------------------------------------------ def _mode_of_job(payload: dict) -> str: if payload.get("sport"): return "sports" if payload.get("longform"): return "longform" if payload.get("vertical_live"): return "vertical_live" if payload.get("podcast"): return "podcast" if payload.get("gaming") or payload.get("gaming_scoring"): return "gaming" if payload.get("pipeline"): return "pipeline" return "regular" def _mode_of_render(opts: dict) -> str: if opts.get("profile"): return "longform" if opts.get("sport_moment"): return "sports" if opts.get("gaming"): return "gaming" return "unknown" def _pipeline_id(payload: dict) -> str: choice = payload.get("pipeline") if isinstance(choice, dict): return str(choice.get("id") or "") return str(choice or "") def clip_source(db, *, video_id, start_s, end_s, clip_id=0, job_payload=None) -> dict: """{"plugin", "mode", "creator_id"} of a clip, for a campaign's routing. plugin: the id a pipeline stamped on the clip (clips.scores "plugin"), else the job's pipeline id. mode: from the newest processing job of the video, else `job_payload` (automation's, for a job row already cleared), else from the clip's render options; "unknown" when nothing says. creator_id: the video's creator, or None. """ clip = db.get_clip(int(clip_id)) if clip_id else None if clip is None and video_id: clip = db.conn.execute( "SELECT * FROM clips WHERE video_id = ? AND start_s = ? AND end_s = ?", (video_id, round(float(start_s), 2), round(float(end_s), 2)), ).fetchone() video = video_id or (clip["video_id"] if clip is not None else "") scores = json_field(cell(clip, "scores", ""), {}) if clip is not None else {} plugin = str(scores.get("plugin") or "") payload = None if video: job = db.conn.execute( "SELECT payload FROM jobs WHERE type = 'process' AND video_id = ? ORDER BY id DESC LIMIT 1", (video,), ).fetchone() if job is not None: payload = json_field(job["payload"], None) payload = payload if isinstance(payload, dict) else None if payload is None and isinstance(job_payload, dict): payload = job_payload if payload is not None: mode = _mode_of_job(payload) plugin = plugin or _pipeline_id(payload) else: opts = json_field(cell(clip, "render_opts", ""), {}) if clip is not None else {} mode = _mode_of_render(opts) creator_id = None if video: row = db.conn.execute("SELECT creator_id FROM videos WHERE video_id = ?", (video,)).fetchone() if row is not None and row["creator_id"] is not None: creator_id = int(row["creator_id"]) return {"plugin": plugin, "mode": mode, "creator_id": creator_id} # ---- the checks ------------------------------------------------------------------------------ def _caption_checks(campaign: dict, caption: str) -> list[Check]: hashtags = json_field(campaign.get("required_hashtags"), []) mentions = json_field(campaign.get("required_mentions"), []) phrase = str(campaign.get("required_text") or "").strip() asked = [("hashtags", hashtags), ("mentions", mentions), ("text", phrase)] if not any(rule for _, rule in asked): return [] text = str(caption or "") if not text.strip(): return [Check(name, UNKNOWN, "Clips Kitty doesn't know what the caption said") for name, rule in asked if rule] out = [] if hashtags: have = {t.lower() for t in _HASHTAG.findall(text)} missing = [t for t in hashtags if str(t).lower() not in have] out.append(Check("hashtags", FAIL, "Missing " + ", ".join(f"#{t}" for t in missing)) if missing else Check("hashtags", PASS)) if mentions: have = {m.rstrip(".").lower() for m in _MENTION.findall(text)} missing = [m for m in mentions if str(m).lower() not in have] out.append(Check("mentions", FAIL, "Missing " + ", ".join(f"@{m}" for m in missing)) if missing else Check("mentions", PASS)) if phrase: words = r"\s+".join(re.escape(w) for w in phrase.split()) found = re.search(rf"(?<!\w){words}(?!\w)", text, re.IGNORECASE) out.append(Check("text", PASS) if found else Check("text", FAIL, f"Missing “{phrase}”")) return out def _seconds(value: float) -> str: return f"{value:g} s" def precheck(campaign, *, platform: str, duration_s: float, caption: str = "", scheduled_for: str = "", now) -> list[Check]: """The checks that can run before the post exists: platform, deadline (against the scheduled time when one is given), duration, and hashtags, mentions and the phrase on a draft caption.""" campaign = dict(campaign) current = posts.as_utc(now) out = [] platforms = json_field(campaign.get("platforms"), []) if not platforms or platform in platforms: out.append(Check("platform", PASS)) else: out.append(Check("platform", FAIL, f"This campaign doesn't take {platform_label(platform)} posts")) deadline_text = str(campaign.get("deadline") or "") deadline = parse_utc(deadline_text) scheduled = parse_utc(scheduled_for) if scheduled_for else None if not deadline_text: out.append(Check("deadline", PASS)) elif deadline is None: out.append(Check("deadline", UNKNOWN, "Clips Kitty can't read the campaign's deadline")) elif current >= deadline: out.append(Check("deadline", FAIL, "The campaign's deadline has passed")) elif scheduled is not None and scheduled > deadline: out.append(Check("deadline", FAIL, "Its deadline is before this post's time")) else: out.append(Check("deadline", PASS)) low = float(campaign.get("min_seconds") or 0) high = float(campaign.get("max_seconds") or 0) length = float(duration_s or 0) if length < low or (high and length > high): wanted = f"{_seconds(low)} to {_seconds(high)}" if high else f"at least {_seconds(low)}" out.append(Check("duration", FAIL, f"The clip is {_seconds(round(length, 1))}; this campaign takes {wanted}")) else: out.append(Check("duration", PASS)) out += _caption_checks(campaign, caption) return out def _bare_account(name: str) -> str: return re.sub(r"\s+", "", str(name or "")).lstrip("@").lower() def _public(sub: dict, res) -> Check: url = res.url or str(sub.get("post_url") or "") _, problem = urls.validate(sub["platform"], url) if problem: return Check("public", FAIL, problem) if res.source == "manual" or sub.get("url_source") == "manual": return Check("public", PASS, "You entered this link") seen = str(res.visibility or "").lower() if seen in ("public", "default"): return Check("public", PASS) if not seen: return Check("public", UNKNOWN, "Clips Kitty didn't record who can see this post") shown = {"friends": "only for friends"}.get(seen, seen) return Check("public", FAIL, f"This post is {shown}, not public") def _account(campaign: dict, sub: dict, res) -> list[Check]: wanted = json_field(campaign.get("required_accounts"), {}).get(sub["platform"]) if not wanted: return [] seen = res.account or str(sub.get("account") or "") if not seen: return [Check("account", UNKNOWN, "Clips Kitty can't see which account posted this")] if _bare_account(seen) == _bare_account(wanted): return [Check("account", PASS)] return [Check("account", FAIL, f"Posted from {seen}, and this campaign wants {wanted}")] def _routing(campaign: dict, source: dict | None) -> list[Check]: filters = json_field(campaign.get("filters"), {}) plugins = filters.get("plugins") or [] modes = filters.get("modes") or [] creators = filters.get("creator_ids") or [] if not (plugins or modes or creators): return [] if source is None: return [Check("routing", UNKNOWN, "Clips Kitty can't tell where this clip came from")] mode = source.get("mode") or "unknown" results = [] if plugins: plugin = source.get("plugin") or "" if plugin: results.append(plugin in plugins) else: # No plugin stamp: a clip a plugin made always has one, so it is # not from a plugin, unless nothing at all is known about it. results.append(None if mode == "unknown" else False) if modes: results.append(None if mode == "unknown" else mode in modes) if creators: creator = source.get("creator_id") results.append(None if creator is None else int(creator) in [int(c) for c in creators]) if False in results: return [Check("routing", FAIL, "This clip isn't one the campaign's routing takes")] if None in results: return [Check("routing", UNKNOWN, "Clips Kitty can't tell where this clip came from")] return [Check("routing", PASS)] def routes(campaign, source: dict | None) -> bool: """Whether a campaign's routing takes a clip from `source`: always when it has no filters, and never when a filter is set and the source can't be told (automation attaches only what is known to match).""" found = _routing(dict(campaign), source) return not found or found[0].result == PASS def check(campaign, sub, res, *, source: dict | None, now, destination=None) -> list[Check]: """Every check once the post has a link: precheck on the caption it went out with (no scheduled time: it is live by now), then public, account, routing, and the way of submitting: a webhook's destination exists and is connected (`destination`, its discord_destinations row), an assisted campaign has a channel saved.""" campaign, sub = dict(campaign), dict(sub) out = precheck(campaign, platform=sub["platform"], duration_s=float(sub["end_s"]) - float(sub["start_s"]), caption=res.caption, now=now) out.append(_public(sub, res)) out += _account(campaign, sub, res) out += _routing(campaign, source) warning = urls.link_warning(sub["platform"], res.url or str(sub.get("post_url") or "")) if warning: out.append(Check("link", UNKNOWN, warning)) method = str(sub.get("method") or campaign.get("method") or "") if method == "discord_webhook": state = str(cell(destination, "state", "")) if destination is not None else "" if destination is None: out.append(Check("destination", FAIL, "The webhook to post to is missing: pick one")) elif state != "connected": out.append(Check("destination", FAIL, f"The webhook is {state or 'not usable'}: submit it by hand or pick another")) else: out.append(Check("destination", PASS)) elif method == "discord_assisted": if str(campaign.get("channel_id") or ""): out.append(Check("channel", PASS)) else: out.append(Check("channel", UNKNOWN, "No channel saved: you'll find the channel yourself")) return out def verdict(checks: list[Check], *, method: str, trust_account: bool) -> str: """What to do next: "ok", "review", "manual" or "expired". A failed deadline expires it; a failed destination sends it to be done by hand; any other failure stops for review. An unknown stops for review only when the next step would be automatic (a webhook); an account Clips Kitty can't see counts as right with `trust_account`. """ failed = {c.name for c in checks if c.result == FAIL} if "deadline" in failed: return "expired" if "destination" in failed: return "manual" if failed: return "review" unknown = [c for c in checks if c.result == UNKNOWN and not (c.name == "account" and trust_account)] if unknown and method == "discord_webhook": return "review" return "ok" - campaigns/formatter.py (11104 bytes)
"""The words of a submission: the message, the slash command a person types, and the payload a webhook posts. Placeholders are filled by plain replacement in one pass, like the watch footer (server/automation.py render_footer): any other braces stay exactly as typed, and a value that happens to contain "{title}" is not filled again. A webhook can only post a message. It cannot run another bot's slash command, so the command fields are for the person submitting by hand, and Discord is never asked to mention anyone. """ import re from campaigns.model import ( FAIL, Check, cell, json_field, parse_utc, platform_label, ) #: The values Clips Kitty fills in itself. An extra field can't take these names. BUILTINS = ("url", "platform", "platform_id", "title", "campaign", "account", "creator", "source_url", "duration", "posted_at", "hashtags", "discord_user") #: Discord's limits for a message and a forum thread's title. MAX_CONTENT = 2000 MAX_THREAD_NAME = 100 _PLACEHOLDER = re.compile(r"\{([a-z0-9_]{1,32})\}") _MORE = "…" #: The values the campaign form's preview shows when it has no clip to use. SAMPLE = { "url": "https://www.tiktok.com/@example/video/7000000000000000000", "platform": "TikTok", "platform_id": "tiktok", "title": "Example clip title", "campaign": "Example Campaign", "account": "@example", "creator": "Example Creator", "source_url": "https://www.youtube.com/watch?v=xxxxxxxxxxx", "duration": "30", "posted_at": "2026-01-31", "hashtags": "#example #clips", "discord_user": "example_user", } def _local_date(text) -> str: moment = parse_utc(text) return moment.astimezone().date().isoformat() if moment is not None else "" def _clip(db, sub: dict): clip = db.get_clip(int(sub.get("clip_id") or 0)) if sub.get("clip_id") else None if clip is None: clip = db.conn.execute( "SELECT * FROM clips WHERE video_id = ? AND start_s = ? AND end_s = ?", (sub["video_id"], round(float(sub["start_s"]), 2), round(float(sub["end_s"]), 2)), ).fetchone() return clip def _extra_values(campaign: dict, given: dict) -> dict[str, str]: out = {} for field in json_field(campaign.get("extra_fields"), []): name = str(field.get("name") or "") if isinstance(field, dict) else "" if not name or name in BUILTINS: continue value = given.get(name) out[name] = str(field.get("default") or "") if value in (None, "") else str(value) return out def values_for(db, sub, campaign, *, discord_user: str = "") -> dict[str, str]: """Every placeholder's value for one submission: the builtins, then each extra field from the row's extra_values, else the field's default.""" from server.woopsocial_service import tags_of sub, campaign = dict(sub), dict(campaign) video = db.conn.execute( "SELECT channel_name, source_url FROM videos WHERE video_id = ?", (sub["video_id"],) ).fetchone() clip = _clip(db, sub) values = { "url": str(sub.get("post_url") or ""), "platform": platform_label(sub["platform"]), "platform_id": str(sub["platform"]), "title": str(sub.get("clip_title") or ""), "campaign": str(campaign.get("name") or ""), "account": str(sub.get("account") or ""), "creator": str(cell(video, "channel_name", "")) if video is not None else "", # Blank when it isn't known: never made up. "source_url": str(cell(video, "source_url", "")) if video is not None else "", "duration": str(round(float(sub["end_s"]) - float(sub["start_s"]))), "posted_at": _local_date(sub.get("posted_at")), "hashtags": " ".join(f"#{t}" for t in tags_of(clip)) if clip is not None else "", "discord_user": str(discord_user or ""), } values.update(_extra_values(campaign, json_field(sub.get("extra_values"), {}))) return values def render(template: str, values: dict[str, str]) -> str: """Fill {name} placeholders from `values`; an unknown one stays as typed.""" def fill(found: re.Match) -> str: name = found.group(1) return str(values[name]) if name in values else found.group(0) return _PLACEHOLDER.sub(fill, str(template or "")) def command(campaign, values: dict, *, arg_values: dict | None = None) -> dict: """The slash command to type: {"name", "args": [{"name", "value"}], "url_arg", "line"}. The link's argument comes first, and its value is the link when its template is blank. A per-row `arg_values[name]` replaces that argument's value. `line` is the whole command written out, for reference only: nobody has checked that Discord runs a pasted line.""" campaign = dict(campaign) name = str(campaign.get("command_name") or "").lstrip("/") url_arg = str(campaign.get("url_argument") or "") if not name: return {"name": "", "args": [], "url_arg": "", "line": ""} overrides = arg_values or {} args = [a for a in json_field(campaign.get("command_args"), []) if isinstance(a, dict) and a.get("name")] args.sort(key=lambda a: a["name"] != url_arg) out = [] for arg in args: arg_name = str(arg["name"]) if overrides.get(arg_name) is not None: value = str(overrides[arg_name]) else: template = str(arg.get("value") or "") if arg_name == url_arg and not template: template = "{url}" value = render(template, values) out.append({"name": arg_name, "value": value}) line = " ".join([f"/{name}", *(f"{a['name']}:{a['value']}" for a in out)]) return {"name": name, "args": out, "url_arg": url_arg, "line": line} def assisted_text(campaign, values: dict, *, message_text: str = "", arg_values: dict | None = None) -> str: """What the person submits by hand: the row's own text when it has one, else the command line when the campaign uses a command, else its message.""" campaign = dict(campaign) if str(message_text or "").strip(): return render(message_text, values) if campaign.get("command_name"): return command(campaign, values, arg_values=arg_values)["line"] return render(campaign.get("message_template") or "{url}", values) def fit(template: str, values: dict[str, str], limit: int = MAX_CONTENT) -> str: """The rendered template, at most `limit` characters when trimming can get it there: the hashtags lose tags from the end, then the title is cut. The link is never cut, so a message can still come back too long.""" text = render(template, values) if len(text) <= limit: return text trimmed = dict(values) tags = str(trimmed.get("hashtags") or "").split() while tags and "{hashtags}" in template and len(text) > limit: tags.pop() trimmed["hashtags"] = " ".join(tags) text = render(template, trimmed) title = str(trimmed.get("title") or "") uses = template.count("{title}") while title and uses and len(text) > limit: over = len(text) - limit keep = max(0, len(title.rstrip(_MORE)) - (-(-over // uses)) - 1) title = title[:keep].rstrip() + _MORE if keep else "" trimmed["title"] = title text = render(template, trimmed) return text def _thread_name(campaign: dict, values: dict) -> str: name = render(campaign.get("thread_name_template") or "{title}", values).strip() name = name or str(values.get("title") or "") or str(values.get("campaign") or "") or "Clip" if len(name) > MAX_THREAD_NAME: name = name[:MAX_THREAD_NAME - 1].rstrip() + _MORE return name def webhook_payload(campaign, sub, values: dict, dest) -> dict: """The JSON a webhook posts: the row's own text or the campaign's message, fitted to Discord's 2,000 characters, with every mention switched off; a thread title when the destination is a forum and no thread is set.""" campaign, sub = dict(campaign), dict(sub) own = str(sub.get("message_text") or "") template = own if own.strip() else str(campaign.get("message_template") or "{url}") payload = { "content": fit(template, values), "allowed_mentions": {"parse": []}, } if dest is not None and int(cell(dest, "is_forum", 0) or 0) and not cell(dest, "thread_id", ""): payload["thread_name"] = _thread_name(campaign, values) return payload def message_checks(campaign, values: dict, *, method: str, message_text: str = "", dest=None) -> list[Check]: """What stops a submission's words from going out as they are: the link or a required extra field left empty ("Fill in ..."), a command typed by hand whose arguments leave the link out, and for a webhook a message still over 2,000 characters after trimming, or empty. Each one fails, so the verdict stops it for review.""" campaign = dict(campaign) out = [] url = str(values.get("url") or "") if not url: out.append(Check("fields", FAIL, "Fill in the post's link")) elif method != "discord_webhook" and campaign.get("command_name") and not str(message_text or "").strip(): args = command(campaign, values)["args"] if args and not any(url in a["value"] for a in args): out.append(Check("fields", FAIL, "The command leaves out the post's link: " "pick which argument takes it")) for field in json_field(campaign.get("extra_fields"), []): if isinstance(field, dict) and field.get("required") and not str(values.get(field.get("name"), "")): out.append(Check("fields", FAIL, f"Fill in {field.get('label') or field.get('name')}")) if method == "discord_webhook": content = webhook_payload(campaign, {"message_text": message_text}, values, dest)["content"] if not content.strip(): out.append(Check("length", FAIL, "The message is empty")) elif len(content) > MAX_CONTENT: out.append(Check("length", FAIL, f"The message is longer than Discord's {MAX_CONTENT:,} characters: shorten it")) return out def preview(campaign, sample: dict | None) -> dict: """What a campaign's settings produce, for the form: {"message", "command", "thread_name", "length", "problems"}. `sample` gives the values to use (a real clip's, or a draft caption's); the rest are SAMPLE's.""" campaign = dict(campaign) values = {**SAMPLE, "campaign": str(campaign.get("name") or SAMPLE["campaign"]), **(sample or {})} for name, value in _extra_values(campaign, {}).items(): values.setdefault(name, value) message = fit(campaign.get("message_template") or "{url}", values) method = str(campaign.get("method") or "") return { "message": message, "command": command(campaign, values), "thread_name": _thread_name(campaign, values), "length": len(message), "problems": [c.message for c in message_checks(campaign, values, method=method)], } - campaigns/links.py (4397 bytes)
"""Discord links: a channel or message link pasted into a campaign, and a webhook URL pasted as a destination. A webhook URL carries the token that lets anyone post to its channel. Nothing here puts it into a result, a problem message or an exception: a parsed link names its kind and ids, and the token comes back only from parse_webhook_url, to be stored with core/secrets. """ import re from dataclasses import asdict, dataclass #: A Discord id. Kept as text everywhere: it overflows JavaScript numbers. SNOWFLAKE = r"\d{17,20}" SNOWFLAKE_RE = re.compile(rf"^{SNOWFLAKE}$") _HOST = r"(?:(?:ptb|canary|www)\.)?discord(?:app)?\.com" _CHANNELS = re.compile( rf"^{_HOST}/channels/(?P<guild>{SNOWFLAKE}|@me)" rf"(?:/(?P<channel>{SNOWFLAKE})(?:/(?P<message>{SNOWFLAKE}))?)?/?$", re.IGNORECASE, ) _WEBHOOK = re.compile( rf"^{_HOST}/api(?:/v\d+)?/webhooks/(?P<id>{SNOWFLAKE})/(?P<token>[\w-]+)/?$", re.IGNORECASE ) _INVITE = re.compile(rf"^(?:(?:www\.)?discord\.gg|{_HOST}/invite)/[\w-]+/?$", re.IGNORECASE) COPY_LINK = "right-click the channel → Copy Link" INVITE_PROBLEM = f"That's an invite link. Paste the channel's link: {COPY_LINK}." WEBHOOK_PROBLEM = "That's a webhook link: add it in Settings → Discord → Destinations" DIRECT_PROBLEM = f"That's a direct message, not a server channel. Paste the channel's link: {COPY_LINK}." NOT_DISCORD_PROBLEM = f"That isn't a Discord channel link. In Discord, {COPY_LINK}, then paste it here." BAD_WEBHOOK = ( "That isn't a Discord webhook link. In Discord: Server Settings → Integrations → Webhooks " "→ Copy Webhook URL." ) @dataclass class DiscordLink: kind: str # channel | message | guild | webhook | invite | none guild_id: str = "" channel_id: str = "" message_id: str = "" problem: str = "" def as_dict(self) -> dict: return asdict(self) def _bare(text) -> str: """The link without its scheme, query or fragment, ready to match. Discord wraps a link in <> to hide its preview, and people paste that too.""" link = str(text or "").strip().strip("<>").strip() link = re.sub(r"^[a-z]+://", "", link, flags=re.IGNORECASE) return re.split(r"[?#]", link, maxsplit=1)[0] def parse_link(text) -> DiscordLink: """What a pasted Discord link points at. Never echoes a webhook's token.""" link = _bare(text) if not link: return DiscordLink("none") if _WEBHOOK.match(link): return DiscordLink("webhook", problem=WEBHOOK_PROBLEM) if _INVITE.match(link): return DiscordLink("invite", problem=INVITE_PROBLEM) found = _CHANNELS.match(link) if found is None: return DiscordLink("none", problem=NOT_DISCORD_PROBLEM) guild, channel, message = found.group("guild"), found.group("channel"), found.group("message") if guild == "@me": return DiscordLink("none", problem=DIRECT_PROBLEM) if message: return DiscordLink("message", guild, channel, message) if channel: return DiscordLink("channel", guild, channel) return DiscordLink("guild", guild) def parse_channel_link(text) -> DiscordLink: """A campaign's channel link: a channel, a message in it (its channel is the one meant) or a server. Raises ValueError with the message to show for anything else: an invite, a webhook, a direct message, not Discord at all. An empty link is no link (kind "none"), not an error.""" link = parse_link(text) if link.kind in ("channel", "message", "guild") or (link.kind == "none" and not link.problem): return link raise ValueError(link.problem) def parse_webhook_url(text) -> tuple[str, str]: """(webhook_id, token) from a pasted webhook URL. Raises ValueError with a message that never contains what was pasted.""" found = _WEBHOOK.match(_bare(text)) if found is None: raise ValueError(BAD_WEBHOOK) return found.group("id"), found.group("token") def channel_url(guild_id: str, channel_id: str = "", message_id: str = "") -> str: """The link that opens a server, a channel or a message in Discord; '' when the server isn't known.""" parts = [p for p in (guild_id, channel_id, message_id) if p] if not guild_id or not all(SNOWFLAKE_RE.match(p) for p in parts): return "" if message_id and not channel_id: return "" return "https://discord.com/channels/" + "/".join(parts) - campaigns/model.py (23678 bytes)
"""The campaign vocabulary: methods, the submission statuses in the owner's words and the groups they are shown in, the platforms a campaign can take, a check's shape, the waiting times, and every rule a saved campaign must keep. clean_campaign holds the rules rather than a pydantic model in the route, so CI's main job, which has no FastAPI, tests them. """ import json import math import re from dataclasses import dataclass from datetime import datetime, timezone from server import posts METHODS = ("discord_assisted", "discord_webhook", "website") # ---- where the secrets live ---------------------------------------------------------- # # In core/secrets (DPAPI on Windows), never in the database, a log line or an # API answer: the sign-in's tokens under one name, and each webhook # destination's id and token under a name of its own (DESIGN §3.4). # The name core/secrets keeps the sign-in under (a name, not a secret). OAUTH_SECRET_NAME = "discord_oauth" OAUTH_SECRET = OAUTH_SECRET_NAME def webhook_secret_name(destination_id: int) -> str: """The secret holding one webhook destination's {"webhook_id", "token"}.""" return f"discord_webhook_{int(destination_id)}" # ---- statuses ------------------------------------------------------------------ STATUSES = ( "waiting_publication", "waiting_url", "ready", "review", "manual", "sending", "sent", "awaiting_confirmation", "confirmed", "rejected", "failed", "expired", "cancelled", ) #: The owner's wording for each status (request §11). LABELS = { "waiting_publication": "Waiting for publication", "waiting_url": "Waiting for public URL", "ready": "Ready to submit", "review": "Awaiting review", "manual": "Manual action required", "sending": "Sending", "sent": "Message sent", "awaiting_confirmation": "Awaiting confirmation", "confirmed": "Confirmed", "rejected": "Rejected", "failed": "Failed", "expired": "Expired", "cancelled": "Cancelled", } #: Nothing has been sent yet. PRE_SEND = ("waiting_publication", "waiting_url", "ready", "review", "manual") #: Nothing moves a row out of these, except an explicit re-attach. TERMINAL = ("confirmed", "rejected", "expired", "cancelled") #: How the lists group rows (DESIGN §4.1 "Group shown"). GROUPS = ("to_submit", "outcome_to_record", "waiting", "sent", "done") #: "Message sent" for a row the user submitted by hand: Clips Kitty sent nothing. USER_SENT_LABEL = "You submitted it" def cell(row, key, default=None): """A field of a dict or a sqlite3.Row, or `default` when it has none.""" try: value = row[key] except (KeyError, IndexError): return default return default if value is None else value def to_submit(row) -> bool: """Work the user still has to do: review it, submit it by hand, retry or cancel a failed one, submit one saved for later, or check Discord after a send nobody can vouch for. Drives the badge and the notifications, so a webhook message already posted and waiting for the campaign's verdict is never counted here.""" status = cell(row, "status", "") if status in ("review", "manual", "failed"): return True if status == "ready": return bool(int(cell(row, "hold", 0) or 0)) if status == "awaiting_confirmation": return not cell(row, "sent_by", "") return False def outcome_to_record(row) -> bool: """Sent (by the webhook or by the user), so only the campaign's verdict is left to record.""" return cell(row, "status", "") == "awaiting_confirmation" and bool(cell(row, "sent_by", "")) def group(row) -> str: """Which of GROUPS a row is listed under.""" if to_submit(row): return "to_submit" if outcome_to_record(row): return "outcome_to_record" status = cell(row, "status", "") if status == "sent": return "sent" if status in TERMINAL: return "done" return "waiting" def status_label(row) -> str: status = cell(row, "status", "") if status == "sent" and cell(row, "sent_by", "") == "user": return USER_SENT_LABEL return LABELS.get(status, status) # ---- platforms and modes ------------------------------------------------------------ #: Not places a clipping campaign takes a post link from: Upload-Post's #: Discord and Telegram upload the video file itself, and a Google Business #: update is a listing post with no public link form Clips Kitty can check or #: open (DESIGN §5.2 has no rule for it). NOT_CAMPAIGN_PLATFORMS = ("discord", "telegram", "google_business") def _platforms() -> tuple[str, ...]: from publish.uploadpost import VIDEO_PLATFORMS as upload_post from publish.woopsocial import PLATFORMS as woopsocial out: list[str] = [] for platform in (*woopsocial, *upload_post): if platform not in out and platform not in NOT_CAMPAIGN_PLATFORMS: out.append(platform) return tuple(out) #: Every platform the publishers can post a video to that a campaign can take. PLATFORMS = _platforms() PLATFORM_LABELS = { "youtube": "YouTube", "tiktok": "TikTok", "instagram": "Instagram", "facebook": "Facebook", "x": "X", "threads": "Threads", "linkedin": "LinkedIn", "pinterest": "Pinterest", "bluesky": "Bluesky", "reddit": "Reddit", } def platform_label(platform: str) -> str: return PLATFORM_LABELS.get(platform, str(platform or "").replace("_", " ").title()) #: What made a clip, for routing (eligibility.clip_source). "unknown" is a #: source's answer when nothing says, and never a filter value. MODES = ("regular", "sports", "longform", "vertical_live", "podcast", "gaming", "pipeline") # ---- checks ------------------------------------------------------------------------- PASS, FAIL, UNKNOWN = "pass", "fail", "unknown" @dataclass(frozen=True) class Check: name: str result: str # pass | fail | unknown message: str = "" def as_dict(self) -> dict: return {"name": self.name, "result": self.result, "message": self.message} # ---- times (seconds unless named otherwise) ------------------------------------------- #: Waits before each new try of a webhook send that failed before reaching #: Discord; after the last one the row is failed. RETRY_DELAYS = (60, 300, 1800, 7200, 21600) #: A row still `sending` this long after it started is swept to #: awaiting_confirmation at the start of a pass. SENDING_STALE_SECONDS = 120 #: A post published without a link is waited on this long for one. URL_WAIT_HOURS = 24 #: A row with no progress this long after its post's due time goes to manual. STALE_DAYS = posts.FOLLOW_DAYS #: A failed watch post may still be re-sent by the watch until this long after. FAILED_POST_WAIT_HOURS = 48 #: How often a waiting row is looked at again: its post is due (or its time #: unknown), its post is scheduled ahead, it waits on a failed post or is a #: manual row with no link. CHECK_DUE_SECONDS = 60 CHECK_SCHEDULED_SECONDS = 10 * 60 CHECK_FAILED_SECONDS = 60 * 60 #: At most one send per destination this often, and this many sends a pass. SEND_SPACING_SECONDS = 2 SENDS_PER_PASS = 5 # ---- JSON columns ------------------------------------------------------------------------- def json_field(value, default): """A JSON column's value, decoded; `default` when it is empty, unreadable or not the same kind of thing (a list where a dict belongs). A value that is already decoded is taken as it is.""" if isinstance(value, (list, dict)): parsed = value else: try: parsed = json.loads(value) if value else default except (TypeError, ValueError): return default if default is not None and not isinstance(parsed, type(default)): return default return parsed def utc_text(moment: datetime) -> str: """An aware datetime as the UTC text the campaign tables keep.""" return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") def parse_utc(text) -> datetime | None: """RFC 3339 text as an aware UTC datetime; a value without a zone is UTC. None when it can't be read.""" raw = str(text or "").strip() if not raw: return None try: moment = datetime.fromisoformat(raw.replace("Z", "+00:00").replace("z", "+00:00")) except ValueError: return None if moment.tzinfo is None: moment = moment.replace(tzinfo=timezone.utc) return moment.astimezone(timezone.utc) # ---- the campaign rules ------------------------------------------------------------------- #: What a campaign form may set: every column but the id, the timestamps and #: `archived`, which only delete_campaign sets. EDITABLE = ( "name", "enabled", "method", "guild_id", "guild_name", "channel_id", "channel_name", "channel_link", "destination_id", "website_url", "requirements", "platforms", "required_accounts", "trust_account", "required_hashtags", "required_mentions", "required_text", "min_seconds", "max_seconds", "deadline", "approval_required", "instructions", "command_name", "command_args", "url_argument", "extra_fields", "message_template", "thread_name_template", "review_first", "filters", "notes", ) TEMPLATE_MAX = 2000 NAME_MAX = 100 MAX_ARGS = 25 COMMAND_RE = re.compile(r"^/?([\w-]{1,32})$") ARG_NAME_RE = re.compile(r"^[\w-]{1,32}$") EXTRA_NAME_RE = re.compile(r"^[a-z0-9_]{1,32}$") FILTER_KEYS = ("plugins", "modes", "creator_ids") # A new campaign's values, for the rules that look at two fields at once. _NEW = { "method": "discord_assisted", "destination_id": 0, "website_url": "", "command_name": "", "command_args": [], "url_argument": "", "min_seconds": 0.0, "max_seconds": 0.0, "guild_id": "", "channel_id": "", } def carries_link(args, url_argument: str) -> bool: """Whether a command's arguments put the post's link in it: the link's argument left blank (it is then filled with the link), or any argument whose value has {url} in it.""" for arg in args: if not isinstance(arg, dict): continue value = str(arg.get("value") or "") if "{url}" in value or (url_argument and arg.get("name") == url_argument and not value): return True return False def _text(value, what: str, *, limit: int = 0) -> str: if value is None: return "" if not isinstance(value, str): raise ValueError(f"{what} must be text") value = value.strip() if limit and len(value) > limit: raise ValueError(f"{what} is too long: at most {limit:,} characters") return value def _flag(value, what: str) -> int: if isinstance(value, bool) or value in (0, 1): return int(value) raise ValueError(f"{what} must be true or false") def _number(value, what: str) -> float: if isinstance(value, bool): raise ValueError(f"{what} must be a number") try: number = float(value or 0) except (TypeError, ValueError): raise ValueError(f"{what} must be a number") from None if not math.isfinite(number): raise ValueError(f"{what} must be a number") if number < 0: raise ValueError(f"{what} can't be negative") return number def _whole(value, what: str) -> int: if isinstance(value, bool): raise ValueError(f"{what} must be a whole number") try: number = int(value or 0) except (TypeError, ValueError): raise ValueError(f"{what} must be a whole number") from None if number < 0 or float(value or 0) != number: raise ValueError(f"{what} must be a whole number") return number def _decoded(value, kind, what: str): """A JSON field given as JSON text or already decoded.""" if isinstance(value, str): try: value = json.loads(value) if value.strip() else kind() except ValueError: raise ValueError(f"{what} isn't valid JSON") from None if value is None: value = kind() if not isinstance(value, kind): raise ValueError(f"{what} must be a {'list' if kind is list else 'set of named values'}") return value def _https(value, what: str) -> str: url = _text(value, what, limit=TEMPLATE_MAX) if not url: return "" if re.search(r"\s", url) or not re.match(r"^https://[^/\s?#@]+\.[^/\s?#@]+(?:[/?#].*)?$", url, re.I): raise ValueError(f"{what} must be a full https:// link") return url def _snowflake(value, what: str) -> str: from campaigns import links text = _text(value, what) if text and not links.SNOWFLAKE_RE.match(text): raise ValueError(f"{what} must be a Discord ID: 17 to 20 digits") return text def _platform_list(value) -> list[str]: out: list[str] = [] for item in _decoded(value, list, "Platforms"): platform = str(item or "").strip().lower() if platform not in PLATFORMS: raise ValueError(f"Clips Kitty can't submit {platform or 'that'} posts to a campaign") if platform not in out: out.append(platform) return out def _accounts(value) -> dict[str, str]: out: dict[str, str] = {} for platform, handle in _decoded(value, dict, "Required accounts").items(): if platform not in PLATFORMS: raise ValueError(f"Required accounts: {platform} isn't a platform a campaign takes") name = _text(handle, f"The {platform_label(platform)} account", limit=NAME_MAX).lstrip("@").strip() if name: out[platform] = name return out def _tags(value, what: str, mark: str) -> list[str]: out: list[str] = [] for item in _decoded(value, list, what): tag = _text(item, what, limit=NAME_MAX).lstrip(mark).strip() if not tag: continue if re.search(r"\s", tag): raise ValueError(f"{what}: “{tag}” has a space in it; add one per line") if tag.lower() not in (t.lower() for t in out): out.append(tag) return out def _deadline(value) -> str: text = _text(value, "The deadline") if not text: return "" moment = parse_utc(text) if moment is None: raise ValueError("The deadline isn't a date and time Clips Kitty can read") return utc_text(moment) def _command(value) -> str: text = _text(value, "The command") if not text: return "" found = COMMAND_RE.match(text) if found is None: raise ValueError( "The command is its name only, like /submit: letters, digits, - and _, at most 32" ) return found.group(1) def _args(value) -> list[dict]: out, seen = [], set() items = _decoded(value, list, "The command's arguments") if len(items) > MAX_ARGS: raise ValueError(f"A command takes at most {MAX_ARGS} arguments") for item in items: if not isinstance(item, dict): raise ValueError("Each argument needs a name and a value") name = _text(item.get("name"), "An argument's name") if no - campaigns/post_links.py (7168 bytes)
"""Send published clip links to a chosen webhook using the durable queue.""" import json import time from campaigns import queue PENDING_PREFIX = "discord_post_link:" def validate_destination(db, destination_id: int | None) -> None: if destination_id is not None and db.get_destination(destination_id) is None: raise ValueError("That Discord destination doesn't exist.") def campaign_for_destination(db, destination_id: int) -> int: validate_destination(db, destination_id) # Keep the original internal name so manual publishes and watched # channels share the same delivery identity, including after an upgrade. name = f"Watched channel links: Discord {destination_id}" with db.transaction(): row = db.conn.execute( "SELECT id FROM campaigns WHERE name = ? AND destination_id = ? " "AND method = 'discord_webhook' ORDER BY id LIMIT 1", (name, destination_id), ).fetchone() return int(row["id"]) if row is not None else db.insert_campaign( name, method="discord_webhook", destination_id=destination_id, message_template="{title}\n{url}", approval_required=0, ) def prepare(db, *, destination_id: int | None, clip_id: int, platforms: list[str], provider: str, title: str = "", publish_job_id: int = 0) -> None: """Save delivery intent before uploading; only a live, public post sends. A request cut off after uploading still has its intent on disk. Missing, failed or refused posts never supply a live URL and cannot send a message. Repeated requests keep the original submission, so never send twice. """ if destination_id is None: return validate_destination(db, destination_id) if provider != "youtube": # The next provider request id is not known until upload returns. # Keep intent separately so a previous publish's live URL cannot be # picked up while the new upload is still in progress. recover(db) clip = db.get_clip(clip_id) if clip is None: raise ValueError("That clip no longer exists.") previous = {row["platform"]: [row["request_id"], row["created_at"]] for row in db.clip_publishes(clip_id)} intent = {"destination_id": destination_id, "clip_id": clip_id, "provider": provider, "title": title, "video_id": clip["video_id"], "start_s": clip["start_s"], "end_s": clip["end_s"], "before": {p: previous.get(p) for p in platforms}} db.set_flag(f"{PENDING_PREFIX}{destination_id}:{provider}:{clip_id}", json.dumps(intent)) return _attach(db, destination_id=destination_id, clip_id=clip_id, platforms=platforms, provider=provider, title=title, publish_job_id=publish_job_id) def _attach(db, *, destination_id: int, clip_id: int, platforms: list[str], provider: str, title: str, publish_job_id: int = 0) -> dict: with db.transaction(): campaign_id = campaign_for_destination(db, destination_id) result = queue.attach( db, items=[{"clip_id": clip_id, "platforms": platforms}], campaign_ids=[campaign_id], provider=provider, publish_job_id=publish_job_id, now=time.time(), ) for row in result["created"]: if title: db.set_submission(int(row["id"]), clip_title=title) return result def send_existing(db, *, destination_id: int, clip_id: int, platform: str, provider: str) -> dict: """Queue a saved live post from the editor, without publishing it again.""" from server import posts post = next((p for p in posts.posts_for_clip(db, clip_id) if p["platform"] == platform and p["provider"] == provider and p["live"]), None) if post is None: raise ValueError("This post doesn't have a live public link yet.") with db.transaction(): result = _attach(db, destination_id=destination_id, clip_id=int(post["clip_id"]), platforms=[platform], provider=provider, title="") rows = result["created"] + result["existing"] if not rows: raise ValueError(result["refused"][0]["reason"] if result["refused"] else "Couldn't queue this link.") row = rows[0] if row["status"] in ("manual", "review", "failed", "expired", "cancelled"): raise ValueError(row["status_reason"] or "This delivery needs attention before it can be sent.") return {"submission_id": int(row["id"]), "status": row["status"], "already_sent": row["sent_by"] in ("webhook", "user"), "check_discord": row["status"] == "awaiting_confirmation" and not row["sent_by"]} def recover(db, *, completed_clip: int = 0, provider: str = "") -> None: """Attach saved outcomes, including after a crash between publish and attach. A refresh changes neither request_id nor created_at, so refreshing an older post cannot fulfil the new publish's intent. Explicit completion also handles providers returning the same idempotent result again. """ from server.woopsocial_service import ALIVE pending = db.conn.execute("SELECT key FROM app_state WHERE key GLOB ?", (PENDING_PREFIX + "*",)).fetchall() for entry in pending: with db.transaction(): # Re-read under the write lock: another worker may have settled it. saved = db.get_flag(entry["key"]) if not saved: continue intent = json.loads(saved) if db.get_destination(intent["destination_id"]) is None: db.conn.execute("DELETE FROM app_state WHERE key = ?", (entry["key"],)) continue finished = completed_clip == intent["clip_id"] and provider == intent["provider"] rows = db.conn.execute( "SELECT * FROM clip_publishes WHERE provider = ? AND (clip_id = ? OR " "(video_id = ? AND start_s = ? AND end_s = ?))", (intent["provider"], intent["clip_id"], intent["video_id"], intent["start_s"], intent["end_s"]), ).fetchall() for row in rows: platform = row["platform"] if platform not in intent["before"]: continue if not finished and intent["before"][platform] == [row["request_id"], row["created_at"]]: continue if row["state"] in ALIVE: _attach(db, destination_id=intent["destination_id"], clip_id=int(row["clip_id"]), platforms=[platform], provider=intent["provider"], title=intent["title"]) elif row["state"] not in ("failed", "skipped"): continue del intent["before"][platform] if intent["before"]: db.conn.execute("UPDATE app_state SET value = ? WHERE key = ?", (json.dumps(intent), entry["key"])) else: db.conn.execute("DELETE FROM app_state WHERE key = ?", (entry["key"],)) - campaigns/queue.py (62413 bytes)
"""The submission queue: attaching posts to campaigns, following each post until it is live with a public link, evaluating it then (DESIGN §5.3), and the background step that sends what a webhook may send (DESIGN §6). Everything lives in campaign_submissions. The background step (`advance`, run by server/publisher.py's PublishWorker) keeps nothing in memory but the global rate-limit pause it is handed, so after a restart it carries on from the rows. Every status change goes through the state machine, compare-and-set, so a route and the background step never both win. A webhook send is written `sending` before the request goes out, and a row left `sending` (a crash, or an exception between the request and its outcome) is never sent again by itself: it becomes awaiting_confirmation with a "check Discord" note, because a webhook has no way to ask whether a message already arrived. """ import json import sqlite3 from datetime import datetime, timedelta from campaigns import eligibility, formatter, statemachine, urls, views from campaigns.model import ( CHECK_DUE_SECONDS, CHECK_FAILED_SECONDS, CHECK_SCHEDULED_SECONDS, FAIL, FAILED_POST_WAIT_HOURS, LABELS, METHODS, PLATFORMS, PRE_SEND, RETRY_DELAYS, SEND_SPACING_SECONDS, SENDING_STALE_SECONDS, SENDS_PER_PASS, STALE_DAYS, TEMPLATE_MAX, UNKNOWN, URL_WAIT_HOURS, cell, json_field, parse_utc, platform_label, utc_text, ) from server import posts #: The reason shown on a row that is now the user's to submit by hand. BY_HAND = { "discord_assisted": "Submit it in Discord", "website": "Submit it on the campaign's page", } #: When an attached post is submitted (DESIGN §7 "Attach modes"). MODES = ("after_publish", "now", "review", "later") #: Where an attach from a route came from (a watch attaches through attach_for_watch). ORIGINS = ("editor", "later", "publish_all") PROVIDERS = ("", "woopsocial", "upload_post", "youtube", "manual") PROVIDER_LABELS = {"woopsocial": "WoopSocial", "upload_post": "Upload-Post", "youtube": "YouTube"} #: Statuses an explicit re-attach (or a move) starts again. REOPENABLE = ("cancelled", "expired", "failed") #: Statuses a row can be edited (or moved) in. EDITABLE = (*PRE_SEND, "failed") #: Statuses the background step follows the post for (manual only without a link). FOLLOWED = ("waiting_publication", "waiting_url") GONE_CLIP = "That clip no longer exists" NO_CAMPAIGN = "That campaign doesn't exist" ARCHIVED = "That campaign is archived" SWITCHED_OFF = "That campaign is switched off" NO_DESTINATION = "The webhook to post to is missing: pick one or submit it by hand" NO_TOKEN = ("This webhook's link isn't stored on this PC: paste it again in Settings → Discord " "(Edit), or submit it by hand") RATE_LIMITED = "Discord asked Clips Kitty to slow down: it tries again shortly" REFUSED = "Discord refused the message: change it, then retry" SENT_IT = "Posted in Discord. This doesn't mean the campaign accepted it." # ---- evaluation (DESIGN §5.3) ----------------------------------------------------------- def _duplicate(db, row: dict, url: str): return db.active_submission_with_url(int(row["campaign_id"]), url, exclude_id=int(row["id"])) def evaluate(db, sub_id: int, res, *, now, event: str = "url_found", source: dict | None = None, discord_user: str = "", changes: dict | None = None): """Give submission `sub_id` the link in `res` (a urls.Resolution with a URL: the one found, or the one typed) and move it to what its checks say. 1. Another row of the campaign that still counts already holds the link: this one is cancelled, "Already submitted to this campaign as #N". 2. The link and what is known with it are saved (url, its source, post id, account, first seen, provider, due time) with every check. 3. The verdict: expired; review (or the row's Review first); manual when its webhook can't be used; otherwise a webhook row is ready (due now, or held when saved for later) and an assisted or website row is the user's to submit by hand. `event` is the state machine's (url_found for the background step, set_url for a typed link, edit or campaign_changed when a change re-evaluates). `source` is eligibility.clip_source's answer, looked up here when the campaign routes and none is given. `changes` are other fields of the row (an edit's method, destination or message) evaluated as if already saved and written in the same compare-and-set as the outcome. Returns the row after it. Only a live resolution is taken (a typed link is built as one). A blocked one is the caller's to hand to the user with its reason: its link, if it has one, is for the user to check first. """ if res.state != "live" or not res.url: raise ValueError("evaluate needs a live post's link") found = db.get_submission(sub_id) if found is None: raise statemachine.NotAllowed(f"Submission #{sub_id} doesn't exist") changes = dict(changes or {}) row = {**dict(found), **changes} campaign = db.get_campaign(int(row["campaign_id"])) if campaign is None: raise statemachine.NotAllowed(f"Submission #{sub_id}'s campaign doesn't exist") campaign = dict(campaign) moment = posts.as_utc(now) saved = { "post_url": res.url, "url_source": res.source or row.get("url_source") or "", "post_id": res.post_id or row.get("post_id") or "", "account": res.account or row.get("account") or "", "posted_at": row.get("posted_at") or res.posted_at or utc_text(moment), "provider": res.provider or row.get("provider") or "", "due_at": res.due_at or row.get("due_at") or "", } duplicate = _duplicate(db, row, res.url) if duplicate is not None: return statemachine.apply(db, sub_id, event, now=now, to="cancelled", **{**saved, **changes}, status_reason=f"Already submitted to this campaign as #{duplicate['id']}") method = str(row.get("method") or campaign.get("method") or "") destination = None if method == "discord_webhook" and int(row.get("destination_id") or 0): destination = db.get_destination(int(row["destination_id"])) if source is None and any(json_field(campaign.get("filters"), {}).values()): source = eligibility.clip_source(db, video_id=row["video_id"], start_s=row["start_s"], end_s=row["end_s"], clip_id=int(row.get("clip_id") or 0)) seen = {**row, **saved} checks = eligibility.check(campaign, seen, res, source=source, now=now, destination=destination) values = formatter.values_for(db, seen, campaign, discord_user=discord_user) checks += formatter.message_checks(campaign, values, method=method, message_text=str(row.get("message_text") or ""), dest=destination) verdict = eligibility.verdict(checks, method=method, trust_account=bool(int(cell(campaign, "trust_account", 0) or 0))) target, reason, extra = _outcome(verdict, checks, row, method, moment.timestamp()) if found["status"] == "failed": # Out of failed with a new link or setting: its sends start counting # again, as a retry's do, and the last send's error is no longer news. extra = {"attempts": 0, "error": "", **extra} fields = {**saved, **extra, **changes} try: return statemachine.apply(db, sub_id, event, now=now, to=target, **fields, checks=json.dumps([c.as_dict() for c in checks]), status_reason=reason) except sqlite3.IntegrityError: # The link index caught a row that took this link meanwhile. duplicate = _duplicate(db, row, res.url) number = f"#{duplicate['id']}" if duplicate is not None else "another submission" return statemachine.apply(db, sub_id, event, now=now, to="cancelled", **{**saved, **changes}, status_reason=f"Already submitted to this campaign as {number}") def reopen(db, sub_id: int, *, now, **fields): """Re-attach a cancelled, expired or failed submission (the user's explicit attach or a move, never automation) and return the row after it. The state machine's reopen clears the row's link, so the link is checked against the campaign's other rows again like any new one. A provider's link is found again by resolution; a link the user typed is found by nothing, so it is given back here, through evaluation and its duplicate check, at once.""" before = db.get_submission(sub_id) if before is None: raise statemachine.NotAllowed(f"Submission #{sub_id} doesn't exist") typed = str(before["post_url"] or "") if before["url_source"] != "manual" and before["provider"] != "manual": typed = "" row = statemachine.apply(db, sub_id, "reopen", now=now, **fields) if not typed: return row url, problem = urls.validate(row["platform"], typed) if problem: return row res = urls.Resolution("live", url=url, source="manual", account=urls.account_from_url(row["platform"], url)) try: return evaluate(db, sub_id, res, now=now, event="set_url") except statemachine.NotAllowed: # The background step moved it on meanwhile: what it did stands. return db.get_submission(sub_id) def _outcome(verdict: str, checks, row: dict, method: str, seconds: float) -> tuple[str, str, dict]: """(status, reason, other fields) for a verdict.""" def said(results) -> str: return "; ".join(c.message for c in checks if c.result in results and c.message) if verdict == "expired": return "expired", "The campaign's deadline passed", {} if verdict == "review": held_up = said((FAIL,)) or (said((UNKNOWN,)) if method == "discord_webhook" else "") return "review", held_up or "Check it before it is submitted", {} if int(row.get("review_first") or 0): return "review", "Check it before it is submitted", {} if verdict == "manual": return "manual", said((FAIL,)) or "Submit it by hand", {} if int(row.get("hold") or 0): return "ready", "Saved for later", {} if method == "discord_webhook": return "ready", "", {"next_attempt_at": seconds} return "manual", BY_HAND.get(method, "Submit it by hand"), {} def typed_link(platform: str, url: str) -> urls.Resolution: """A link the user typed (already validated), as evaluation takes it.""" return urls.Resolution("live", url=url, source="manual", account=urls.account_from_url(platform, url)) def current_resolution(db, row, *, now) -> urls.Resolution | None: """The row's own link as a live resolution, to evaluate it again after a change of method or destination: a link the user typed as typed, a provider's as resolution finds it now (its caption and visibility included), else as the row holds it. None when it has no link.""" sub = dict(row) url = str(sub.get("post_url") or "") if not url: return None if sub.get("url_source") == "manual": return typed_link(sub["platform"], url) found = urls.resolve(db, sub, now=now) if found.state == "live" and found.url == url: return found return urls.Resolution("live", url=url, source=str(sub.get("url_source") or "provider"), provider=str(sub.get("provider") or ""), post_id=str(sub.get("post_id") or ""), account=str(sub.get("account") or ""), due_at=str(sub.get("due_at") or "")) # ---- attaching ------------------------------------------------------------------------------ def mode_fields(mode: str, campaign) -> tuple[int, int]: """(hold, review_first) for an attach mode.""" review_first = int(cell(campaign, "review_first", 0) or 0) if mode == "review": return 0, 1 if mode == "later": return 1, review_first return 0, review_first def campaign_refusal(campaign, *, platform: str, scheduled_for: str = "", now) -> str: """Why a post on `platform` (due at `scheduled_for`, '' for now) can't be attached to `campaign`; '' when it can. A deadline that passes before the post's time refuses it too, so it never waits for a post that will be late.""" if campaign is None: return NO_CAMPAIGN if int(cell(campaign, "archived", 0) or 0): return ARCHIVED if not int(cell(campaign, "enabled", 1) or 0): return SWITCHED_OFF for check in eligibility.precheck(campaign, platform=platform, duration_s=0, scheduled_for=scheduled_for, now=now): if check.name in ("platform", "deadline") and check.result == FAIL: return check.message return "" def clip_title(clip) -> str: return str(cell(clip, "title", "") or cell(clip, "hook", "") or "") def _identity(db, clip_id: int, platform: str) -> dict | None: """The post's lasting identity for a clip: its window, from the clip row, else from a post recorded for it (a clip deleted after it went out).""" clip = db.get_clip(int(clip_id)) if clip is not None: return {"clip_id": int(clip_id), "video_id": clip["video_id"], "start_s": float(clip["start_s"]), "end_s": float(clip["end_s"]), "clip_title": clip_title(clip)} for table, extra in (("clip_publishes", " ORDER BY (platform = ?) DESC"), ("uploads", "")): args = (int(clip_id), platform) if extra else (int(clip_id),) row = db.conn.execute( f"SELECT video_id, start_s, end_s FROM {table} WHERE clip_id = ? AND video_id != ''{extra} LIMIT 1", args, ).fetchone() if row is not None: return {"clip_id": int(clip_id), "video_id": row["video_id"], "start_s": float(row["start_s"]), "end_s": float(row["end_s"]), "clip_title": ""} return None def _post_time(db, ident: dict, platform: str, provider: str, publish_job_id: int) -> str: """When the post is due (UTC text), '' for now or unknown: the scheduled time a provider row or a YouTube upload (or its queued job) carries.""" if platform == "youtube" and provider in ("youtube", "") and publish_job_id: job = db.get_publish_job(int(publish_job_id)) if job is not None: try: asked = json.loads(job["request"] or "{}") except (TypeError, ValueError): asked = {} if isinstance(asked, dict) and asked.get("publish_at"): return str(asked["publish_at"]) if provider != "youtube": row = db.conn.execute( "SELECT scheduled_for FROM clip_publishes WHERE platform = ? AND (clip_id = ? OR " "(video_id != '' AND video_id = ? AND start_s = ? AND end_s = ?)) - campaigns/service.py (23354 bytes)
"""Discord for Clips Kitty: the settings, the stored secrets, the sign-in and the webhook destinations, with no FastAPI in sight (server/discord_api.py is the routes over this), so CI's main job tests every rule here. Where things live (DESIGN §3.4): - the sign-in's tokens in core/secrets under model.OAUTH_SECRET, and each webhook destination's id and token under model.webhook_secret_name(id); - everything else that isn't secret (who is connected, the scopes, the server list) in the `discord` app_state blob; - the destinations themselves (label, server, channel, state) in the discord_destinations table, never with a token. Nothing returned here, and nothing it prints, holds a token. Signing in is optional: pasted-webhook destinations, assisted campaigns and website campaigns all work without it, and an expired sign-in never blocks a send. """ import json import threading import time from dataclasses import dataclass from pathlib import Path from campaigns import discord_oauth, discord_webhook, links, statemachine from campaigns.discord_http import DiscordError from campaigns.model import OAUTH_SECRET, cell, utc_text, webhook_secret_name from core import secrets from server import posts SETTINGS_KEY = "discord" DEFAULTS = {"client_id": "", "user": {}, "scopes": [], "connected_at": "", "expired": False, "guilds": [], "guilds_at": 0} #: The server list is asked for again after this long (Refresh servers asks at once). GUILDS_CACHE_SECONDS = 10 * 60 #: The access token is refreshed once it has less than this left. REFRESH_WITHIN_SECONDS = 24 * 3600 NOT_CONFIGURED = "Discord sign-in isn't set up in this version of Clips Kitty." LABEL_MAX = 100 #: Held while the sign-in's tokens are refreshed, replaced or wiped. Routes #: run in a thread pool, and Discord replaces the refresh token each time it #: is used: two requests refreshing with the same one would see the second #: refused, which would end a sign-in that is fine. _TOKENS_LOCK = threading.Lock() # ---- settings -------------------------------------------------------------------- def load_settings(db) -> dict: try: stored = json.loads(db.get_flag(SETTINGS_KEY, "") or "{}") except (ValueError, TypeError): stored = {} return {**DEFAULTS, **(stored if isinstance(stored, dict) else {})} def save_settings(db, patch: dict) -> dict: merged = {**load_settings(db), **{k: v for k, v in patch.items() if k in DEFAULTS}} db.set_flag(SETTINGS_KEY, json.dumps(merged)) return merged def client_id(db) -> tuple[str, str]: """(the application id in use, where it came from): the Advanced override, the one Clips Kitty ships with, or none.""" override = str(load_settings(db).get("client_id") or "").strip() if override: return override, "override" if discord_oauth.DISCORD_CLIENT_ID: return discord_oauth.DISCORD_CLIENT_ID, "shipped" return "", "none" def set_client_id(db, value) -> None: """The Advanced override: a Discord application id, or '' for the shipped one. Raises ValueError with the message to show.""" text = str(value or "").strip() if text and not links.SNOWFLAKE_RE.match(text): raise ValueError("A Discord application ID is 17 to 20 digits (Developer Portal → " "your application → General Information).") save_settings(db, {"client_id": text}) # ---- the sign-in's tokens ----------------------------------------------------------- def save_tokens(data_dir, tokens: dict, *, now: float, keep: dict | None = None) -> dict: """Keep a token answer from Discord. A refresh that sends no new refresh token keeps the one it was made with (`keep`).""" payload = { "access_token": str(tokens.get("access_token") or ""), "refresh_token": str(tokens.get("refresh_token") or (keep or {}).get("refresh_token") or ""), "expires_at": float(now) + float(tokens.get("expires_in") or 0), "scope": str(tokens.get("scope") or (keep or {}).get("scope") or ""), "token_type": str(tokens.get("token_type") or "Bearer"), } secrets.save(Path(data_dir), OAUTH_SECRET, payload) return payload def load_tokens(data_dir) -> dict | None: got = secrets.load(Path(data_dir), OAUTH_SECRET) return got if got and got.get("access_token") else None def wipe_tokens(data_dir) -> bool: return secrets.wipe(Path(data_dir), OAUTH_SECRET) def _expired(db, data_dir) -> DiscordError: """The sign-in can't be used any more: its tokens go, the card says Reconnect (the user's name stays so it can say whose).""" wipe_tokens(data_dir) save_settings(db, {"expired": True, "guilds": [], "guilds_at": 0}) return DiscordError("expired", "Your Discord sign-in has expired. Reconnect Discord in Settings → Discord.") def _stored_tokens(db, data_dir) -> dict: tokens = load_tokens(data_dir) if tokens is None: if load_settings(db).get("expired"): raise DiscordError("expired", "Your Discord sign-in has expired. Reconnect Discord in Settings → Discord.") raise DiscordError("not_connected", "Connect Discord first (Settings → Discord).") return tokens def access_token(db, data_dir, http, *, now: float) -> str: """A usable access token, refreshed when it has less than a day left. Discord refusing the refresh ends the sign-in (tokens wiped, expired); Discord not answering keeps the old token while it still works.""" tokens = _stored_tokens(db, data_dir) if float(tokens.get("expires_at") or 0) - float(now) >= REFRESH_WITHIN_SECONDS: return tokens["access_token"] with _TOKENS_LOCK: # Read again: another request may have refreshed while this one waited. tokens = _stored_tokens(db, data_dir) expires_at = float(tokens.get("expires_at") or 0) if expires_at - float(now) >= REFRESH_WITHIN_SECONDS: return tokens["access_token"] cid, _source = client_id(db) sent = str(tokens.get("refresh_token") or "") if sent and cid: try: fresh = discord_oauth.refresh(http, cid, sent) except DiscordError as e: if e.code in ("expired", "denied"): stored = load_tokens(data_dir) if stored is not None and str(stored.get("refresh_token") or "") != sent: return stored["access_token"] # replaced meanwhile: the refusal was for the old ones raise _expired(db, data_dir) from None if expires_at > float(now): return tokens["access_token"] raise return save_tokens(data_dir, fresh, now=now, keep=tokens)["access_token"] if expires_at > float(now): return tokens["access_token"] raise _expired(db, data_dir) def connected(db, data_dir) -> bool: settings = load_settings(db) user = settings.get("user") or {} return bool(isinstance(user, dict) and user.get("id")) and not settings.get("expired") \ and secrets.has(Path(data_dir), OAUTH_SECRET) # ---- webhook secrets -------------------------------------------------------------------- def save_webhook(data_dir, dest_id: int, webhook_id: str, token: str) -> None: secrets.save(Path(data_dir), webhook_secret_name(dest_id), {"webhook_id": str(webhook_id), "token": str(token)}) def load_webhook(data_dir, dest_id: int) -> dict | None: """{"webhook_id", "token"} of a destination, or None.""" got = secrets.load(Path(data_dir), webhook_secret_name(dest_id)) return got if got and got.get("webhook_id") and got.get("token") else None def wipe_webhook(data_dir, dest_id: int) -> bool: return secrets.wipe(Path(data_dir), webhook_secret_name(dest_id)) # ---- what the Settings card sees ------------------------------------------------------------- def destination_views(db, data_dir) -> list[dict]: uses: dict[int, int] = {} for campaign in db.list_campaigns(): dest = int(cell(campaign, "destination_id", 0) or 0) if dest: uses[dest] = uses.get(dest, 0) + 1 return [destination_view(db, data_dir, row, uses=uses) for row in db.list_destinations()] def destination_view(db, data_dir, row, *, uses: dict[int, int] | None = None) -> dict: """A destination as the card shows it: no token, whether one is stored, and how many campaigns post to it.""" dest_id = int(row["id"]) if uses is None: uses = {dest_id: sum(1 for c in db.list_campaigns() if int(cell(c, "destination_id", 0) or 0) == dest_id)} return { "id": dest_id, "label": row["label"], "source": row["source"], "guild_id": row["guild_id"], "guild_name": row["guild_name"], "channel_id": row["channel_id"], "webhook_name": row["webhook_name"], "is_forum": bool(int(row["is_forum"] or 0)), "thread_id": row["thread_id"], "state": row["state"], "last_error": row["last_error"], "has_token": secrets.has(Path(data_dir), webhook_secret_name(dest_id)), "campaigns": uses.get(dest_id, 0), } def status_payload(db, data_dir, *, redirect_uri: str = "") -> dict: """What the renderer may know about Discord. Never a token.""" settings = load_settings(db) cid, source = client_id(db) user = settings.get("user") if isinstance(settings.get("user"), dict) else {} return { "configured": bool(cid), "client_id": cid, "client_id_source": source, "connected": connected(db, data_dir), "expired": bool(settings.get("expired")), "user": {"id": str(user.get("id") or ""), "username": str(user.get("username") or ""), "global_name": str(user.get("global_name") or "")}, "scopes": list(settings.get("scopes") or []), "connected_at": str(settings.get("connected_at") or ""), "redirect_uri": redirect_uri or discord_oauth.redirect_uri(discord_oauth.DEFAULT_PORT), "destinations": destination_views(db, data_dir), "storage": secrets.backend_name(), } def guild_list(db, data_dir, http, *, now: float, refresh: bool = False) -> list[dict]: """The user's servers, kept in the settings blob for GUILDS_CACHE_SECONDS.""" settings = load_settings(db) fetched = float(settings.get("guilds_at") or 0) if not refresh and fetched and float(now) - fetched < GUILDS_CACHE_SECONDS and connected(db, data_dir): return list(settings.get("guilds") or []) token = access_token(db, data_dir, http, now=now) try: listed = discord_oauth.guilds(http, token) except DiscordError as e: if e.code == "expired": raise _expired(db, data_dir) from None raise save_settings(db, {"guilds": listed, "guilds_at": float(now)}) return listed def _guild_name(db, guild_id: str) -> str: for guild in load_settings(db).get("guilds") or []: if isinstance(guild, dict) and str(guild.get("id")) == str(guild_id): return str(guild.get("name") or "") return "" # ---- signing in ----------------------------------------------------------------------------- def start_sign_in(db, pending: discord_oauth.PendingSignIns, *, purpose: str, port: int, now: float, label: str = "") -> str: """The authorize link for the user's own browser: purpose "connect" (identify + guilds) or "webhook" (webhook.incoming only). The redirect is kept with the sign-in, so the exchange sends the same one.""" cid, _source = client_id(db) if not cid: raise DiscordError("not_configured", NOT_CONFIGURED) redirect = discord_oauth.redirect_uri(port) scopes = (discord_oauth.WEBHOOK_SCOPE,) if purpose == "webhook" else discord_oauth.SIGN_IN_SCOPES extra = {"label": _label(label)} if purpose == "webhook" else {} state, challenge = pending.start(purpose, redirect, now=now, extra=extra) return discord_oauth.authorize_url(cid, redirect, state, challenge, scopes) @dataclass class SignIn: """How a browser sign-in ended: what its page says, and the event the app's windows get ('' when there is nothing to tell them).""" ok: bool event: str message: str destination_id: int = 0 EXPIRED_STATE = "This sign-in has expired or was already used. Start it again from Clips Kitty." NO_WEBHOOK = ("Discord didn't hand back a webhook. Paste the channel's webhook link in Clips Kitty instead: " "Server Settings → Integrations → Webhooks → Copy Webhook URL.") def finish_sign_in(db, data_dir, http, pending: discord_oauth.PendingSignIns, *, state: str, code: str = "", error: str = "", now: float) -> SignIn: """Where the browser comes back (GET /discord/callback). Single use.""" entry = pending.take(state, now=now) if entry is None: return SignIn(False, "", EXPIRED_STATE) if error or not code: if error == "access_denied": return SignIn(False, "failed", "You didn't approve Clips Kitty in Discord, so nothing changed.") return SignIn(False, "failed", "Discord didn't finish signing you in. Start it again from Clips Kitty.") cid, _source = client_id(db) if not cid: return SignIn(False, "failed", NOT_CONFIGURED) try: tokens = discord_oauth.exchange(http, cid, code, entry.verifier, entry.redirect) except DiscordError as e: return SignIn(False, "failed", e.message) if entry.purpose == "webhook": return _webhook_from(db, data_dir, http, cid, tokens, entry) try: user = discord_oauth.me(http, tokens["access_token"]) except DiscordError as e: discord_oauth.revoke(http, cid, str(tokens.get("refresh_token") or tokens["access_token"]), "refresh_token" if tokens.get("refresh_token") else "access_token") return SignIn(False, "failed", e.message) with _TOKENS_LOCK: saved = save_tokens(data_dir, tokens, now=now) save_settings(db, {"user": user, "scopes": saved["scope"].split(), "expired": False, "connected_at": utc_text(posts.as_utc(now)), "guilds": [], "guilds_at": 0}) name = user.get("global_name") or user.get("username") or "your account" return SignIn(True, "connected", f"Connected to Discord as {name}. You can close this tab and go back " "to Clips Kitty.") def _webhook_from(db, data_dir, http, cid: str, tokens: dict, entry) -> SignIn: """webhook.incoming: the webhook becomes a destination; the token that came with it is revoked at once and never kept.""" discord_oauth.revoke(http, cid, str(tokens.get("access_token") or ""), "access_token") hook = tokens.get("webhook") if not isinstance(hook, dict) or not links.SNOWFLAKE_RE.match(str(hook.get("id") or "")) \ or not str(hook.get("token") or ""): return SignIn(False, - campaigns/statemachine.py (11302 bytes)
"""Which status changes a submission may make, and making one safely. TRANSITIONS is DESIGN §4.2 as data: for each event, the statuses it may start from, the statuses it may lead to, and who causes it (W the background step, U the user through a route, A an attach, R startup recovery). apply() checks the event against the row as it is now and writes the change compare-and-set (on the status, and on the columns the event's guard read), so a route and the background step can never both win: the loser gets Conflict and nothing is written. The event's target is decided here when the row and its campaign say what it is (approving a webhook submission makes it ready, an assisted one manual); otherwise the caller names it with `to=` (an evaluation's verdict, a send that ran out of retries). """ from collections.abc import Callable from dataclasses import dataclass from campaigns.model import PRE_SEND, STATUSES, TERMINAL, cell, utc_text from server import posts #: The outcomes of evaluating a submission that has a link (DESIGN §5.3). EVALUATED = ("ready", "review", "manual", "cancelled", "expired") #: A target meaning "stays in the status it is in". SAME = "=" _WAITING = ("waiting_publication", "waiting_url") _EDITABLE = (*PRE_SEND, "failed") _CANCELLABLE = tuple(s for s in STATUSES if s != "sending" and s not in TERMINAL) class NotAllowed(Exception): """The event can't happen to the submission as it is.""" class Conflict(NotAllowed): """The submission changed between reading it and writing the event.""" @dataclass(frozen=True) class Rule: sources: tuple[str, ...] targets: tuple[str, ...] who: str #: Says why the row can't take the event ('' when it can), beyond its status. guard: Callable[[dict, float], str] | None = None #: The columns the guard reads. The write compares them as well as the #: status, so a change to one of them made meanwhile stops the event too. watch: tuple[str, ...] = () def _no_url(row: dict, now: float) -> str: if row["status"] == "manual" and row.get("post_url"): return "It already has its link" return "" def _held(row: dict, now: float) -> str: return "" if int(row.get("hold") or 0) else "It isn't saved for later" def _webhook(row: dict, now: float) -> str: return "" if row.get("method") == "discord_webhook" else "It isn't sent by a webhook" def _sendable(row: dict, now: float) -> str: if row.get("method") != "discord_webhook": return "It isn't sent by a webhook" if int(row.get("hold") or 0): return "It's saved for later" if float(row.get("next_attempt_at") or 0) > now: return "It isn't due yet" return "" def _not_by_user(row: dict, now: float) -> str: return "You submitted it yourself" if row.get("sent_by") == "user" else "" def _not_opened(row: dict, now: float) -> str: return "Its Discord channel or page was already opened" if row.get("opened_at") else "" TRANSITIONS: dict[str, Rule] = { # Following the post (the background step). "published_no_url": Rule(("waiting_publication",), ("waiting_url",), "W"), "post_failed": Rule(("waiting_publication",), ("manual", "waiting_publication"), "W"), "post_skipped": Rule(("waiting_publication",), ("manual",), "W"), "url_found": Rule((*_WAITING, "manual"), EVALUATED, "W", _no_url, ("post_url",)), "url_overdue": Rule(("waiting_url",), ("manual",), "W"), "stale": Rule(_WAITING, ("manual",), "W"), "expire": Rule(PRE_SEND, ("expired",), "W"), # Sending by webhook. "send_start": Rule(("ready",), ("sending",), "W", _sendable, ("method", "hold", "next_attempt_at")), "destination_unusable": Rule(("ready",), ("manual",), "W", _webhook, ("method",)), "send_ok": Rule(("sending",), ("sent", "awaiting_confirmation"), "W"), "rate_limited": Rule(("sending",), ("ready",), "W"), "send_retry": Rule(("sending",), ("ready", "failed"), "W"), "send_ambiguous": Rule(("sending",), ("awaiting_confirmation",), "W"), "send_gone": Rule(("sending",), ("manual",), "W"), "send_denied": Rule(("sending",), ("manual",), "W"), "send_forum": Rule(("sending",), ("manual",), "W"), "send_refused": Rule(("sending",), ("failed",), "W"), "sending_stale": Rule(("sending",), ("awaiting_confirmation",), "W"), "restart": Rule(("sending",), ("awaiting_confirmation",), "R"), # The user. "set_url": Rule((*_WAITING, "review", "manual", "failed"), EVALUATED, "U"), "approve": Rule(("review",), ("ready", "manual"), "U"), "submit": Rule(("ready",), ("ready", "manual"), "U", _held, ("hold",)), "user_submitted": Rule(("manual", "ready", "review", "failed", "awaiting_confirmation"), ("sent", "awaiting_confirmation"), "U"), "confirmed": Rule(("sent", "awaiting_confirmation", "manual"), ("confirmed",), "U"), "rejected": Rule(("sent", "awaiting_confirmation"), ("rejected",), "U"), "retry": Rule(("failed",), ("ready",), "U"), "send_again": Rule(("awaiting_confirmation",), ("ready",), "U", _not_by_user, ("sent_by",)), "cancel": Rule(_CANCELLABLE, ("cancelled",), "U"), "move": Rule(_EDITABLE, ("cancelled",), "U"), "edit": Rule(_EDITABLE, (SAME, *EVALUATED), "U"), "campaign_changed": Rule(_EDITABLE, (SAME, *EVALUATED), "U", _not_opened, ("opened_at",)), # An explicit re-attach by the user (never automation). "reopen": Rule(("cancelled", "expired", "failed"), ("waiting_publication",), "A"), } #: The reason shown after an event, when the caller gives none. REASONS = { "post_failed": "The post failed: publish it again or enter its link", "post_skipped": "Nothing was posted on this platform: publish it there or enter its link", "url_overdue": "No link for this post a day after it went live: enter the post's link", "stale": "Not seen live 14 days after its time: enter its link if it went out", "expire": "The campaign's deadline passed", "destination_unusable": "The webhook can't be used: submit it by hand or pick another", "send_ambiguous": "Check Discord: it may have been posted", "sending_stale": "Check Discord: it may have been posted", "restart": "Clips Kitty closed while sending. Check the channel.", "send_gone": "The webhook is gone: submit it by hand or pick another", "send_denied": "The webhook isn't allowed to post there any more", "send_forum": "This channel is a forum: set Forum on the destination", } def refusal(row, event: str, *, now=None) -> str: """Why `event` can't happen to the row as it is; '' when it can.""" rule = TRANSITIONS.get(event) if rule is None: raise ValueError(f"unknown submission event: {event}") row = dict(row) status = row.get("status", "") if status not in rule.sources: return f"Not possible while it is {status.replace('_', ' ')}" if rule.guard is not None: return rule.guard(row, posts.as_utc(now).timestamp()) return "" def allowed(row, event: str, *, now=None) -> bool: return not refusal(row, event, now=now) def _approval_required(db, row: dict) -> bool: campaign = db.get_campaign(int(row["campaign_id"])) return bool(int(cell(campaign, "approval_required", 1))) if campaign is not None else True def _target(db, row: dict, event: str, rule: Rule, to: str) -> str: status = row["status"] if to: if to == status and SAME in rule.targets: return to if to not in rule.targets or to == SAME: raise NotAllowed(f"{event} can't make it {to}") return to if event in ("approve", "submit"): return "ready" if row.get("method") == "discord_webhook" else "manual" if event in ("send_ok", "user_submitted"): return "awaiting_confirmation" if _approval_required(db, row) else "sent" if rule.targets[0] == SAME: return status if event in ("url_found", "set_url"): raise ValueError(f"{event} needs to=: the evaluation decides where it goes") return rule.targets[0] def _defaults(event: str, row: dict, target: str, now) -> dict: moment = posts.as_utc(now) stamp, seconds = utc_text(moment), moment.timestamp() out: dict = {} if target != row["status"] or event not in ("edit", "campaign_changed"): out["status_reason"] = REASONS.get(event, "") if event == "published_no_url" and not row.get("posted_at"): out["posted_at"] = stamp elif event == "submit": out["hold"] = 0 if target == "ready": out["next_attempt_at"] = seconds elif event == "approve": if target == "ready": out["next_attempt_at"] = seconds elif event == "send_start": out["sending_started_at"] = seconds elif event == "send_ok": out.update(sent_by="webhook", sent_at=stamp, error="") elif event in ("send_ambiguous", "sending_stale", "restart"): out["sent_by"] = "" elif event == "user_submitted": out.update(sent_by="user", sent_at=stamp) elif event == "confirmed": out["confirmed_at"] = stamp elif event == "retry": out.update(attempts=0, next_attempt_at=seconds, error="") elif event == "send_again": out.update(next_attempt_at=seconds, sent_by="", error="") elif event == "reopen": # A fresh start on the same post: what the user set on the row stays # (message, values, method), what happened to it goes. Its link goes # too and is found again, so it is checked against the campaign's # other rows like any new link. Nothing can find a link the user # typed, so reopen through queue.reopen, which gives it back. out.update( attempts=0, error="", next_attempt_at=seconds, post_url="", url_source="", checks="[]", sending_started_at=0, sent_at="", sent_by="", discord_message_id="", discord_channel_id="", opened_at="", confirmed_at="", confirmation_note="", ) return out def apply(db, sub_id: int, event: str, *, now, to: str = "", **fields): """Make `event` happen to submission `sub_id` and return the row after it. Raises NotAllowed when its status (or the row) can't take the event, and Conflict when it changed while this ran. `fields` are written with the status, over the event's own defaults (a reason, a timestamp, a counter reset); the status itself comes only from the event and `to`. """ if "status" in fields: raise ValueError("the status comes from the event and to=, not from fields") rule = TRANSITIONS.get(event) if rule is None: raise ValueError(f"unknown submission event: {event}") found = db.get_submission(sub_id) if found is None: raise NotAllowed(f"Submission #{sub_id} doesn't exist") row = dict(found) problem = refusal(row, event, now=now) if problem: raise NotAllowed(problem) target = _target(db, row, event, rule, to) changes = {**_defaults(event, row, target, now), **fields, "status": target} match = {column: row[column] for column in rule.watch} if not db.set_submission(sub_id, expect=(row["status"],), match=match, **changes): raise Conflict(f"Submission #{sub_id} changed meanwhile") return db.get_submission(sub_id) - campaigns/urls.py (16307 bytes)
"""A submission's public link: where it comes from, and checking one. Nothing is ever invented. A link comes from a provider's row (WoopSocial or Upload-Post), from a YouTube video Clips Kitty uploaded itself, or from the user. When none of those has one yet, the answer says why and the submission keeps waiting. """ import re from dataclasses import dataclass from datetime import datetime from urllib.parse import parse_qs, urlsplit from campaigns.model import cell, platform_label, utc_text from server import posts # Hosts per platform: which platform a link is on. _HOSTS = ( ("tiktok", r"(?:www\.|m\.|vm\.|vt\.)?tiktok\.com"), ("instagram", r"(?:www\.)?instagram\.com"), ("youtube", r"(?:www\.|m\.)?youtube\.com|(?:www\.)?youtu\.be"), ("facebook", r"(?:www\.|m\.|web\.)?facebook\.com|fb\.watch"), ("x", r"(?:www\.)?(?:x|twitter)\.com"), ("threads", r"(?:www\.)?threads\.(?:net|com)"), ("linkedin", r"(?:www\.)?linkedin\.com"), ("pinterest", r"(?:[a-z]{2}\.|www\.)?pinterest\.[a-z.]{2,6}"), ("bluesky", r"(?:www\.)?bsky\.app"), ("reddit", r"(?:www\.)?reddit\.com"), ) _HOST_RES = tuple((platform, re.compile(rf"^(?:{hosts})$")) for platform, hosts in _HOSTS) # What a post's link looks like (host and path, plus ?v= for a YouTube watch # page). Threads to Reddit are checked by host only: their paths change shape # without notice, as ui/src/main/index.ts EXTERNAL_ALLOWED says. _POSTS = { "tiktok": (r"(?:www\.|m\.)?tiktok\.com/@[\w.]+/video/\d+",), "instagram": (r"(?:www\.)?instagram\.com/(?:reel|reels|p|tv)/[\w-]+",), "youtube": ( r"(?:www\.|m\.)?youtube\.com/shorts/[\w-]{11}", r"(?:www\.|m\.)?youtube\.com/watch\?v=[\w-]{11}", r"(?:www\.)?youtu\.be/[\w-]{11}", ), "facebook": (r"(?:www\.|m\.|web\.)?facebook\.com/.+", r"fb\.watch/.+"), "x": (r"(?:www\.)?(?:x|twitter)\.com/\w+/status/\d+",), "threads": (r"(?:www\.)?threads\.(?:net|com)/.+",), "linkedin": (r"(?:www\.)?linkedin\.com/.+",), "pinterest": (r"(?:[a-z]{2}\.|www\.)?pinterest\.[a-z.]{2,6}/.+",), "bluesky": (r"(?:www\.)?bsky\.app/.+",), "reddit": (r"(?:www\.)?reddit\.com/.+",), } _POST_RES = {p: tuple(re.compile(rf"^{r}/?$") for r in rs) for p, rs in _POSTS.items()} # One spelling per post, so the same post typed or returned two ways is one # link to the campaign's duplicate check: the host every one of these opens # on (ui/src/main/index.ts EXTERNAL_ALLOWED), and no trailing slash. _CANONICAL_HOSTS = { "tiktok.com": "www.tiktok.com", "m.tiktok.com": "www.tiktok.com", "youtube.com": "www.youtube.com", "m.youtube.com": "www.youtube.com", "www.youtu.be": "youtu.be", "instagram.com": "www.instagram.com", "facebook.com": "www.facebook.com", "m.facebook.com": "www.facebook.com", "web.facebook.com": "www.facebook.com", } _TIKTOK_SHORT = re.compile(r"^(?:vm|vt)\.tiktok\.com/[\w-]+/?$") _TIKTOK_HANDLE = re.compile(r"tiktok\.com/(@[\w.]+)/video/", re.IGNORECASE) SHORT_LINK_WARNING = "This is a short link: some campaigns want the full one" BLOCKED_PROVIDER = ("The link the publisher returned doesn't look like a public post: " "check it and enter it by hand") PRIVATE = "This video is private on YouTube" STILL_PRIVATE = ("Still not public on YouTube a day after its scheduled time: " "check it, then enter its link") UPLOAD_GONE = ("Clips Kitty no longer has this YouTube upload's record (its clip was deleted): " "check the link, then confirm it") @dataclass class Resolution: state: str # not_published | published_no_url | live | blocked url: str = "" post_id: str = "" account: str = "" caption: str = "" posted_at: str = "" # UTC; known for a direct YouTube upload only source: str = "" # provider | youtube | manual provider: str = "" # of the row found; copied onto the submission visibility: str = "" # clip_publishes.visibility, or the YouTube effective privacy due_at: str = "" # the followed post's due time (UTC) outcome: str = "" # "" | failed | skipped failed_at: str = "" # when the failure was recorded (UTC); a re-send restamps it reason: str = "" # shown as status_reason # ---- checking a link --------------------------------------------------------------------- def platform_of(url: str) -> str | None: """The platform a link is on, from its host; None when it is on none of them.""" try: host = (urlsplit(str(url or "").strip()).hostname or "").lower() except ValueError: return None for platform, pattern in _HOST_RES: if pattern.match(host): return platform return None def validate(platform: str, url: str) -> tuple[str, str]: """(normalised_url, problem). The problem is '' when the link is a public post's link on `platform`; it is then given back in one spelling: its query and fragment removed (except a YouTube watch page's ?v=), no trailing slash, and the www. host for TikTok, YouTube, Instagram and Facebook. https only.""" text = str(url or "").strip() if not text: return "", "Enter the post's link" try: parts = urlsplit(text) host = (parts.hostname or "").lower() port = parts.port except ValueError: return "", "That isn't a link" if parts.scheme.lower() != "https": return "", "Paste the full link, starting with https://" if not host or port is not None or "@" in parts.netloc or re.search(r"\s", text): return "", "That isn't a link to a post" found = platform_of(text) if found is None: return "", "Clips Kitty can't tell which platform that link is on" if found != platform: return "", f"That's a {platform_label(found)} link, not a {platform_label(platform)} one" host = _CANONICAL_HOSTS.get(host, host) path = parts.path.rstrip("/") query = "" if platform == "youtube" and path == "/watch": video = (parse_qs(parts.query).get("v") or [""])[0] query = f"?v={video}" normalised = f"https://{host}{path}{query}" shape = f"{host}{path}{query}" if platform == "tiktok" and _TIKTOK_SHORT.match(shape): return normalised, "" if not any(p.match(shape) for p in _POST_RES.get(platform, ())): return "", f"That doesn't look like the link of a {platform_label(platform)} post" return normalised, "" def link_warning(platform: str, url: str) -> str: """A link that passes but may not be what a campaign wants: a TikTok short link (vm./vt.tiktok.com) instead of the full one.""" try: host = (urlsplit(str(url or "").strip()).hostname or "").lower() except ValueError: return "" if platform == "tiktok" and host in ("vm.tiktok.com", "vt.tiktok.com"): return SHORT_LINK_WARNING return "" def account_from_url(platform: str, url: str) -> str: """The posting account a link shows: TikTok's "@handle" only.""" if platform != "tiktok": return "" found = _TIKTOK_HANDLE.search(str(url or "")) return found.group(1) if found else "" # ---- finding a submission's link ------------------------------------------------------------- def _local_utc(text) -> str: """created_at / updated_at / uploaded_at (local time, no zone) as UTC text; a value with a zone is read with it.""" try: moment = datetime.fromisoformat(str(text or "").strip().replace("Z", "+00:00")) except ValueError: return "" if moment.tzinfo is None: moment = moment.astimezone() return utc_text(moment) def resolve(db, sub, *, now=None) -> Resolution: """Where the post a submission follows stands now, and its link if it has one. `sub` is a campaign_submissions row (or its dict). A YouTube post that failed (or was skipped) on its route is looked for on the other one too: a direct upload made since a WoopSocial or Upload-Post post failed, or a publisher's post published since a direct upload failed, is the post (the clip published again, with or without the campaign picked). Its provider is the one copied onto the submission.""" sub = dict(sub) provider = str(sub.get("provider") or "") if provider == "manual": return _manual(sub) if provider == "youtube": res = _youtube(db, sub, now) else: row = _provider_row(db, sub) if row is not None: res = _from_provider_row(row, sub, now) elif not provider and sub.get("platform") == "youtube": res = _youtube(db, sub, now) else: return Resolution("not_published", provider=provider) if res.outcome and sub.get("platform") == "youtube": return _other_route(db, sub, res, now) or res return res def _other_route(db, sub: dict, failed: Resolution, now) -> Resolution | None: """The YouTube post made through the other route at or after `failed` (a failed or skipped one), or None when there is none.""" since = failed.failed_at if not since: return None if failed.provider == "youtube": row = _provider_row(db, sub) if row is None or row["state"] != "published" or _local_utc(row["updated_at"]) < since: return None return _from_provider_row(row, sub, now) upload = _upload(db, sub, int(sub.get("clip_id") or 0)) if upload is None or _local_utc(upload["uploaded_at"]) < since: return None found = _youtube(db, {**sub, "provider": "youtube", "publish_job_id": 0}, now) return None if found.outcome else found def _manual(sub: dict) -> Resolution: url = str(sub.get("post_url") or "") if not url: # Only the user can give this one its link, so it goes to them now # rather than waiting for a publisher that will never answer. return Resolution("blocked", provider="manual", source="manual", reason="Enter the post's link") normalised, problem = validate(sub["platform"], url) if problem: return Resolution("blocked", url=url, provider="manual", source="manual", reason=problem) return Resolution("live", url=normalised, provider="manual", source="manual", post_id=str(sub.get("post_id") or ""), account=account_from_url(sub["platform"], normalised)) def _provider_row(db, sub: dict): """The clip's row for this platform, by id or by the post's window, the published one first, then the newest. No provider filter: a clip has one row per platform, and publishing it again through the other provider overwrites that row's provider.""" return db.conn.execute( "SELECT * FROM clip_publishes WHERE platform = ? AND (clip_id = ? OR " "(video_id != '' AND video_id = ? AND start_s = ? AND end_s = ?)) " "ORDER BY (state = 'published') DESC, updated_at DESC LIMIT 1", (sub["platform"], int(sub.get("clip_id") or 0), sub["video_id"], round(float(sub["start_s"]), 2), round(float(sub["end_s"]), 2)), ).fetchone() def _from_provider_row(row, sub: dict, now) -> Resolution: base = { "provider": str(row["provider"] or sub.get("provider") or ""), "post_id": str(row["post_id"] or ""), "caption": str(cell(row, "caption", "")), "visibility": str(cell(row, "visibility", "")), "due_at": utc_text(posts.due_time(row, now=now)), } state = row["state"] if state == "published": url = str(row["post_url"] or "") if not url: return Resolution("published_no_url", **base) normalised, problem = validate(sub["platform"], url) if problem: return Resolution("blocked", url=url, source="provider", reason=BLOCKED_PROVIDER, **base) return Resolution("live", url=normalised, source="provider", account=account_from_url(sub["platform"], normalised), **base) if state in ("failed", "skipped"): return Resolution("not_published", outcome=state, failed_at=_local_utc(row["updated_at"]), reason=str(row["error"] or ""), **base) # queued | processing | sending: still on its way. return Resolution("not_published", **base) def _upload(db, sub: dict, clip_id: int): return db.conn.execute( "SELECT * FROM uploads WHERE youtube_id != '' AND (clip_id = ? OR " "(video_id != '' AND video_id = ? AND start_s = ? AND end_s = ?)) " "ORDER BY (clip_id = ?) DESC, uploaded_at DESC LIMIT 1", (clip_id, sub["video_id"], round(float(sub["start_s"]), 2), round(float(sub["end_s"]), 2), clip_id), ).fetchone() def _youtube(db, sub: dict, now) -> Resolution: """A direct YouTube upload: through its publish job while that runs, then its uploads row read as posts.youtube_view sees it. After a job that failed, an upload made since (the clip published again, with or without the campaign picked) is the post; without one, the post failed.""" clip_id = int(sub.get("clip_id") or 0) job_id = int(sub.get("publish_job_id") or 0) job = db.get_publish_job(job_id) if job_id else None upload = None if job is not None: status = job["status"] if status in ("queued", "running"): return Resolution("not_published", provider="youtube") if status in ("failed", "cancelled", "interrupted"): ended = _local_utc(job["updated_at"]) upload = _upload(db, sub, clip_id) if upload is None or not ended or _local_utc(upload["uploaded_at"]) < ended: return Resolution("not_published", provider="youtube", outcome="failed", failed_at=ended, reason=str(job["error"] or "") or f"The upload {status}.") else: # done: _publish keeps the job's clip_id on the clip it uploaded, # which a re-render made just before may have changed. clip_id = int(job["clip_id"] or clip_id) if upload is None: upload = _upload(db, sub, clip_id) if upload is None: post_id = str(sub.get("post_id") or "") if post_id: url = posts.youtube_url({"youtube_id": post_id, "start_s": sub["start_s"], "end_s": sub["end_s"]}) return Resolution("blocked", url=url, post_id=post_id, provider="youtube", source="youtube", reason=UPLOAD_GONE) return Resolution("not_published", provider="youtube") clip = db.get_clip(int(upload["clip_id"])) seen = posts.youtube_view(upload, clip, now) scheduled = str(upload["publish_at"] or "") actual = str(upload["actual_privacy"] or "").lower() asked = str(upload["privacy"] or "").lower() base = { "provider": "youtube", "post_id": str(upload["youtube_id"]), "account": str(upload["channel_title"] or ""), "caption": str( - campaigns/views.py (9151 bytes)
"""What the renderer is shown of campaigns and their submissions (DESIGN §7 "Submission JSON" and "Campaign JSON"), and the `campaign` event the background step sends on the existing /ws. Plain Python with no FastAPI, so CI's main job tests the shapes. Nothing here holds a token: a destination is shown by its id, label and state only. """ from campaigns import formatter, links from campaigns.model import ( LABELS, cell, json_field, outcome_to_record, platform_label, status_label, to_submit, ) #: The campaign columns kept as JSON text, decoded for the renderer. CAMPAIGN_JSON = { "platforms": [], "required_accounts": {}, "required_hashtags": [], "required_mentions": [], "command_args": [], "extra_fields": [], "filters": {}, } #: The campaign columns that are switches (0/1 in the table). CAMPAIGN_FLAGS = ("enabled", "archived", "trust_account", "approval_required", "review_first") def discord_user(db) -> str: """The connected Discord username for {discord_user}, or ''.""" from campaigns import service user = service.load_settings(db).get("user") return str(user.get("username") or "") if isinstance(user, dict) else "" def counts_of(rows) -> dict: """{"counts": {status: n}, "to_submit": n, "outcome_to_record": n} from StateDB.submission_counts rows.""" counts: dict[str, int] = {} todo = outcomes = 0 for row in rows: n = int(row["n"]) counts[row["status"]] = counts.get(row["status"], 0) + n sample = {"status": row["status"], "hold": row["hold"], "sent_by": row["sent_by"]} todo += n if to_submit(sample) else 0 outcomes += n if outcome_to_record(sample) else 0 return {"counts": counts, "to_submit": todo, "outcome_to_record": outcomes} def campaign_view(db, row, *, counts: dict | None = None) -> dict: """A campaign as the renderer gets it: every column, its JSON columns decoded and its switches as true/false, plus its webhook destination (id, label, state) and how many of its submissions are in each status.""" out = dict(row) for name, default in CAMPAIGN_JSON.items(): out[name] = json_field(out.get(name), default) for name in CAMPAIGN_FLAGS: out[name] = bool(int(out.get(name) or 0)) dest = None dest_id = int(out.get("destination_id") or 0) if dest_id: found = db.get_destination(dest_id) dest = {"id": dest_id, "label": found["label"], "state": found["state"]} if found is not None \ else {"id": dest_id, "label": "", "state": "missing"} out["destination"] = dest if counts is None: counts = counts_of(db.submission_counts(campaign_id=int(out["id"])))["counts"] out["counts"] = counts return out def submission_view(db, row, *, campaign=None, user: str | None = None) -> dict: """A submission as DESIGN §7 "Submission JSON" lists it: its fields, the owner's wording for its status, the words to submit (the message or the command, with per-row overrides applied) and the links to open.""" sub = dict(row) if campaign is None: campaign = db.get_campaign(int(sub["campaign_id"])) campaign = dict(campaign) if campaign is not None else {} if user is None: user = discord_user(db) arg_values = json_field(sub.get("arg_values"), {}) extra_values = json_field(sub.get("extra_values"), {}) if campaign: values = formatter.values_for(db, sub, campaign, discord_user=user) command = formatter.command(campaign, values, arg_values=arg_values) assisted = formatter.assisted_text(campaign, values, message_text=str(sub.get("message_text") or ""), arg_values=arg_values) else: command = {"name": "", "args": [], "url_arg": "", "line": ""} assisted = "" message_url = "" if sub.get("discord_message_id"): guild = str(campaign.get("guild_id") or "") dest_id = int(sub.get("destination_id") or 0) if dest_id: dest = db.get_destination(dest_id) guild = str(cell(dest, "guild_id", "")) if dest is not None and cell(dest, "guild_id", "") else guild message_url = links.channel_url(guild, str(sub.get("discord_channel_id") or ""), str(sub["discord_message_id"])) return { "id": int(sub["id"]), "campaign_id": int(sub["campaign_id"]), "campaign_name": str(campaign.get("name") or ""), "clip_id": int(sub.get("clip_id") or 0), "clip_title": str(sub.get("clip_title") or ""), "video_id": sub["video_id"], "start_s": float(sub["start_s"]), "end_s": float(sub["end_s"]), "platform": sub["platform"], "provider": str(sub.get("provider") or ""), "method": str(sub.get("method") or ""), "overridden": json_field(sub.get("overridden"), []), "status": sub["status"], "status_label": status_label(sub), "status_reason": str(sub.get("status_reason") or ""), "to_submit": to_submit(sub), "outcome_to_record": outcome_to_record(sub), "uncertain": sub["status"] == "awaiting_confirmation" and not sub.get("sent_by"), "hold": bool(int(sub.get("hold") or 0)), "review_first": bool(int(sub.get("review_first") or 0)), "due_at": str(sub.get("due_at") or ""), "post_url": str(sub.get("post_url") or ""), "url_source": str(sub.get("url_source") or ""), "account": str(sub.get("account") or ""), "checks": json_field(sub.get("checks"), []), "message_text": str(sub.get("message_text") or ""), "arg_values": arg_values, "extra_values": extra_values, "assisted_text": assisted, "command": {"name": command["name"], "line": command["line"], "args": command["args"], "url_arg": command["url_arg"]}, "discord_url": links.channel_url(str(campaign.get("guild_id") or ""), str(campaign.get("channel_id") or "")), "message_url": message_url, "website_url": str(campaign.get("website_url") or ""), "deadline": str(campaign.get("deadline") or ""), "attempts": int(sub.get("attempts") or 0), "next_attempt_at": float(sub.get("next_attempt_at") or 0), "opened_at": str(sub.get("opened_at") or ""), "sent_at": str(sub.get("sent_at") or ""), "sent_by": str(sub.get("sent_by") or ""), "confirmed_at": str(sub.get("confirmed_at") or ""), "confirmation_note": str(sub.get("confirmation_note") or ""), "error": str(sub.get("error") or ""), "created_at": sub["created_at"], "updated_at": sub["updated_at"], } # ---- the `campaign` event ---------------------------------------------------------------- def event_message(row, campaign_name: str) -> str: """One line for a status a submission just reached.""" sub = dict(row) name = campaign_name or "the campaign" status = sub["status"] method = str(sub.get("method") or "") if status == "manual": if sub.get("post_url") and method == "discord_assisted": return f"Ready to submit to {name} in Discord" if sub.get("post_url") and method == "website": return f"Ready to submit to {name} on its page" return f"{name}: {sub.get('status_reason') or 'it needs you'}" if status == "review": return f"Check a {platform_label(sub['platform'])} clip for {name} before it is submitted" if status == "ready": return f"Saved for later for {name}" if int(sub.get("hold") or 0) else f"Ready to post to {name}" if status == "sent": return f"Posted to {name} in Discord" if status == "awaiting_confirmation": if not sub.get("sent_by"): return f"Check Discord: a clip for {name} may already be posted" return f"Posted to {name}: record what the campaign says" if status == "failed": return f"Couldn't post a clip to {name}: {sub.get('error') or 'retry or submit it by hand'}" if status == "expired": return f"{name}'s deadline passed" return f"{name}: {LABELS.get(status, status)}" def event(row, previous: str, campaign_name: str) -> dict: """The `campaign` WebSocket event for a submission that changed status.""" sub = dict(row) return { "type": "campaign", "submission_id": int(sub["id"]), "campaign_id": int(sub["campaign_id"]), "clip_id": int(sub.get("clip_id") or 0), "status": sub["status"], "previous": previous, "to_submit": to_submit(sub), "message": event_message(sub, campaign_name), } #: Statuses a list's group asks for (filtered again by the group's own rule). GROUP_STATUSES = { "to_submit": ("ready", "review", "manual", "failed", "awaiting_confirmation"), "outcome_to_record": ("awaiting_confirmation",), "waiting": ("waiting_publication", "waiting_url", "ready", "sending"), "sent": ("sent",), "done": ("confirmed", "rejected", "expired", "cancelled"), } - CHANGELOG.md (81629 bytes)
# Changelog What changed, written for people who use Clips Studio rather than people who read the commits. Dates are release dates. Clips Kitty is a stable release, and new versions come often. Things listed as fixed were often broken in a way that only showed up on somebody else's machine. --- ## Unreleased ### Added - **Optional Clip updates.** App-wide Approve/Deny cards, private Discord updates and an Away from desk approval gate, with local-model wording and plain-text fallback. Includes a configurable attorney joke popup. Off by default. - **Connect your Twitch account** (new). In Settings → Connected accounts you register a free Twitch application once (the card lists the steps), paste its Client ID, then sign in on Twitch's own page with a code. No Clips Kitty account is needed, and the sign-in is kept encrypted on this PC. Twitch is asked only for the permissions you tick, and none is ticked to start with. An account that needs signing in again shows Reconnect, which asks for the same permissions. - **Watch my channel.** Adds your connected channel to Watched channels with the same choices as Add a channel (from Settings: your Generate settings, and clips wait for you). Its new videos are then found through your connection, and through the public video list, as before, whenever the connection needs reconnecting or Twitch doesn't answer, so the watch keeps going. If you disable or uninstall a Gaming detector while the Watch form is open, it returns to **All games (Clips Kitty)** so adding a channel still works. - **Viewer nominations.** With the chat permission ticked you can start and stop collecting !clip, !chaos and !funny from your channel or a channel you watch, with command counts and a timeline for each stream. After a broadcast ends, Clips Kitty looks for its matching Twitch recording and keeps checking for up to six hours, including across restarts. In Connected accounts, use **Bind recording** to check the recording against your stream clock and confirm the offset. The complete downloaded recording can then use those mapped nominations. These workflows still need live-account verification. - **For people who make pipelines:** a pipeline you allow can read your channel's Twitch details, link to a moment of the Twitch VOD it is clipping, and make a Twitch clip of that moment, all through Clips Kitty and never with your Twitch sign-in. Links and clips work when the video is the whole Twitch broadcast, as Watched channels downloads it. A pipeline can't clip your live stream. ### Changed - **The Clips Kitty end card is now off by default.** Clips no longer end with the 2.9-second card unless you want it. If it was on, this update switches it off once. To add it again, tick **Add the Clips Kitty end card to every clip** in Settings › Branding, and it stays on. - Plugin local-model requests now share their requested timeout between checking model capabilities and generating an answer. A capability check that uses the budget no longer starts another full-length request. - Plugins can reuse validated results in an app-managed cache without changing their installed files, and can see the runner's actual deadline to finish optional review before it expires. Unavailable cache storage does not stop a run. Clips are framed and chosen in two new ways, on for everyone. Either can be turned off with one line in your settings file: `%LOCALAPPDATA%\Clips Studio\settings.yaml` in the installed app (`config/settings.yaml` if you run Clips Kitty from its code). Restart Clips Kitty after you change it. - **Framing follows the cut, and a clip opens on the person's face.** In an edited video, a new shot could open framed on where the last shot's person stood, and move to the new person only about a second later, and many cuts were missed or caught a few frames late. Now cuts are found to the frame, and the crop moves to the person in the new shot on the cut itself. A clip usually opens on the person's face, where it used to open on their body and slide over to the face during the first second. A shot with nobody in it opens on the centre (or keeps the last framing, with `shot_aware_no_subject: hold` under `tracking:`) until someone clearly visible appears. Framing takes about a third longer on a video with many cuts, and a little longer on handheld footage. Only the standard crop changes: Podcast (which has its own entry below), Gaming / Reaction layouts and Sports frame their own way. To frame as before, add the line `shot_aware: false` under `tracking:`. - **Clip selection 2.** The moments found from the sound, the picture and chat are scored by your AI model eight at a time, so they no longer all fall back to a flat 50 when a local model runs out of room scoring them at once. A long video's later moments are no longer crowded out by its first half, and a transcript pick scored below your minimum score no longer hides a moment it overlaps. A clip's start no longer skips past the line it was picked for, and a clip that would start partway into a sentence goes back to where the sentence begins, up to 8 seconds earlier. Videos scored as gaming (Gaming / Reaction, or Vertical Live's Gaming / reaction choice) and Sports matches choose clips as before, and so does Longform unless you add `longform_selection: 2` under `scoring:`. To choose clips as before, add the line `selection: 1` under `scoring:`. - Podcast mode now frames whoever is talking in each shot (not the person moving their mouth most), follows a new speaker in a long shot, and snaps on every camera cut. ### Fixed - **A video added with "Upload video file" now keeps every setting you chose for it.** The chosen Gaming detector or Marketplace pipeline, Sports, the clip direction, Gaming scoring and the Rate, Understand and Suggest edits plugins were left out for uploaded files, so they ran with Clips Kitty's own scoring instead. Pasted links were not affected. - **Longform no longer fails with "concat join failed" when a video's title has an apostrophe** (#197), as in "I'll" or "Let's". Clips of such a video also keep their end card, which they had quietly been made without, and its story reels join again. - **A clip no longer plays for a few seconds while its card says a minute** (#213). When the graphics card stopped decoding a clip partway, the rest was dropped and the clip was saved anyway. It now finishes on the processor, and a clip that still comes out much shorter than planned is reported as failed instead of saved. - **"Use standard processing" on a gaming video that Vertical Live turned away now makes it with Gaming / Reaction,** the split-screen layout. It used to score the clips as a game but frame them with the plain crop, facecam and all. - **A bug report sent from the app now describes the video you picked in it,** not whichever video was processed last. - **Basketball clips that cut to a close-up no longer start the close-up off-centre and jump onto the player.** A close-up that moved fast from its first frame was taken for a fade, and the crop kept the wide shot's framing over it until the "fade" ended. - A basketball close-up no longer slides off the player onto a fan or the empty seats at the side of the picture. - **Clips no longer open with a black moment on Reels, TikTok, Shorts and other apps,** and story reels no longer flash a frame of the end card between moments. - **In Soccer clips the crop now jumps to the new shot on the cut instead of sliding across it**, a cut straight out of a fast pan included, and on a fade into a replay it jumps as the fade completes. A fast pan is no longer taken for a cut, which made the crop jump and start looking for the ball again. Each shot also opens on the play, where it could open beside it and slide over during its first seconds. --- ## 2.1.0: Basketball, and a Marketplace for new pipelines Basketball joins Soccer under Sports, and the new Marketplace installs pipelines and plugins that other people build for Clips Kitty. This release also fixes model downloads that stopped at 100%, new Kick VODs that read as deleted, and caption corrections in other languages. ### Added - **YouTube cookies, for when YouTube keeps saying "confirm you're not a bot".** Under **Settings → Advanced settings**, choose **From Firefox** (signed in to YouTube there) or a cookies.txt file exported from a browser signed in to YouTube. The file is copied into Clips Kitty's data folder, so moving the original doesn't break it, and turning the setting off deletes that copy. Clips Kitty uses it only when YouTube has refused a download both the ordinary way and over IPv4; every other download stays signed out. Signed in, YouTube's player needs its JavaScript run, so the app now carries QuickJS (2 MB) for that one request. Chrome and Edge lock their cookies on Windows, so they are not offered. (#136) - **A Marketplace, and Awesome Clips Kitty.** Other developers can now write their own "find the best moments" step (a pipeline) and list it; you install one from the new **Marketplace** screen and switch it on for a video with the **Pipeline** option. Every listing says what it may do, where it runs, what it needs and its licence, and asks you to confirm before anything is installed. Nothing changes until you install something. The Marketplace also shows **Awesome Clips Kitty**, the list of integrations, apps and tools that other people have built with Clips Kitty. It lists only what runs in Clips Kitty or uses it: a link to another project is not an entry. The list is its own repository, github.com/ColinGPT9/awesome-clips-kitty, where a developer adds a pipeline with a pull request of one file. Labels: ✓ Official (made by this project), ✓ Compatible (passed automated checks; not a security review), ★ Featured and Community. Installs from the Marketplace can be counted anonymously, with a switch to turn it off; nothing is counted yet. The developer kit is now MIT-licensed; the app stays AGPL-3.0-or-later. (Experimental routes: `/plugins`, `/marketplace`, `/marketplace/catalog`, `/marketplace/counting`, `/plugin-models`; a plan from `POST /plugins/plan` carries the install screen as `text`.) (#129) - **New pipelines without waiting for an update, and nothing else to install.** The Marketplace checks Clips Kitty's online list when it opens, at most once a day, so a newly listed pipeline or a new version shows up without a new version of the app; **Check for new pipelines** checks at once. With a pipeline installed, the app also checks when it starts, so a warning about a harmful version arrives without opening the Marketplace. Until a release includes it, what the list adds shows as Community. The check fetches one file from GitHub and sends nothing about you or what you browse; a switch at the bottom of Browse turns it off. Pipelines also run on the Python inside Clips Kitty now, so a creator never installs Python for one. (Experimental routes: `/marketplace/online`; `/marketplace/refresh` now needs the desktop app.) - **The Marketplace in plain words.** Browse and Installed are the only tabs: installing a pipeline you're writing, from a folder or a link, is now **For developers**, a link at the bottom of Browse. The Marketplace and the install screen say what a pipeline is, where it comes from and what it may do in everyday words, and keep the exact details one click away under **Technical details**. What you tick before installing, the warning that a pipeline can do anything you can do on your PC, and what "Clips Kitty hands this over" and "the developer says so" mean are in all 18 of the app's other languages. A download that fails says why in one sentence (no connection, files no longer there, a full disk or a damaged download) and keeps the details in the log. Apps, tools and the other entries of Awesome Clips Kitty each get one button, to their download page or website where the list has one, which opens your browser after showing you the address; entries that need the command line or Python say so and come last. - **Rate & understand: Marketplace plugins that look at your moments.** Once you have installed a Marketplace plugin that can rate or understand moments, adding a video shows a new switch, **Rate & understand**. Clips Kitty (or Sports, Gaming scoring or a pipeline) still finds the moments. Then up to three plugins can say what happens in each one, and up to three can give each one a new score, in the order you choose. What a plugin says happens in a moment goes to the AI that writes the clip's title, description and hashtags. A plugin's score decides which clips are made and their order: the last plugin's score counts, and a moment it scores under your minimum score is set aside (a must-have never is). Clips Kitty does any step you leave to it, and a video without the switch is made exactly as before. Not with Longform. The clip editor shows each plugin's score and notes, and the video page says what each plugin did, or why Clips Kitty made the clips without it. The Marketplace marks what each plugin does (Finds moments, Understands moments, Rates moments), how long Clips Kitty lets it work on your moments (10 minutes unless the plugin says otherwise), and when an update starts doing something new. **A rating plugin decides what gets posted.** Where a watched channel or the daily schedule posts only the best few clips, the scores decide which ones go out, so a rating plugin decides that; elsewhere it decides the order. The job form, the channel's settings and the Marketplace say so. **A channel that posts by itself waits for you when a chosen plugin didn't run.** If a plugin chosen for a video couldn't run, Clips Kitty still makes the clips, without it. A watched channel that posts automatically then holds that video's clips for you to check and publish, and a channel that asks first says why. A job told to publish its clips when it finishes waits too. The command-line daily upload isn't held. In a channel's settings you can keep its plugins or turn them off; choosing them for a channel is done through the API for now. For developers: the plugin kit (SDK 1.1.0) adds `job.rate()` and `job.understand()`, with an example plugin, and [docs/developers/steps.md](docs/developers/steps.md) explains them. (New job options: `rate`, `understand`.) - **Suggest edits: Marketplace plugins that suggest edits for your clips.** Once you have installed a Marketplace plugin that can suggest edits, adding a video shows a new switch, **Suggest edits**. After the clips are chosen, up to three plugins, in the order you choose, can each suggest an edit for every clip: parts to cut, stretches to mute (their captions are hidden too), the volu
- CODE_OF_CONDUCT.md (5447 bytes)
# Contributor Covenant Code of Conduct ## Our Pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our Standards Examples of behavior that contributes to a positive environment for our community include: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences * Giving and gracefully accepting constructive feedback * Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience * Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: * The use of sexualized language or imagery, and sexual attention or advances of any kind * Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or email address, without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement through [GitHub's report abuse form](https://github.com/contact/report-abuse) or by opening a private report via the repository's [Security Advisories](../../security/advisories/new) if the matter is sensitive. All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ## Enforcement Guidelines Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: ### 1. Correction **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning **Community Impact**: A violation through a single incident or series of actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban **Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct.html. Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations. [homepage]: https://www.contributor-covenant.org [Mozilla CoC]: https://github.com/mozilla/diversity
- config/audioset_labels.txt (6337 bytes)
Speech Male speech, man speaking Female speech, woman speaking Child speech, kid speaking Conversation Narration, monologue Babbling Speech synthesizer Shout Bellow Whoop Yell Battle cry Children shouting Screaming Whispering Laughter Baby laughter Giggle Snicker Belly laugh Chuckle, chortle Crying, sobbing Baby cry, infant cry Whimper Wail, moan Sigh Singing Choir Yodeling Chant Mantra Male singing Female singing Child singing Synthetic singing Rapping Humming Groan Grunt Whistling Breathing Wheeze Snoring Gasp Pant Snort Cough Throat clearing Sneeze Sniff Run Shuffle Walk, footsteps Chewing, mastication Biting Gargling Stomach rumble Burping, eructation Hiccup Fart Hands Finger snapping Clapping Heart sounds, heartbeat Heart murmur Cheering Applause Chatter Crowd Hubbub, speech noise, speech babble Children playing Animal Domestic animals, pets Dog Bark Yip Howl Bow-wow Growling Whimper (dog) Cat Purr Meow Hiss Caterwaul Livestock, farm animals, working animals Horse Clip-clop Neigh, whinny Cattle, bovinae Moo Cowbell Pig Oink Goat Bleat Sheep Fowl Chicken, rooster Cluck Crowing, cock-a-doodle-doo Turkey Gobble Duck Quack Goose Honk Wild animals Roaring cats (lions, tigers) Roar Bird Bird vocalization, bird call, bird song Chirp, tweet Squawk Pigeon, dove Coo Crow Caw Owl Hoot Bird flight, flapping wings Canidae, dogs, wolves Rodents, rats, mice Mouse Patter Insect Cricket Mosquito Fly, housefly Buzz Bee, wasp, etc. Frog Croak Snake Rattle Whale vocalization Music Musical instrument Plucked string instrument Guitar Electric guitar Bass guitar Acoustic guitar Steel guitar, slide guitar Tapping (guitar technique) Strum Banjo Sitar Mandolin Zither Ukulele Keyboard (musical) Piano Electric piano Organ Electronic organ Hammond organ Synthesizer Sampler Harpsichord Percussion Drum kit Drum machine Drum Snare drum Rimshot Drum roll Bass drum Timpani Tabla Cymbal Hi-hat Wood block Tambourine Rattle (instrument) Maraca Gong Tubular bells Mallet percussion Marimba, xylophone Glockenspiel Vibraphone Steelpan Orchestra Brass instrument French horn Trumpet Trombone Bowed string instrument String section Violin, fiddle Pizzicato Cello Double bass Wind instrument, woodwind instrument Flute Saxophone Clarinet Harp Bell Church bell Jingle bell Bicycle bell Tuning fork Chime Wind chime Change ringing (campanology) Harmonica Accordion Bagpipes Didgeridoo Shofar Theremin Singing bowl Scratching (performance technique) Pop music Hip hop music Beatboxing Rock music Heavy metal Punk rock Grunge Progressive rock Rock and roll Psychedelic rock Rhythm and blues Soul music Reggae Country Swing music Bluegrass Funk Folk music Middle Eastern music Jazz Disco Classical music Opera Electronic music House music Techno Dubstep Drum and bass Electronica Electronic dance music Ambient music Trance music Music of Latin America Salsa music Flamenco Blues Music for children New-age music Vocal music A capella Music of Africa Afrobeat Christian music Gospel music Music of Asia Carnatic music Music of Bollywood Ska Traditional music Independent music Song Background music Theme music Jingle (music) Soundtrack music Lullaby Video game music Christmas music Dance music Wedding music Happy music Funny music Sad music Tender music Exciting music Angry music Scary music Wind Rustling leaves Wind noise (microphone) Thunderstorm Thunder Water Rain Raindrop Rain on surface Stream Waterfall Ocean Waves, surf Steam Gurgling Fire Crackle Vehicle Boat, Water vehicle Sailboat, sailing ship Rowboat, canoe, kayak Motorboat, speedboat Ship Motor vehicle (road) Car Vehicle horn, car horn, honking Toot Car alarm Power windows, electric windows Skidding Tire squeal Car passing by Race car, auto racing Truck Air brake Air horn, truck horn Reversing beeps Ice cream truck, ice cream van Bus Emergency vehicle Police car (siren) Ambulance (siren) Fire engine, fire truck (siren) Motorcycle Traffic noise, roadway noise Rail transport Train Train whistle Train horn Railroad car, train wagon Train wheels squealing Subway, metro, underground Aircraft Aircraft engine Jet engine Propeller, airscrew Helicopter Fixed-wing aircraft, airplane Bicycle Skateboard Engine Light engine (high frequency) Dental drill, dentist's drill Lawn mower Chainsaw Medium engine (mid frequency) Heavy engine (low frequency) Engine knocking Engine starting Idling Accelerating, revving, vroom Door Doorbell Ding-dong Sliding door Slam Knock Tap Squeak Cupboard open or close Drawer open or close Dishes, pots, and pans Cutlery, silverware Chopping (food) Frying (food) Microwave oven Blender Water tap, faucet Sink (filling or washing) Bathtub (filling or washing) Hair dryer Toilet flush Toothbrush Electric toothbrush Vacuum cleaner Zipper (clothing) Keys jangling Coin (dropping) Scissors Electric shaver, electric razor Shuffling cards Typing Typewriter Computer keyboard Writing Alarm Telephone Telephone bell ringing Ringtone Telephone dialing, DTMF Dial tone Busy signal Alarm clock Siren Civil defense siren Buzzer Smoke detector, smoke alarm Fire alarm Foghorn Whistle Steam whistle Mechanisms Ratchet, pawl Clock Tick Tick-tock Gears Pulleys Sewing machine Mechanical fan Air conditioning Cash register Printer Camera Single-lens reflex camera Tools Hammer Jackhammer Sawing Filing (rasp) Sanding Power tool Drill Explosion Gunshot, gunfire Machine gun Fusillade Artillery fire Cap gun Fireworks Firecracker Burst, pop Eruption Boom Wood Chop Splinter Crack Glass Chink, clink Shatter Liquid Splash, splatter Slosh Squish Drip Pour Trickle, dribble Gush Fill (with liquid) Spray Pump (liquid) Stir Boiling Sonar Arrow Whoosh, swoosh, swish Thump, thud Thunk Electronic tuner Effects unit Chorus effect Basketball bounce Bang Slap, smack Whack, thwack Smash, crash Breaking Bouncing Whip Flap Scratch Scrape Rub Roll Crushing Crumpling, crinkling Tearing Beep, bleep Ping Ding Clang Squeal Creak Rustle Whir Clatter Sizzle Clicking Clickety-clack Rumble Plop Jingle, tinkle Hum Zing Boing Crunch Silence Sine wave Harmonic Chirp tone Sound effect Pulse Inside, small room Inside, large room or hall Inside, public space Outside, urban or manmade Outside, rural or natural Reverberation Echo Noise Environmental noise Static Mains hum Distortion Sidetone Cacophony White noise Pink noise Throbbing Vibration Television Radio Field recording
- config/gaming.yaml (18991 bytes)
# What a gaming / reaction highlight looks like, for scoring (analysis/gaming.py, # analysis/chat_moments.py). Used only when a job scores as a gaming stream: # Gaming / Reaction, or "Gaming stream" with Vertical Live. # # Sources for the choices here are in docs/GAMING.md ("How game moments are # found"): stream-highlight research, what gaming clip tools look for, and # what performs as a Short. # ---- chat --------------------------------------------------------------------- # Chat reacts after the moment: the stream's delay plus the time to type. A # spike is dated this many seconds earlier. chat_lag_seconds: 6 # What chat says tells what kind of moment it was, in any language chat # types in. # - words: a whole word in a message, exactly as written (so "W" and "HOW" in # capitals, not "how do I"); entries with a space, or emoji, anywhere in it, # ignoring case. # - suffixes: the end of a channel's own emote, after its lowercase prefix # (kittyPog, kittyLUL, kittySad). # - patterns: regular expressions tried on each word, ignoring case (laughter # typed as www, ㅋㅋㅋ, jajaja, kkkk, 哈哈). # - message_patterns: the same, on the whole message (a lone "あ" or "what?!"). chat_classes: hype: label: hype words: [POG, POGGERS, PogU, PogChamp, Pog, POGGIES, PagMan, PagChomp, POGSLIDE, LETSGO, LETSGOOO, INSANE, insane, HOW, HOLY, CLUTCH, clutch, W, WW, GIGACHAD, EZ, Clap, GG, gg, Kreygasm, SeemsGood, CurseLit, PopNemo, WOW, wow, OMG, omg, "no way", "let's go", "lets go", "what a", "holy shit", "oh my god", "すご", "すげ", "神", "대박", "牛逼", "太强", "brutal", "increible", "incrível", "🔥", "😱", "🤯", "🐐", "👏", "💪", "🏆"] suffixes: [Pog, POG, Hype, HYPE, Clap, GG, Win, WIN, W, Cheer, Fire, Goat, GOAT, Wow, WOW] patterns: ["^6{3,}$", "^(gg)+$"] funny: label: laughing words: [LUL, LULW, KEKW, OMEGALUL, ICANT, KEK, LOL, lol, LMAO, lmao, LMFAO, lmfao, xD, XD, pepeLaugh, HAHAHA, hahaha, 4Head, Jebaited, EleGiggle, mdr, MDR, ptdr, "😂", "🤣", "💀", "草"] suffixes: [LUL, LOL, KEKW, KEK, Laugh, LAUGH, Lmao, LMAO, Lol, Haha, HAHA] patterns: ["^[ww]{2,}$", "ㅋ{2,}", "^(ja){2,}j?a?$", "^(ha){2,}h?$", "^k{4,}$", "^x+d+$", "哈哈", "^(lo)+l$", "^(ah){2,}$", "хаха", "^(he){3,}$"] surprise: label: surprised words: [HUH, WHAT, wtf, WTF, "?!", "!?", "!?", "?!", "は?", "やば", "やばい", "マジ", "헐", "뭐야", "什么", "😮", "😯", "🫢"] suffixes: [Huh, HUH, What, WHAT, Shock, Omg, OMG] message_patterns: ["^[??!!]{2,}$", "^[あえぁぇ]+[!!??ー~]*$", "^hu+h+[?!]*$", "^wha+t+[?!]*$"] scare: label: scared words: [monkaS, monkaW, monkaGIGA, "D:", WutFace, AAAA, AAAAA, JUMPSCARE, "jump scare", "怖", "무서", "😨", "😰", "😳"] suffixes: [Scared, Fear, Monka, Spooky, Scream] patterns: ["^a{4,}h*$"] fail: label: a fail words: [F, L, LL, Sadge, BibleThump, NotLikeThis, BigSad, FeelsBadMan, RIP, rip, NOOO, THREW, threw, copium, Copium, ";;", "T_T", "😭", "💔"] suffixes: [Sad, SAD, Cry, CRY, Rip, RIP, F, L, Dead] patterns: ["ㅠ{2,}", "ㅜ{2,}", "^no{3,}$"] clip: label: asking to clip it words: [CLIP, clip, clipped, clipit, "+2", "clip it", "clip that", "someone clip", "clipping", "クリップ", "클립"] # Messages that are only these (hellos, good lucks, goodbyes) aren't reactions # to anything: a stream's start, a raid arriving, or its end is full of them. chat_ignore: [hi, Hi, HI, hii, hiii, hello, Hello, hey, Hey, HEY, HEYY, HEYYY, yo, Yo, sup, gl, GL, glgl, o7, gn, gm, wb, bye, cya, "good luck", "welcome", "hello chat", "hi chat"] # ---- genres ----------------------------------------------------------------- # highlights: what a clip channel for this kind of game would post. # callouts: what a streamer says when it happens (shown to the model as # examples; phase 3 also looks for these words on screen). genres: shooter: label: shooter highlights: kills and multi-kills, a clutch against several enemies, a flick, one-tap or no-scope, wiping a squad, a round-winning play, and deaths that are funny or unfair callouts: ["got him", "one tap", "headshot", "ace", "clutch", "one left", "one hp", "he's one", "no scope", "double kill", "triple", "wiped", "let's go"] battle_royale: label: battle royale highlights: winning the game, the final fight, wiping a squad, a clutch with teammates down, a third-party ambush, and getting knocked or finished in a funny way callouts: ["victory royale", "winner", "we won", "one left", "knocked", "cracked", "squad wiped", "last circle", "third party", "got him", "let's go"] moba: label: MOBA highlights: winning teamfights, multi-kills up to a pentakill, stealing a big objective, an outplay or escape at low health, a game-winning push, and throws callouts: ["penta", "quadra", "triple", "ace", "steal", "stole", "outplay", "baron", "dragon", "roshan", "shutdown", "backdoor", "gg", "let's go"] sports: label: sports game highlights: goals and winning goals, saves, last-minute equalisers, skill moves, dunks, touchdowns and home runs, comebacks, and funny own goals callouts: ["goal", "scored", "what a goal", "top bins", "save", "last minute", "overtime", "penalty", "touchdown", "dunk", "home run", "buzzer beater"] fighting: label: fighting game highlights: big combos, perfect rounds, comebacks from low health, parries and reads, and the round-winning hit callouts: ["combo", "perfect", "parry", "comeback", "one hit", "read", "let's go"] racing: label: racing game highlights: last-lap overtakes, photo finishes, big crashes, and winning the race callouts: ["overtake", "crash", "first place", "photo finish", "last lap", "wrecked"] horror: label: horror game highlights: jump scares and the scream, being chased, a narrow escape, getting caught callouts: ["what was that", "oh my god", "run", "it's behind", "nope", "get away", "caught"] soulslike: label: action / soulslike highlights: beating a boss (especially first try or at low health), a boss's new phase, dying at the last moment, a clean parry or dodge, and rage callouts: ["boss", "phase two", "you died", "first try", "one hit", "parry", "no way", "let's go", "finally"] adventure: label: action-adventure highlights: beating a boss or a hard section, discovering something, a clever solve, and a funny death or glitch callouts: ["boss", "finally", "let's go", "no way", "found it", "secret"] sandbox: label: sandbox / survival highlights: a raid or fight, a betrayal, a rare find, a death (especially a funny one), revealing a build, and chaos with friends callouts: ["raid", "diamonds", "betrayed", "base", "run", "we're dead", "no way"] speedrun: label: speedrun highlights: a personal best or world record, pulling off a hard skip or trick, and the mistake that kills the run callouts: ["pb", "world record", "wr", "skip", "trick", "reset", "gold", "split"] party: label: party game highlights: an impostor reveal, a betrayal, a chaotic fail, a clutch win, and laughing with friends callouts: ["sus", "impostor", "vote", "betrayed", "crown", "let's go"] strategy: label: strategy / chess highlights: a brilliant move, a blunder, a comeback, and checkmate callouts: ["blunder", "brilliant", "checkmate", "mate in", "sacrifice", "gg"] reaction: label: reaction highlights: the streamer's biggest reactions to what they're watching (shock, laughter, a strong take), with the moment that caused it callouts: ["no way", "what", "oh my god", "bro", "let's go"] generic: label: game highlights: a big play or win, a funny or unlucky death, a surprise, and the streamer's reaction to it callouts: ["let's go", "no way", "oh my god", "got him", "what"] # ---- game sounds ------------------------------------------------------------ # What the sound tagger listens for (analysis/game_audio.py): groups of # AudioSet classes (config/audioset_labels.txt, exactly as written there). # "game" sounds come from the game; "people" sounds are mostly the streamer # or friends (and a horror game's screams), which count like a shout. sound_groups: gunfire: {label: gunfire, side: game, classes: ["Gunshot, gunfire", "Machine gun", "Fusillade", "Artillery fire", "Cap gun"]} explosion: {label: an explosion, side: game, classes: [Explosion, Boom]} crash: {label: a crash, side: game, classes: ["Smash, crash", Shatter, Breaking, Skidding, "Tire squeal"]} crowd: {label: a crowd cheering, side: game, classes: [Cheering, Applause, Crowd]} whistle: {label: a whistle, side: game, classes: [Whistle]} scream: {label: a scream, side: people, classes: [Screaming, Shout, Yell, "Battle cry"]} laughter: {label: laughter, side: people, classes: [Laughter, "Belly laugh", Giggle, "Chuckle, chortle", Snicker]} # How much each sound means in each kind of game, 0-1 (a group left out: 0.5). # In a shooter the gunfire and explosions are the fight (and a shield # breaking is a crash); in a sports game the crowd roars at a goal, and # Rocket League's goals explode; in a racing game the crash is the moment; # in horror, the scream. genre_sounds: shooter: {gunfire: 1.0, explosion: 1.0, crash: 0.6, crowd: 0.2, whistle: 0.0, scream: 0.8, laughter: 0.6} battle_royale: {gunfire: 1.0, explosion: 1.0, crash: 0.7, crowd: 0.2, whistle: 0.0, scream: 0.8, laughter: 0.6} moba: {gunfire: 0.3, explosion: 0.8, crash: 0.5, crowd: 0.4, whistle: 0.0, scream: 0.8, laughter: 0.6} sports: {gunfire: 0.0, explosion: 0.9, crash: 0.3, crowd: 1.0, whistle: 0.6, scream: 0.8, laughter: 0.6} fighting: {gunfire: 0.2, explosion: 0.6, crash: 0.6, crowd: 0.8, whistle: 0.0, scream: 0.8, laughter: 0.6} racing: {gunfire: 0.0, explosion: 0.8, crash: 1.0, crowd: 0.5, whistle: 0.0, scream: 0.8, laughter: 0.6} horror: {gunfire: 0.5, explosion: 0.6, crash: 0.7, crowd: 0.0, whistle: 0.0, scream: 1.0, laughter: 0.6} soulslike: {gunfire: 0.2, explosion: 0.7, crash: 0.6, crowd: 0.2, whistle: 0.0, scream: 0.8, laughter: 0.5} adventure: {gunfire: 0.5, explosion: 0.8, crash: 0.6, crowd: 0.3, whistle: 0.0, scream: 0.8, laughter: 0.6} sandbox: {gunfire: 0.7, explosion: 0.9, crash: 0.6, crowd: 0.3, whistle: 0.0, scream: 0.8, laughter: 0.7} speedrun: {gunfire: 0.2, explosion: 0.5, crash: 0.4, crowd: 0.6, whistle: 0.0, scream: 0.8, laughter: 0.6} party: {gunfire: 0.3, explosion: 0.6, crash: 0.5, crowd: 0.5, whistle: 0.2, scream: 0.9, laughter: 1.0} strategy: {gunfire: 0.3, explosion: 0.6, crash: 0.3, crowd: 0.3, whistle: 0.0, scream: 0.8, laughter: 0.6} reaction: {gunfire: 0.2, explosion: 0.3, crash: 0.3, crowd: 0.3, whistle: 0.0, scream: 0.9, laughter: 1.0} generic: {gunfire: 0.6, explosion: 0.8, crash: 0.6, crowd: 0.5, whistle: 0.2, scream: 0.8, laughter: 0.6} # ---- on-screen text --------------------------------------------------------- # What the game writes on screen (analysis/game_text.py reads it with OCR in # the game-moment windows only). Matched ignoring case, accents and spaces # (a stylised banner often loses its spaces) and the OCR's usual misreads # (Q for O, 1 for I), as a whole word when 5 letters or fewer ("PLAY" is not # in "PLAYER"). Games are played in many languages; the common ones for # each word are listed, and kanji where the OCR reads them. screen_text: # A banner: big text near the middle of the screen, where games announce a # kill, a goal or a win (a stream's chat overlay is small and ignored). events: generic: [VICTORY, DEFEAT, WINNER, "YOU WIN", "YOU LOSE", "GAME OVER", "LEVEL UP", "NEW RECORD", "ACHIEVEMENT UNLOCKED", "MISSION COMPLETE", "MISSION PASSED", "MISSION FAILED", "YOU DIED", CHAMPION, VICTOIRE, "DÉFAITE", VICTORIA, DERROTA, SIEG, NIEDERLAGE, "VITÓRIA", "勝利", "敗北"] shooter: [ELIMINATED, ELIMINATION, HEADSHOT, "DOUBLE KILL", "TRIPLE KILL", "MULTI KILL", "KILL STREAK", ACE, CLUTCH, FLAWLESS, "ROUND WON", "ROUND LOST", "TEAM ELIMINATED", "SQUAD WIPED", "SPIKE PLANTED", "SPIKE DEFUSED", "BOMB PLANTED", "BOMB DEFUSED", "ÉLIMINÉ", ELIMINADO, ELIMINIERT, "キル"] battle_royale: [ELIMINATED, "KNOCKED DOWN", KNOCKED, "SQUAD ELIMINATED", "SQUAD WIPED", "VICTORY ROYALE", "WINNER WINNER", "CHICKEN DINNER", "YOU ARE THE CHAMPION", "KILL LEADER", "CHAMPION SQUAD", "ÉLIMINÉ", ELIMINADO, ELIMINIERT, "チャンピオン"] moba: ["FIRST BLOOD", "DOUBLE KILL", "TRIPLE KILL", "QUADRA KILL", "PENTA KILL", PENTAKILL, ACE, SHUTDOWN, LEGENDARY, GODLIKE, RAMPAGE, UNSTOPPABLE, DOMINATING, "KILLING SPREE", "MEGA KILL", "ULTRA KILL", "BARON NASHOR", "DRAGON SLAIN", ROSHAN, "AEGIS", "INHIBITOR DESTROYED", "TOWER DESTROYED", "ANCIENT"] sports: [GOAL, GOL, "GOLAZO", TOR, BUT, SCORED, "A MARQUÉ", "MARQUÉ", "MARCÓ", ERZIELT, "HAT-TRICK", "HAT TRICK", "OWN GOAL", PENALTY, "EPIC SAVE", SAVE, OVERTIME, "FULL TIME", TOUCHDOWN, "HOME RUN", "BUZZER BEATER", DEMOLITION, "DÉMOLITION", MVP, "ゴール"] fighting: ["K.O.", KO, PERFECT, FATALITY, "FLAWLESS VICTORY", "FINAL ROUND", "YOU WIN", "CRITICAL ART", "SUPER ART", "COUNTER", "PUNISH COUNTER"] racing: [FINISH, "1ST", WINNER, "FASTEST LAP", "NEW RECORD", "PERSONAL BEST", WRECKED, TAKEDOWN, "FINAL LAP"] horror: ["YOU DIED", "GAME OVER", CAUGHT, ESCAPED, "YOU ESCAPED", SACRIFICED, "YOU SURVIVED"] soulslike: ["YOU DIED", "ENEMY FELLED", "GREAT ENEMY FELLED", "DEMIGOD FELLED", "LEGEND FELLED", "GOD SLAIN", "VICTORY ACHIEVED", "HEIR OF FIRE DESTROYED", "BOSS DEFEATED", "SHINOBI EXECUTION", "VOUS ÊTES MORT", "HAS MUERTO", "DU BIST GESTORBEN", "YOU DEFEATED", "死"] adventure: ["YOU DIED", "BOSS DEFEATED", "QUEST COMPLETE", "SHRINE COMPLETE", "SPIRIT ORB", "HEART CONTAINER", "NEW RECORD", "GAME OVER"] sandbox: [WASTED, BUSTED, "MISSION PASSED", "MISSION FAILED", "YOU DIED", "ADVANCEMENT MADE", "CHALLENGE COMPLETE", "RAIDED", "YOU WERE KILLED"] speedrun: ["PERSONAL BEST", "NEW RECORD", "WORLD RECORD", "PB", "WR"] party: [IMPOSTOR, EJECTED, "WAS THE IMPOSTOR", "WAS NOT THE IMPOSTOR", CROWNED, QUALIFIED, ELIMINATED, "YOU WIN"] strategy: [CHECKMATE, BRILLIANT, BLUNDER, RESIGNED, "WINS BY", "ÉCHEC ET MAT", "JAQUE MATE"] reaction: [] # A menu, a queue, a loading screen or a settings page: two of these on one # screen and it isn't gameplay. menu: [PLAY, READY, "READY - config/prompts/edit_clip.txt (3171 bytes)
You are the editing assistant inside a video clipping app. The user is looking at one vertical clip and wants to change it. Translate their request into the app's edit controls. CURRENT CLIP STATE: {state} BURNED-IN CAPTIONS (numbered lines, times are within the clip): {captions} THE CONTROLS YOU CAN SET (include ONLY the ones the user's request needs): - "start", "end": clip boundaries in seconds within the source video (source is {source_duration} seconds long). Use for "longer", "shorter", "start earlier", "trim the ending", etc. - "crop": one of "track" (camera follows the person), "center" (fixed center of frame), "bias_left" (show more of the left side), "bias_right" (show more of the right side). Use when framing/crop is off or someone is cut off. - "captions": true or false. False removes the burned-in captions entirely. - "filter": color look for the whole clip, one of: "none", "vibrant", "warm", "cool", "cinematic", "vintage", "bw" (black & white), "fade", "golden" (golden-hour sunset glow), "tealorange" (Hollywood teal & orange), "cleangirl" (bright airy minimal), "pinkglow" (soft Barbie pink), "peachy" (warm strawberry-girl flush), "dreamy" (soft pastel light-leak), "coast" (punchy ocean blue), "glow" (dewy white boost). Use when the user asks for a look, mood, filter, or color grade — e.g. "make it aesthetic" → "cleangirl" or "dreamy", "sunset vibes" → "golden". - "adjust": manual picture tweaks, object with any of: "brightness" (-0.5 to 0.5, 0 = unchanged), "saturation" (0 to 3, 1 = unchanged), "contrast" (0.5 to 2, 1 = unchanged), "temperature" (-1 cool to 1 warm, 0 = neutral), "tint" (-1 green to 1 magenta, 0 = neutral), "sharpen" (0 to 1), "vignette" (0 to 1, darkened edges). Use for "brighter", "darker", "warmer", "sharper", "more/less color", "punchier", etc. Small steps: brightness ±0.08, saturation/contrast ±0.15, temperature/tint ±0.3, sharpen/vignette 0.3-0.5 feel noticeable. - "caption_style": object with any of: "font": one of "Arial", "Arial Black", "Impact", "Verdana", "Tahoma", "Trebuchet MS", "Segoe UI", "Georgia", "Comic Sans MS", "Courier New" "font_size": number 40-140 (default 84) "color": hex like "#FFE135" "position": "bottom" | "middle" | "top" "words_per_caption": 1-6 (1 = one word at a time) "uppercase": true | false - "caption_fixes": list of corrections to caption TEXT, each {"line": <number from the list above>, "text": "corrected full text of that line"}. Use when the user says a caption has a typo/wrong word. Rewrite the whole line's text with the fix applied. USER REQUEST: {message} Respond with ONLY valid JSON, no other text: {"reply": "one or two friendly sentences telling the user what you changed", "needs_render": true} plus whichever control fields the request needs at the top level. Rules: - Keep clip duration between 5 and 120 seconds, inside the source bounds. - If the request is unclear or impossible with these controls (you cannot zoom, cut mid-clip sections, add music, or change the speaker's audio), set "needs_render": false and use "reply" to briefly say what you CAN do. - Never invent fields that are not listed above. - config/prompts/extract_knowledge_notes.txt (783 bytes)
Below is a rough live-stream/video transcript. List everything we learn about the CREATOR/STREAMER as short bullet points: life details, pets, family, purchases, plans, goals, running jokes, catchphrases, strong opinions, friends/collaborators, games played, recurring shows or segments. Rules: - Only include things actually said in the transcript. Never invent details. - Casual banter counts — jokes, personal stories and opinions are all useful. - Only call something a catchphrase or running joke if it comes up repeatedly, and quote their exact words for it. A good line said once is not a catchphrase. - Ignore greetings, filler, and moment-to-moment gameplay commentary. - If there is truly nothing about the creator, reply with exactly: nothing TRANSCRIPT: {transcript} - config/prompts/extract_knowledge.txt (3340 bytes)
You are building a knowledge base about a content creator from their stream/video transcript. Extract facts that are stated in the transcript below. Never invent details that are not in the transcript. FACT TYPES: - "topic": a subject they talk about at length (e.g. "powerlifting", "dating advice", "trading") - "game": a game or activity they are playing/doing, mentioned by name - "series": a named recurring show/format they run - "catchphrase": a short phrase the creator says OVER AND OVER in this transcript. Only use this type if you can see the phrase repeated several times — a memorable line said once is NOT a catchphrase. Write the phrase exactly as spoken. - "joke": a running joke or recurring bit they come back to more than once. Write the wording they use for it. - "collaborator": a named person appearing with them or described as a friend/collab partner - "format": a structural habit (e.g. "ends streams with viewer Q&A") - "life": a personal detail useful as a future callback (pet, family, habit, strong preference) EVENT TYPES (time-bound plans or achievements): - status "announced": something they say they WILL do ("next month I'm attempting...") - status "in_progress": an ongoing challenge/goal they are working on - status "completed": something they say they have now finished/achieved EXAMPLE 1 — transcript: "chat we're back with another episode of Monday Mailbag... yo what's good it's your boy... okay Duke is pulling up later we're gonna hoop... also big news, in two weeks I'm doing the 24 hour subathon" { "facts": [ {"type": "series", "information": "runs a recurring show called Monday Mailbag", "confidence": "high"}, {"type": "collaborator", "information": "Duke - friend who joins streams to play basketball", "confidence": "medium"} ], "events": [ {"name": "24 hour subathon", "description": "announced a 24 hour subathon in two weeks", "status": "announced"} ] } EXAMPLE 2 — casual stream banter still contains durable bits. Transcript: "my dog is staring at me again he does this every stream... nah I only drink the watermelon gfuel, the other flavors are trash... bro my squat is stuck at 315 I've been chasing 365 for a month" { "facts": [ {"type": "life", "information": "has a dog that appears on stream", "confidence": "high"}, {"type": "life", "information": "only drinks watermelon GFuel, calls other flavors trash", "confidence": "medium"} ], "events": [ {"name": "365 squat goal", "description": "working toward a 365 lb squat, currently stuck at 315", "status": "in_progress"} ] } Rules: - Facts must be about the CREATOR or their content, stated in the transcript. - A "catchphrase" or "joke" must use the creator's exact words, copied from the transcript — not your paraphrase of them. If they only said it once, leave it out. - Confidence "high" when explicit and unambiguous, "medium" otherwise. - Keep each "information" under 25 words, specific and self-contained (it will be read months later without this transcript). - Maximum 5 facts and 3 events. Casual banter counts: recurring jokes, personal details, strong opinions and goals are all durable. Use empty lists only when there is truly nothing about the creator. Respond with ONLY JSON in exactly the format of the example, no other text. TRANSCRIPT: {transcript} - config/prompts/look_at_game.txt (1396 bytes)
You are checking a candidate clip from a gaming stream by looking at it. The game is {game}. In this kind of game the best clips are: {highlights}. You are shown {count} frames from the clip, in order, taken at {times} seconds into it. What the streamer said in it: {transcript} What the stream's signals picked up in it (chat, the game's sound, on-screen text): {events} First say what is on the screen in each frame, in a few words: is it the game being played, or a menu, a settings page, a character or loadout select, a lobby or queue, a loading screen, a black screen, a "be right back" or starting-soon screen? Then answer: - gameplay: true if most frames show the game being played (or the streamer reacting to it); false if most frames show any of the other screens. - moment: a few words on what happens in the game, as the frames show it (for example "kills two enemies at close range", "scores a goal", "dies to the boss", "walks around the map"). Say only what you can see. - strength: 0 to 10, how clearly the frames show a real in-game moment worth clipping (a fight, a kill, a goal, a boss, a win, a funny fail, a big reaction): 0 nothing is happening, 5 ordinary play, 10 an unmistakable highlight. Respond with ONLY valid JSON: {"frames": ["<frame 1>", "<frame 2>", "<frame 3>", "<frame 4>"], "gameplay": <true or false>, "moment": "<what happens>", "strength": <0 to 10>} - config/prompts/metadata_batch.txt (871 bytes)
You write YouTube Shorts metadata that maximizes clicks without lying. Below are transcripts of {count} short clips, all taken from a longer video titled "{video_title}". Each clip is numbered. For EVERY clip, write: - "index": the clip's number, copied exactly - "title": under 90 characters, punchy, curiosity-driven, built from what is actually said in THAT clip. No clickbait the clip doesn't deliver. No quotation marks. Each title must be different. - "description": 1-2 sentences describing that clip, grounded in what is said. - "hashtags": 3-5 lowercase hashtags relevant to that clip's topic, each starting with #. Do not include #shorts (added automatically). Respond with ONLY valid JSON in exactly this shape, one entry per clip, no other text: {"items": [{"index": 0, "title": "...", "description": "...", "hashtags": ["#...", "#..."]}]} CLIPS: {clips} - config/prompts/metadata.txt (763 bytes)
You write YouTube Shorts metadata that maximizes clicks without lying. Below is the transcript of one short clip, taken from a longer video titled "{video_title}". Write metadata for this Short. Rules: - "title": under 90 characters, punchy, curiosity-driven, built from what is actually said in the clip. No clickbait that the clip doesn't deliver. No quotation marks. - "description": 1-2 sentences describing the clip, also grounded in what is said. - "hashtags": 3-5 lowercase hashtags relevant to the clip's topic, each starting with #. Do not include #shorts (it is added automatically). Respond with ONLY valid JSON in exactly this shape, no other text: {"title": "...", "description": "...", "hashtags": ["#...", "#..."]} CLIP TRANSCRIPT: {clip_text} - config/prompts/rerank.txt (631 bytes)
You are an expert short-form video editor choosing which clips to publish. Below are {count} candidate clips from one video, each with its index, timestamps, signal scores (0-100 audio excitement, visual activity, on-screen reaction), and what is spoken in it. Order ALL {count} clips from MOST likely to go viral to LEAST. Judge them against each other, considering: hook strength in the first seconds, emotional intensity, payoff, quotability, and whether the audio/visual energy backs up the words.{mode_guidance} Respond with ONLY valid JSON, listing every index exactly once: {"order": [2, 0, 1]} CANDIDATES: {candidates} - config/prompts/score_clips.txt (2952 bytes)
You are an expert short-form video editor. You find the moments in long videos that perform best as viral vertical Shorts/Reels/TikToks. Below is a transcript with timestamps in seconds, formatted as: [start - end] spoken text You may also see an AUDIO/VISUAL EVENTS list from automated signal analysis (loud reactions, laughter, scene cuts, high motion). A mild-sounding transcript line that coincides with a big audio spike is often a GREAT clip — weigh the events seriously.{mode_guidance} Find the best self-contained clip moments. Judge each candidate against this virality framework: - Hook moments and strong opening lines - Emotional peaks (laughter, anger, excitement, shock) - Opinion-driven statements ("opinion bombs") - Revelations or disclosures - Conflict or tension - Quotable lines - Story-structure peaks (setup that pays off) - Practical or actionable value - TRENDING & DRAMA (weight this heavily): the speaker names or reacts to another creator, streamer, celebrity, influencer, or public figure; discusses beef, callouts, controversy, or drama; reacts to viral/trending content, current events, or something everyone is talking about. These moments ride existing attention and perform extremely well — score them high and set "trending": true. Rules: - Each clip must be {min_duration} to {max_duration} seconds long. Let the moment decide its own length within that range — a punchy one-liner might be 12 seconds, a story that builds to a payoff might be 45. Do NOT default to the shortest length; capture the whole self-contained moment including its setup and payoff. - Each clip must make sense on its own with no extra context. - start and end must be numbers in seconds taken from the transcript timestamps. - score is 0-100 predicted viral potential. Be critical: most moments score below 50. Only truly strong moments score above 80. - engagement is 0-100 for hook strength + payoff + quotability specifically: would someone stop scrolling in the first 2 seconds, and is there a satisfying payoff? - trending is true if this moment names/reacts to another creator, celebrity, or public figure, or discusses drama/beef/controversy/trending topics; otherwise false. When true, boost the score. - Return at most 5 clips. If nothing is good, return an empty list. Respond with ONLY valid JSON in exactly this shape, no other text: {"clips": [{"start": 124.5, "end": 162.0, "score": 87, "engagement": 80, "trending": true, "hook": "...", "reason": "..."}]} Field meanings: - "hook": an attention-grabbing one-liner about THIS moment. It MUST be built from words actually spoken in the clip — quote or closely paraphrase the transcript lines between your start and end timestamps. Do not invent events that are not in the transcript. Do not write a generic description. - "reason": one sentence on why this specific moment will perform, referencing what is actually said in it. {events} TRANSCRIPT: {transcript} - config/prompts/score_windows.txt (1004 bytes)
You are an expert short-form video editor. Automated signal analysis found moments in a video with unusually high audio/visual excitement (possible laughter, shouting, hype, action). Your job: judge each window's clip potential from its spoken content.{mode_guidance} For EVERY window below, return one entry with: - "start"/"end": copy the window's own numbers - "score": 0-100 viral potential of this moment. Be critical — high energy alone is not enough; crowd noise over nothing scores low, a genuine reaction to something scores high. - "engagement": 0-100 hook strength + payoff + quotability - "hook": attention-grabbing one-liner built ONLY from words actually spoken in that window (or "High-energy moment" if there is no speech). Never invent content. - "reason": one sentence on why it will or won't perform. Respond with ONLY valid JSON, entries in the same order as the windows: {"clips": [{"start": 0, "end": 0, "score": 0, "engagement": 0, "hook": "...", "reason": "..."}]} {windows} - config/prompts/translate_captions.txt (959 bytes)
You are translating the spoken words of a short social video into {language}. Each numbered item is one sentence the creator said. RULES - Translate each numbered sentence. Return EXACTLY the same numbers, one per line, nothing else. - Keep each sentence about the same LENGTH as the original — it has to fit on screen for the same amount of time. - Sound like a real person talking, not a textbook. Keep slang casual, keep jokes funny, keep the energy. A natural equivalent beats a literal translation. - Keep profanity as profanity (the same strength), do not soften or censor it. - NEVER translate these names — copy them exactly as written: {terms} - Keep @handles, #hashtags, numbers and units as they are. - If an item is only a filler sound ("uh", "yeah"), give the natural equivalent in {language}. - Never merge or split items, never add commentary, never explain your choices. SENTENCES {lines} Return only the numbered translated sentences. - config/prompts/translate_metadata.txt (969 bytes)
Translate this short-video post into {language} so the creator can publish it to a {language}-speaking audience. RULES - Write it the way a native creator posts, not like a translated document. Hooks stay punchy, jokes stay funny. - The title must stay under 90 characters and keep its curiosity — rewrite rather than translate literally if that reads better. - NEVER translate these names — copy them exactly as written: {terms} - Hashtags: translate the ones that describe the topic, keep brand names, handles and #shorts as they are, keep them lowercase, 3-5 total. - No quotation marks around the title. No commentary, no explanations. - Keep the three labels in ENGLISH exactly as shown below (TITLE:, DESCRIPTION:, HASHTAGS:). Translate only what comes after them. No markdown, no ## or **. POST TITLE: {title} DESCRIPTION: {description} HASHTAGS: {hashtags} Reply in exactly this format, nothing else: TITLE: ... DESCRIPTION: ... HASHTAGS: #... #... #... - config/settings.yaml (8192 bytes)
# ============================================================== # QUICK SETUP — edit these lines, ignore everything below # ============================================================== model: gemma:7b # your AI model (see: python main.py models) channel: "" # your channel, e.g. "@YourHandle" content_language: auto # language of the streams you clip: auto = detect per # video (fine for most). Force a code (es / pt / hi / # id / en ...) if bilingual streams get transcribed in # the wrong language — captions burn in this language. auto_upload: false # true = post Shorts to YouTube automatically # (needs one-time setup: README "Enable uploads") # NOTE: this is the command-line daemon only. To publish # from the editor instead, turn on Settings -> Publish # to YouTube; it ignores both of these keys. privacy: public # public / unlisted / private for posted Shorts. # NOTE: until your API project passes YouTube's free # audit, YouTube locks API uploads to private whatever # this says — permanently, and it cannot be undone in # Studio. See README "Posting publicly". # ============================================================== # Advanced settings — defaults are fine for most people # ============================================================== # Extra channels beyond the one above (handles, URLs, or IDs), # or use: python main.py channels add @SomeHandle channels: [] poll_interval_minutes: 15 llm: # Translation (multilingual publishing) can use a DIFFERENT local model # than clipping. Clipping models are picked for judgement; translation # wants multilingual strength, where the Qwen line is markedly better than # gemma — qwen3:8b, or qwen3:14b if it fits. Both are Apache-2.0, which # matters if you earn from the clips. qwen2.5:7b remains a smaller option. # Empty = use the main model above. Install it from the Models page; # nothing is downloaded until you ask. translation_model: "" temperature: 0.4 num_ctx: 8192 # context window; raise for models that support more ollama_host: http://localhost:11434 whisper: model: auto # auto = large-v3-turbo on GPU, small on CPU. Or force: # tiny / base / small / medium / large-v3 / large-v3-turbo device: auto # cuda if available, else cpu # Where transcription runs. "local" is Whisper on this PC (the settings above), # the default. Or send the audio, on YOUR OWN API key, to a provider that # returns word timings: openai / openrouter / xai. Set it in Settings → AI. transcription: backend: local model: "" clips: min_score: 55 # discard candidates scored below this (0-100). # Higher = fewer, better clips. Kept generous on # purpose: creators pick what to post — a ~2h stream # should yield ~50 clips to choose from. max_clips_per_video: 0 # 0 = automatic: keep every clip that passes min_score min_duration: 10 # seconds — clips land on sentence boundaries, so real max_duration: 60 # lengths vary naturally across this 10-60s range # (turn on the 60s+ toggle for TikTok-length clips) vertical: true # true 9:16 1080x1920 with YOLOv8 subject tracking captions: true # burn word-synced captions into the clips outro: false # append the 2.9s Clips Kitty end card to each clip. # Off by default; Settings > Branding turns it on. # Stream-copied, so the clip is never re-encoded. # Absent counts as false. scoring: # Multimodal fusion weights (must sum to 1.0). Raise audio/visual for # gameplay/stream content; raise text for talking-head/educational content. weights: text: 0.30 visual: 0.20 reaction: 0.20 audio: 0.20 engagement: 0.10 signal_peak_percentile: 85 # how unusual a moment must be to become a candidate creator_context_max: 6 # max score bonus for creator-knowledge callbacks # (ongoing storylines, catchphrases, collaborators). # Additive-only and capped so learned knowledge can # never DEGRADE scores. 0 disables entirely. action_bonus: 10 # score bonus for active-content clips (workouts, # sports, dance): a person on-screen and moving # (whether or not they're talking). Surfaces this # content, which performs on social but which the # scorer under-credits. Raise for MORE such clips, # 0 to disable. audience_bonus: 8 # max score bonus from real audience data: Twitch # chat replay speed (unique chatters, so gifted-sub # spam doesn't count) and YouTube's most-replayed # heatmap. Only the very top of the curve earns it; # kept below action_bonus so the video's own content # stays the main signal. 0 disables. (Kick keeps no # chat after a stream ends, so Kick has no chat data.) rerank_pool: 8 # finalists compared head-to-head by the LLM reaction_top_k: 8 # candidates that get the (slower) YOLO reaction pass analysis: # Duplicate prevention thresholds max_overlap: 0.4 # reject >40% timestamp overlap with a kept clip max_text_similarity: 0.7 # reject >70% transcript similarity with a kept clip max_segment_reuse: 0.4 # reject if >40% of transcript segments already used # Long-video chunking. Sized for a 7B model's 8K context: ~5 minutes of # speech per chunk leaves the model ample room to think and respond. # Models with bigger contexts can raise chunk_seconds and llm.num_ctx together. chunk_seconds: 300 chunk_overlap_seconds: 45 long_video_threshold_seconds: 420 video: encoder: auto # auto = first working GPU encoder (NVIDIA nvenc / # AMD amf / Intel qsv), else CPU. Or force one: # "nvenc" / "amf" / "qsv" / "cpu" parallel_renders: 3 # clips rendered at once (one's GPU tracking overlaps # another's NVENC encode). Sized for ~6 CPU cores / # 16GB RAM; raise only with more RAM + spare VRAM. tracking: detector: yolov8n-pose.pt # pose model: head keypoints (nose/eyes/ears) give # head-priority framing even when no face is # visible. Auto-downloaded by ultralytics. sample_fps: 8 # detection sampling rate (higher = tighter tracking) upload: # Both keys belong to the command-line daemon (`python main.py upload`). # Publishing from the editor uses your own key, set up in Settings, and is # capped by YouTube rather than by anything here. daily_limit: 6 # hard cap on Shorts scheduled per calendar day. # Sized for the OLD quota model, where one upload cost # 1,600 of a shared 10,000 units a day. Since June 2026 # videos.insert has its own bucket of 100 uploads a day # per Google Cloud project, so this is now a self-imposed # pacing limit, not a technical one. client_secret: config/client_secret.json # from Google Cloud Console feedback: # Where the in-app Feedback Hub sends reports (a Cloudflare Worker that # files them as GitHub Issues — users need no account; see # feedback-relay/README.md to deploy). Empty = reports save to a file. relay_url: "https://clips-studio-feedback.clipsstudio.workers.dev" paths: data_dir: data - CONTRIBUTING.md (8202 bytes)
# Contributing to Clips Kitty Thanks for helping! Clips Kitty is a local-first AI clipping app. Python FastAPI backend + Electron/React UI. Issues labeled [`good-first-issue`](../../labels/good-first-issue) are small and well-scoped if you want a place to start. ## Fixing a bug from a Feedback Hub report Most bug reports arrive through the app's in-app Feedback Hub (label [`from-app`](../../labels/from-app)). They're written by non-technical streamers, but each one carries an auto-collected **Diagnostics** block: - exact app commit, Windows version, CPU/GPU/VRAM/RAM - the AI model in use (name, parameter size, quantization, Ollama version) - FFmpeg / OpenCV / faster-whisper / yt-dlp versions - the non-secret settings, the video's platform, and the recent log tail That block is the reproduction recipe: match the model + settings and feed a similar video (platform matters. Twitch/Kick VODs are H.264; YouTube sources are H.264 by design, see `video/encoding.py`). Secrets and usernames are redacted before reports ever leave the reporter's machine. ## Dev setup ``` pip install -r requirements.txt pip install ruff pytest # the checks CI runs cd ui && npm install npm run dev # starts Electron + the backend together ``` Working on the interface only? `npm run dev:web` serves it in a browser instead, with no Electron and no rebuild loop. The Electron-only calls (file pickers, the donate window) are stubbed there. See `ui/src/renderer/src/lib/browserShim.ts`. Before a PR: ``` pytest # deterministic logic ruff check . # lint cd ui && npm run typecheck && npm run build ``` Want to add a language, a model, a platform or an export format? Each is a small, well-defined change. See [docs/EXTENDING.md](docs/EXTENDING.md). ### Or use Docker: nothing else to install Install [Docker Desktop](https://docs.docker.com/get-started/get-docker/) and that is the whole list. Compose ships inside it; there is no second install. ``` docker compose pull # prebuilt engine image, skips a 10-min build docker compose up # engine :8765, interface :5173, Ollama docker compose run --rm engine pytest ``` Saves installing Python, FFmpeg, PyTorch, OpenCV and Node locally, and lets you contribute from Linux or macOS to an app that only ships for Windows. Electron itself still runs on your host, since it needs a display. See [docs/DOCKER.md](docs/DOCKER.md). ## Something to run against You do not need a stream to start. There is a written-out transcript and a generator for synthetic footage in [tests/assets/](tests/assets/): ``` python tests/assets/make_sample_video.py # 2 min, 1080p, known audio peaks ``` And [examples/](examples/) has small runnable programs for the two things you are most likely to be doing: ``` python examples/score_a_transcript.py --fake # clip selection, seconds per run python examples/drive_the_api.py # the pipeline over HTTP ``` `--fake` uses a canned model reply, so scoring logic can be worked on with no Ollama and no waiting, and behaves the same way every run. If you are changing how clips get picked, that loop is worth learning first. The alternative is a full transcribe-and-render per attempt. ## Building on the app rather than in it You do not have to change Clips Kitty to use it. The desktop window is one client of a local HTTP API, and anything else can be another. A bot, a batch runner, a different interface entirely. - **[docs/API.md](docs/API.md)**: the endpoints meant to be built on, the supported-versus-internal line, and the traps worth knowing first - **[docs/EXTENDING.md](docs/EXTENDING.md)**: changing the app itself, by adding a language, a platform, an AI model or an export format If you build something, say so in an issue. Needing an endpoint is the fastest way to get it moved from internal to supported. ### Write a plugin A plugin finds, understands or rates the moments in a video, for one game, sport or kind of show, without changing Clips Kitty. Three commands start one and run it on a test video (in PowerShell, write `py -m` for `python -m`): ```bash python -m pip install "clipskitty-sdk[yaml,test] @ git+https://github.com/ColinGPT9/clips-studio#subdirectory=sdk/python" python -m clipskitty_sdk new my-plugin --template game-events --publisher your-github-name python -m clipskitty_sdk run my-plugin --sample ``` Clips Kitty 2.1.0 is the first release that runs plugins, and it comes with SDK 1.3.0. The walk-through is [docs/developers/first-game-pipeline.md](docs/developers/first-game-pipeline.md), and everything else for plugin developers is in [docs/developers/](docs/developers/README.md). The SDK is MIT, so your plugin can use any licence you choose. ## Building the Windows installer ``` pip install -r requirements-build.txt python scripts/build_installer.py ``` That runs the whole chain and stops at the first failure with an explanation: fetch FFmpeg → freeze the Python engine with PyInstaller → smoke-test the frozen engine → build the renderer → wrap it all up. The results land in `release/`: a small **Web Setup .exe**, the **.7z payload** it downloads, and a **.zip** for offline installs. Publish the Web Setup and the .7z to the *same* GitHub release. The setup fetches the payload by name, so one without the other is useless. Full release checklist: [docs/RELEASING.md](docs/RELEASING.md). Expect it to take a while and to need disk: the frozen engine is ~4.8 GB unpacked, mostly CUDA PyTorch. `--skip-backend` and `--skip-ui` reuse the previous run's output while iterating, and `--backend-only` stops after the freeze. Things worth knowing before you change any of it: - **The backend is frozen as a console app on purpose.** A windowed PyInstaller build gives the process no stdout, and every `print()` in the pipeline then raises. Electron passes `windowsHide` so no console appears. - **PyTorch ships as the CUDA build.** It isn't only for tracking: those wheels carry the cuBLAS/cuDNN DLLs that CTranslate2 needs for GPU transcription, so a CPU build silently drops Whisper to CPU as well. - **FFmpeg comes from `vendor/`,** fetched by `scripts/fetch_ffmpeg.py` and gitignored. Never call `ffmpeg` by bare name. Use `core.binaries.ffmpeg()`, or an installed copy will look for a binary the user doesn't have. - **Code signing is off** (`signAndEditExecutable: false`). electron-builder otherwise downloads a bundle containing macOS symlinks, which an ordinary Windows account cannot extract. The trade-off and how to re-enable it are documented in `ui/electron-builder.yml`. - **Don't switch the target back to plain `nsis`.** `makensis.exe` is 32-bit and memory-maps the payload to embed it, so it dies around 2 GB with `failed creating mmap`. The app is ~5 GB unpacked. That is a ceiling, not a setting. ## Pull requests - Keep PRs focused on one issue; link it ("Fixes #123"). - `npm run typecheck` clean; try the affected flow in the running app. - Match the style around you: comments explain *why*, not *what*. CI runs on every PR: Python compiles, config and prompts parse, the desktop app typechecks and builds, and website links resolve. It is fast and it is narrow. A runner has no GPU, no Ollama and no footage, so **a green tick does not mean clips still come out well.** Anything touching scoring, tracking, captions or rendering needs testing against a real video, and the PR should say which one. ## Triage (maintainers) Priority = 👍 reactions + comment count on issues, `critical` / `high-priority` labels first. Duplicates: close with a link to the canonical issue so reactions concentrate in one place. ## Licence Clips Kitty is **AGPL-3.0**, and contributions are accepted under the same terms: opening a PR means you are licensing your change that way. Practically: your work stays available to the creators this was built for, and nobody can take the project closed. Adding a dependency? Check its licence first. Anything the installer bundles has to be compatible with AGPL-3.0, and a new one belongs in the third-party list in [NOTICE](NOTICE). - docker-compose.yml (3655 bytes)
# Contributor environment: the Clips Kitty engine plus Ollama. # # docker compose up --build engine on http://localhost:8765 # docker compose run --rm engine pytest # # Not an installation route — creators use the Windows installer. See # docs/DOCKER.md for what this can and cannot do. services: engine: # Named for GHCR so `docker compose pull` fetches the prebuilt image # (~2 min) instead of building it (~10). `build:` stays as the fallback: # with no image present locally, `up` builds rather than failing, so a # fork with nothing published still works. image: ghcr.io/colingpt9/clips-studio-engine:latest build: . ports: # Host-only. Publishing to 0.0.0.0 would put an unauthenticated API # that reads and writes video files onto the local network. - '127.0.0.1:8765:8765' volumes: # Source is mounted so edits are live without rebuilding. The image # already contains a copy, so it still runs standalone. - .:/app # Downloads, clips and the database persist in a named volume rather # than the container, which is thrown away on every rebuild. - clips-data:/data environment: # Inside the compose network Ollama is a service name, not localhost. CLIPS_STUDIO_OLLAMA_HOST: http://ollama:11434 CLIPS_STUDIO_DATA_DIR: /data # Windows-locale encoding is not a container problem, but the engine # prints titles containing emoji and the default here is ASCII. PYTHONIOENCODING: utf-8 PYTHONUNBUFFERED: '1' depends_on: - ollama # The front-end toolchain: Vite, React, TypeScript. Open # http://localhost:5173 in a browser to work on the interface without # installing Node. # # This serves the RENDERER, not Electron. Electron needs a display, so the # desktop shell still runs on your host — see docs/DOCKER.md. In a browser # the interface works and talks to the engine; only the native bits # (file pickers, the donate window, external links) are unavailable, # because those live in Electron's preload. ui: image: node:20-slim working_dir: /app/ui ports: - '127.0.0.1:5173:5173' volumes: - .:/app # Anonymous volume so the container keeps its OWN node_modules. Without # this the host's are mounted over the top, and esbuild/rollup ship # platform-specific binaries — Windows ones do not run on Linux, and # the failure is a confusing "cannot find module" for a package that # is plainly installed. - ui-node-modules:/app/ui/node_modules # dev:web serves the renderer alone, via ui/vite.config.mts. Plain # `vite` would find no config: the app is built by electron-vite, whose # config describes three bundles and cannot serve one of them on its own. command: > sh -c "npm install && npm run dev:web" environment: # Vite needs polling to see edits made on a Windows or macOS host # through a bind mount; inotify does not cross that boundary. CHOKIDAR_USEPOLLING: 'true' ollama: image: ollama/ollama:latest ports: - '127.0.0.1:11434:11434' volumes: # Models are gigabytes. Losing them on a rebuild would be painful. - ollama-models:/root/.ollama # GPU: uncomment with the NVIDIA Container Toolkit installed. Without it # Ollama runs on CPU, which works and is slow. See docs/DOCKER.md. # deploy: # resources: # reservations: # devices: # - driver: nvidia # count: all # capabilities: [gpu] volumes: clips-data: ollama-models: ui-node-modules: - Dockerfile (2196 bytes)
# Contributor environment for the Clips Studio engine. # # This is NOT how anyone installs Clips Studio — that is the Windows # installer. This image exists so someone can work on the Python engine, run # the tests and hit the API without installing Python, FFmpeg, PyTorch and # the rest on their own machine, and so a contributor on Linux or macOS can # work on a project whose app only ships for Windows. # # The desktop app is not in here. Electron needs a display and a real # windowing system; running it in a container would be more trouble than # installing Node. Run the UI on your host against this engine — see # docs/DOCKER.md. FROM python:3.11-slim # ffmpeg — every decode, encode and probe in the pipeline # libgl1 etc. — OpenCV imports fail without them on slim images, with an # error that names libGL rather than cv2 and wastes an hour # git — yt-dlp and some tooling shell out to it RUN apt-get update && apt-get install -y --no-install-recommends \ ffmpeg \ git \ libgl1 \ libglib2.0-0 \ && rm -rf /var/lib/apt/lists/* WORKDIR /app # CPU PyTorch, installed FIRST so the requirements below resolve against it # rather than pulling ~4 GB of CUDA wheels. A container is for editing code # and running tests; if you want GPU work, do it on the host where the app # actually ships. docs/DOCKER.md explains the GPU variant. COPY requirements.txt requirements-build.txt ./ RUN pip install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu \ torch torchvision \ && pip install --no-cache-dir -r requirements.txt \ && pip install --no-cache-dir ruff pytest # Source is bind-mounted by compose so edits are live; this copy is what # makes the image useful on its own. COPY . . # Data lives on a volume, not in the image — a container should never be # where someone's downloads and clips quietly accumulate. ENV CLIPS_STUDIO_DATA_DIR=/data VOLUME ["/data"] EXPOSE 8765 # Bind to 0.0.0.0 rather than 127.0.0.1: inside a container, localhost is # the container, and nothing on the host could reach it. CMD ["python", "main.py", "serve", "--port", "8765", "--host", "0.0.0.0"] - INSTRUCTIONS.md (2836 bytes)
--- name: clips-kitty description: Turn long videos and streams into short vertical clips with Clips Kitty, running locally on this computer. Use when someone wants clips, Shorts, Reels or TikToks made from a YouTube, Twitch or Kick link or a video file, or wants to read back or export clips Clips Kitty already made. --- # Clips Kitty Clips Kitty finds the moments worth posting in a long video, crops them to 9:16 with the speaker kept in frame, burns in captions and writes titles. Everything runs on the user's own machine: no account, no upload, no API key. ## Connecting The tools come from Clips Kitty's MCP server, which talks to the engine on `127.0.0.1:8765`. ```bash claude mcp add clips-kitty -- python main.py mcp # from a source checkout ``` In an installed build the engine's binary is `api.exe`, inside the app's `resources\backend` folder, so the command is `api.exe mcp`. **Clips Kitty has to be running**, either the app itself or `python main.py serve`. If a tool reports the engine is not answering, say so and stop: starting it is the user's call, not something to work around. ## How to work 1. **Queue** with `queue_video` (a link) or `queue_local_file` (a file on disk). 2. **Check back** with `job_status`. Processing a two-hour stream takes tens of minutes on a good machine. Never sit in a polling loop: report the job id, and check when the user next asks or after a long wait. 3. **Read the results** with `list_clips`, which gives each clip's id, score and title. 4. **Export** with `export_clip` when the user names a folder. ## Things that will otherwise catch you out - **A queued job is not guaranteed.** `queue_video` returns "Not queued" when that video was processed before, because redoing it costs an hour and duplicates its clips. Offer `list_clips` on the existing video, or `force` if they really meant it. - **An unknown video id looks like a video with no clips**, because the API returns an empty list either way. Check `list_videos` before concluding a video produced nothing. - **A paused queue looks like a broken app.** `queue_status` reports it. Check it before telling anyone nothing is happening. - **Scores are the model's ranking, 0-100**, not a promise. The user picks what to post. - **Gaming uses its own framing.** Set `gaming=true` for its webcam/game layout and existing layout detector. Omit `gaming_layout` for automatic detection, or pass the user's selected layout and boxes. An installed game-specific detector is chosen with `pipeline={"id":"publisher/name"}`; discovery and permissions remain in the app. ## What it will not do There is no cloud tier, no upload of the user's footage anywhere, and no posting to social platforms except YouTube, which the user switches on themselves with their own Google key. Do not offer those. - KNOWN-ISSUES.md (9068 bytes)
# Known issues Everything here has actually been observed, most of it while testing releases on a clean machine. Nothing on this list is speculative, and nothing that has been fixed is still listed as broken. Hit something that is not here? The **💬 button on the Dashboard** files a report without needing a GitHub account, or open an [issue](https://github.com/ColinGPT9/clips-studio/issues). --- ## YouTube locks uploads to private until your API project is audited If you publish from the editor and the video arrives on your channel as **private** when you asked for public, this is why, and it is not something Clips Kitty can fix. YouTube restricts uploads made through the API by any Google Cloud project that has not passed its free compliance audit. The lock is **permanent**: you cannot change the video to public in YouTube Studio afterwards, and there is no appeal. The only remedy is uploading the video again from an audited project. Clips Kitty reads the privacy back after every upload and tells you when this has happened, rather than reporting success for a video nobody can watch. **What to do:** submit the [YouTube API Services audit form](https://support.google.com/youtube/contact/yt_api_form) before you rely on public publishing. Uploading as **unlisted** or **private** is not affected, so it is a usable workflow in the meantime. ## Your YouTube sign-in expires weekly if the consent screen is on "Testing" Google expires the sign-in for an OAuth app in *Testing* after 7 days, so Clips Kitty asks you to reconnect every week. **Fix:** in the Google Cloud Console, open the OAuth consent screen and press **Publish app**. "In production" does not mean verified and costs nothing. You will simply see a "Google hasn't verified this app" warning when connecting, and **Advanced → Go to (unsafe)** gets past it. It is your own app warning you about yourself. --- ## Only three AI models have actually been tested Clips Kitty can run any model Ollama serves, and the Models page lists several. **Only these three have been run against real streams:** - `gemma:7b` - `gemma3:4b` - `gemma3:12b` Everything else (Gemma 4, Qwen3, Mistral Nemo, Phi-4, Llama) is listed because it is a sensible size for the hardware and is free to use commercially, **not** because clip quality has been measured with it. They should work; the app talks to all of them the same way. Nobody has checked whether they pick better or worse moments. Three reasoning models on the Models page have had a shorter check: parts of one real gym stream transcript, not full runs. All three returned scored clips with hooks taken from what was actually said. - `deepseek-r1:8b` found about as many moments as `gemma:7b`, at a similar speed. On a line with explicit language it once wrote a refusal ("I'm sorry, but I cannot provide...") as a clip's hook instead of a quote, so check its titles. - `gpt-oss:20b` found good clips but is slow: close to a minute for every five minutes of video on a 12 GB card, because it thinks before every answer. - `nemotron-3-nano:4b` works, but found far fewer moments than the others. If you try one, saying how it went is genuinely useful. That is a gap that only gets closed by people running different models on different content. ## Windows warns that the app is unsigned On first run you get **"Windows protected your PC"**. Click *More info → Run anyway*. The installer is not code-signed, because a certificate costs money this project does not have yet. The warning is about the **absence of a signature**, not about anything Windows found in the file. Signing is on the roadmap. ## Game streams and reactions need Gaming / Reaction switched on Processed the standard way, a game stream is framed poorly: to a person detector a character inside the game is a person on screen, and the crop can follow it instead of the streamer. **Turn on Gaming / Reaction** for game streams and reaction videos ([docs/GAMING.md](docs/GAMING.md)): the webcam goes in one half and the game in the other, the streamer is whoever is talking in sync with the audio, and the split can be checked on the video's own frames before processing. Its own limits: - **VTubers aren't supported.** The detection is for people on camera; a VTuber stream gets the game on its own. - **A streamer who barely speaks** (listening through a reaction, face half hidden) may not be found automatically. Draw the webcam in the setup; ticking "Remember for this creator" means it's done once. - **One layout per clip.** **Top-down games without the switch.** MOBAs, strategy and tower-defence games (Dota 2, League, anything viewed from above) have no person-shaped subject on screen. Part of a clip's standard score measures whether a human is visible and being emphasised, and with nothing detected that part is zero, so scored gameplay can land under the threshold. **Gaming / Reaction leaves that part neutral**, which is the fix. Without it, the app says so where the clips would have been, with the numbers behind it. **What works best without it:** IRL, just chatting, podcasts, vlogs and interviews. ## A download can break when a site changes, until the next release Downloading is handled by yt-dlp, which is bundled inside the app. Twitch, Kick and YouTube change things regularly; yt-dlp fixes them within days, but the copy inside Clips Kitty is fixed at build time. So there is a window, from a site changing to the next Clips Kitty release, where downloads from that one site fail even though the fix already exists. Other sites keep working, which is the tell: **if Twitch works and Kick does not, it is this, not your setup.** Being worked on in [#39](https://github.com/ColinGPT9/clips-studio/issues/39). If you hit it, report it with the site and the error. It helps establish how often this actually bites. ## Podcast mode frames one person per shot, on purpose Podcast mode does not put two people on screen together, and does not split-screen. That is a decision, not a limitation it fell into. Earlier versions tried framing two people at once. Averaging two positions pulled every shot back toward the centre and made both faces small. The "everyone tiny" look, and the same person appearing in two camera angles got counted as two speakers, producing a split screen of somebody with himself. A podcast is already edited so the camera cuts to whoever is talking. So the app follows that edit: detect the camera cuts, and within each shot punch in on the person who matters. If a clip frames the *wrong* person, that is a real bug worth reporting. Framing one person is intended. ## The app looks frozen while it is scoring After transcription there is a long stretch: often the longest part of the whole job, where nothing appears to happen and the log says nothing. It is working; the scoring stage does not report progress yet. **How to tell:** open Task Manager. If `ollama.exe` is using CPU, it is scoring. Being unable to distinguish this from a crash is a real problem and it is being fixed. ## Updating shows no progress after the first megabyte The update downloads a small installer, shows 1 MB / 1 MB and 100%, and then appears to stop. It has not stopped. It is fetching the ~6 GB payload behind that, and the tool it uses does not report progress for that part. Leave it running. A **Restart & install** button appears when it finishes. ## A failed update disappears silently If an update fails, the banner vanishes rather than saying so, and no log is kept. If an update seems to go nowhere, download the installer from the [releases page](https://github.com/ColinGPT9/clips-studio/releases) instead, installing over the top works fine and keeps your settings and models. ## Low-memory machines may fail to render Seen on a machine with **8 GB of RAM and no graphics card**: the whole job runs, picks its clips, and then every render fails with an out-of-memory error from the encoder. **16 GB is the recommended minimum**, and a graphics card matters more than anything else here, with one, the AI model sits in video memory instead of competing with the encoder for system memory. Seen twice now, both times in a memory-capped virtual machine rather than on bare metal: at 8 GB the job analysed a video and produced zero clips, and the first contributor to get a full run through a container had to raise the limit to **12 GB**. At 12 GB, roughly an hour of source video is the practical ceiling. If you are running in Docker, Windows Sandbox, or a VM, that limit is the setting to check first, because the failure looks like a crash rather than a memory limit. Still not seen on a normal 16 GB desktop. If you hit it on a machine that should be capable, please report it with your specifications. ## Everything is slower without a graphics card It works on CPU. Transcription, tracking and scoring all run, just far slower, a video that takes minutes with an NVIDIA card can take hours without one. The app detects this and picks a smaller AI model to compensate. - LICENSE (34523 bytes)
GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007 Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/> Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU Affero General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a cons - main.py (17449 bytes)
"""CLI entry point. For YouTubers — three commands to full automation: python main.py channels add @YourHandle # paste your handle or channel URL python main.py run # daemon: watch, clip, schedule python main.py status # see what it's done More: python main.py process <url> # one video, end-to-end python main.py channels list / remove <id> python main.py models # installed models + GPU guide python main.py models use gemma3:12b # switch the LLM (one command) """ import argparse import os import sys from pathlib import Path def _force_utf8_io() -> None: """Print UTF-8 no matter who started us. Windows picks the process's text encoding from the system locale, which on most Western installs is cp1252 — and cp1252 cannot encode an emoji. Stream and video titles are full of them, so a single print of a title like "🏴 stream" raised UnicodeEncodeError and killed the backend mid-run. The console the developer uses often has UTF-8 set already, which is exactly why this hides until someone else installs the app: Electron spawns the backend WITHOUT inheriting that. Reconfiguring here covers every entry path — CLI, Electron dev, and the packaged executable — because they all come through this file. """ # A frozen build launched without a console has no stdout at all, and # print() to None raises — which would turn every progress line in the # pipeline into a crash. Give it somewhere harmless to write first. for name in ("stdout", "stderr"): if getattr(sys, name, None) is None: setattr(sys, name, open(os.devnull, "w", encoding="utf-8")) for stream in (sys.stdout, sys.stderr): try: stream.reconfigure(encoding="utf-8", errors="replace") except (AttributeError, ValueError, OSError): pass # already UTF-8, detached, or not reconfigurable — fine _force_utf8_io() # A pipeline running on the Python inside the installed app: the engine is # started as `api.exe src/main.py ...` with this marker (plugin_env in # sdk/python/clipskitty_sdk/host.py). Handled here, before the heavy imports # below, so a pipeline starts quickly and loads none of the engine. if os.environ.get("CLIPSKITTY_SCRIPT_HOST") == "1": import _clipskitty_script_host if _clipskitty_script_host.wants_host(sys.argv): sys.exit(_clipskitty_script_host.main(sys.argv[1:])) import yaml from core.paths import resolve_data_dir, user_config_path from core.pipeline import process_video from core.scheduler import run_daemon from core.state import StateDB # What ships with the app. Read-only defaults in an installed build; the file # you edit in a checkout. BUNDLED_CONFIG = Path(__file__).resolve().parent / "config" / "settings.yaml" # What this install actually reads and writes. The same path in a checkout, # per-user app data once frozen — see user_config_path() for why writing to # the bundled copy cannot be relied on. CONFIG_PATH = user_config_path(BUNDLED_CONFIG) def load_config(path: Path) -> dict: with open(path, encoding="utf-8") as f: config = yaml.safe_load(f) # The "quick setup" block at the top of settings.yaml uses flat, # non-coder-friendly keys. Normalize them onto the full structure here # so the rest of the codebase only ever sees one shape. model = config.get("model") if model: spec = str(model) if "/" in str(model) else f"ollama/{model}" config.setdefault("llm", {})["backend"] = spec channel = config.get("channel") if channel and str(channel).strip(): config.setdefault("channels", []) if channel not in config["channels"]: config["channels"].insert(0, str(channel).strip()) if "auto_upload" in config: config.setdefault("upload", {})["enabled"] = bool(config["auto_upload"]) privacy = str(config.get("privacy", "")).strip().lower() if privacy in ("public", "unlisted", "private"): config.setdefault("upload", {})["privacy"] = privacy # Environment overrides. Containers cannot edit settings.yaml — Ollama # lives at a service name rather than localhost, and the data directory # is a mounted volume. These let a compose file say so without anyone # hand-editing config that is checked into the repo. # # The desktop app uses the same door: it starts its own bundled Ollama on # a private port so it cannot collide with one the creator already runs, # and passes the address in here rather than rewriting settings.yaml # underneath them. ollama_host = os.environ.get("CLIPS_STUDIO_OLLAMA_HOST") if ollama_host: config.setdefault("llm", {})["ollama_host"] = ollama_host.rstrip("/") data_dir = os.environ.get("CLIPS_STUDIO_DATA_DIR") if data_dir: config.setdefault("paths", {})["data_dir"] = data_dir config.setdefault("paths", {})["data_dir"] = str(resolve_data_dir(config)) # Where a cloud backend finds the user's own API key (llm/providers/keys). # The path only; the key itself never enters the config. config.setdefault("llm", {})["data_dir"] = config["paths"]["data_dir"] return config def _render_worker(args, config: dict) -> int: from remote_render import worker as worker_mod if args.run_job: return worker_mod.run_job(args.run_job) data_dir = Path(config["paths"]["data_dir"]) if args.pair: try: info = worker_mod.pair(data_dir, args.pair[0], args.pair[1]) except worker_mod.PairingError as e: print(f"Pairing failed: {e}") return 1 print(f"Paired with {info['main']} (certificate {info['fingerprint']}; it should match the one " "the main PC shows).") w = worker_mod.Worker(data_dir) try: return w.run() except KeyboardInterrupt: w.stop() return 0 def main() -> int: # LLM titles may contain emoji; Windows consoles often use cp1252 and # would crash the whole pipeline on a mere print(). Degrade gracefully. if sys.stdout and hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8", errors="replace") parser = argparse.ArgumentParser(description="Local AI YouTube Shorts pipeline") parser.add_argument("--config", type=Path, default=CONFIG_PATH) sub = parser.add_subparsers(dest="command", required=True) p_process = sub.add_parser("process", help="Process one video URL end-to-end") p_process.add_argument("url", help="YouTube video URL") p_process.add_argument("--force", action="store_true", help="Reprocess even if already done") sub.add_parser("run", help="Run the automation daemon (RSS monitor + scheduler)") sub.add_parser("status", help="Show processing/scheduling state") p_outro = sub.add_parser( "outro-backfill", help="Add the end card to finished clips that are missing one") p_outro.add_argument("--video", help="Only this video id (default: all)") sub.add_parser("auth", help="One-time YouTube authorization (opens browser)") sub.add_parser("upload", help="Upload today's scheduled clips now") p_serve = sub.add_parser("serve", help="Run the local API for the desktop app") p_serve.add_argument("--port", type=int, default=8765) # Localhost by DEFAULT and on purpose: this API has no authentication, # and it can read and write video files. Only a container needs to bind # 0.0.0.0, where "localhost" means the container itself and nothing on # the host could otherwise reach it. p_serve.add_argument( "--host", default="127.0.0.1", help="interface to bind (default 127.0.0.1; use 0.0.0.0 only inside a container)", ) sub.add_parser( "mcp", help="Talk MCP over stdin/stdout, so an AI agent can drive the engine") p_channels = sub.add_parser("channels", help="Manage monitored channels") ch_sub = p_channels.add_subparsers(dest="channels_command", required=True) p_ch_add = ch_sub.add_parser("add", help="Add a channel by @handle, URL, or ID") p_ch_add.add_argument("channel", help="e.g. @MrBeast, a channel URL, or UC... id") ch_sub.add_parser("list", help="List monitored channels") p_ch_rm = ch_sub.add_parser("remove", help="Stop monitoring a channel") p_ch_rm.add_argument("channel", help="Channel ID, @handle, or URL") p_models = sub.add_parser("models", help="Show installed LLMs and switch between them") p_models.add_argument("action", nargs="?", choices=["use"], help="'use' to switch models") p_models.add_argument("model", nargs="?", help="Ollama model tag, e.g. gemma3:12b") # Remote rendering (remote_render/): this PC renders clips for another # Clips Kitty. Headless on an always-on box, or started by the desktop # app when "Use this PC as a render worker" is on. p_rw = sub.add_parser("render-worker", help="Render clips for another Clips Kitty (remote rendering)") p_rw.add_argument("--pair", nargs=2, metavar=("MAIN_PC", "CODE"), help="pair with a main PC first: its address (host:port) and the code it shows") p_rw.add_argument("--run-job", type=Path, help=argparse.SUPPRESS) # one job, in a child process args = parser.parse_args() config = load_config(args.config) # The YouTube cookies fallback (Settings → Advanced settings), for the # command line and the daemon as the app's engine sets it (server/api.py). from sources import ytdlp_common ytdlp_common.set_cookies(config.get("youtube_cookies"), config["paths"]["data_dir"]) db = StateDB(Path(config["paths"]["data_dir"]) / "state.db") try: if args.command == "process": clips = process_video(args.url, config, db, force=args.force) if clips: print(f"\nDone. {len(clips)} clip(s) created:") for clip in clips: print(f" {clip.path} (score {clip.candidate.score})") return 0 if args.command == "run": run_daemon(config, db) return 0 if args.command == "status": _print_status(db) return 0 if args.command == "outro-backfill": # A repair pass, not part of normal use: clips that missed their # end card (a locked file, or a build from before the feature) get # one without being re-rendered, which would cost minutes each. from video import outro if not outro.enabled(config): print("clips.outro is off in settings.yaml — nothing to do.") return 0 print("Checking finished clips for a missing end card...") s = outro.backfill(db, config, args.video) print(f"\n checked {s['checked']} | added {s['added']} | " f"already had one {s['already']}") if s["missing"]: print(f" {s['missing']} clip(s) listed in the database are no " f"longer on disk") if s["failed"]: print(f" {s['failed']} could not be updated — see above") return 0 if args.command == "auth": from core.paths import resolve_config_file from publish.youtube_shorts import YouTubeShortsPublisher publisher = YouTubeShortsPublisher( # Not Path(...) directly: the shipped value is relative, and a # relative path here resolved against the working directory, # which Electron does not set. See core/paths.py. client_secret=resolve_config_file( config, config["upload"]["client_secret"], args.config ), token_path=Path(config["paths"]["data_dir"]) / "youtube_token.json", ) publisher.authenticate(interactive=True) print("YouTube authorized. Token saved — uploads can now run unattended.") print("Turn on auto-posting by setting auto_upload: true in config/settings.yaml.") return 0 if args.command == "upload": from core.scheduler import upload_scheduled config.setdefault("upload", {})["enabled"] = True # explicit command overrides the flag db.promote_queued_clips(config["upload"]["daily_limit"]) n = upload_scheduled(config, db) print(f"{n} clip(s) uploaded." if n else "Nothing uploaded.") return 0 if args.command == "render-worker": return _render_worker(args, config) if args.command == "serve": import uvicorn from server.api import create_app db.close() # the server manages its own connections if args.host != "127.0.0.1": print(f" WARNING: binding {args.host} — this API has no authentication.") uvicorn.run(create_app(config, args.config), host=args.host, port=args.port) return 0 if args.command == "mcp": # Nothing may be printed here: stdout carries the protocol, and a # stray line makes the client drop the connection. from server.mcp import serve as serve_mcp db.close() # this talks to the running engine over HTTP, not the DB if sys.stdout and hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(encoding="utf-8") return serve_mcp() if args.command == "channels": return _handle_channels(args, db) if args.command == "models": return _handle_models(args, config) finally: db.close() return 1 def _handle_channels(args, db: StateDB) -> int: from sources.youtube import resolve_channel if args.channels_command == "add": print(f"Resolving {args.channel!r}...") try: info = resolve_channel(args.channel) except Exception as e: print(f"Could not resolve channel: {e}") return 1 db.add_channel(info["channel_id"], info["name"]) print(f"Now monitoring: {info['name']} ({info['channel_id']})") print("Start the daemon with: python main.py run") return 0 if args.channels_command == "list": channels = db.list_channels() if not channels: print("No channels yet. Add yours with: python main.py channels add @YourHandle") return 0 for row in channels: print(f" {row['channel_id']} {row['name']}") return 0 if args.channels_command == "remove": target = args.channel if not target.startswith("UC"): from sources.youtube import resolve_channel target = resolve_channel(target)["channel_id"] if db.remove_channel(target): print(f"Remove - PROJECTS.md (1510 bytes)
# Built with Clips Kitty Projects built with or for Clips Kitty are listed in **[Awesome Clips Kitty](https://github.com/ColinGPT9/awesome-clips-kitty)**, a curated list of pipelines, plugins, integrations, apps and tools, in its own repository. It includes the official ones: the Clips Kitty OBS Plugin, the MCP server and agent skill, and the Python SDK. The Clips Kitty Marketplace shows the same list. Clips Kitty runs on your own PC and exposes the same API its app uses. Anything that can make an HTTP request can queue a video, follow its progress and read the clips it made. Start with [docs/API.md](docs/API.md). ## Add your project Add one small YAML file with a pull request. [CONTRIBUTING.md](https://github.com/ColinGPT9/awesome-clips-kitty/blob/main/CONTRIBUTING.md) has the format and the inclusion criteria. A separate app or tool that uses the API or the SDK is "built with Clips Kitty"; a pipeline that runs inside Clips Kitty is a listing, installed from the Marketplace ([Marketplace publishing](docs/developers/marketplace-publishing.md)). Two things matter most: - **It uses the documented API.** Build on the endpoints in [docs/API.md](docs/API.md), not by reading the database or the data folder directly, which change without notice. - **It is upfront about data.** If it sends anyone's videos, clips or accounts anywhere, it says so, and its entry says so too. Add the `clips-kitty` topic to your repository too, so people can find it on GitHub even before it is listed.
- pyproject.toml (2623 bytes)
# Tooling config for the Python engine. There is no build system here on # purpose — Clips Kitty ships as a frozen application, not a pip package. [tool.ruff] # The code was written around 100 columns and a handful of lines run to ~130. # 110 keeps the existing style honest without a reformat that would rewrite # most of the repo and bury real history under whitespace churn. line-length = 110 target-version = "py310" # third_party holds code vendored verbatim from other projects. Linting it # would only invite edits, and an edited copy is no longer the thing that # was audited and licensed. exclude = ["ui", "vendor", "third_party", "build", "dist", "release", "data", "site"] [tool.ruff.lint] select = [ "E", # pycodestyle "F", # pyflakes — unused imports and names, the ones that bite "I", # import sorting "B", # bugbear — real bug patterns, not style opinions "UP", # pyupgrade "C4", # comprehension shortcuts "RUF", # ruff's own checks ] ignore = [ "E501", # line length is handled by the formatter, not the linter "B008", # function call in a default argument — FastAPI's whole style "RUF001", # "ambiguous" unicode: this codebase uses em dashes deliberately "RUF002", "RUF003", # The rest are switched off deliberately, not because they are wrong. # Each would mean editing working code on the eve of a release to satisfy # taste rather than fix a defect. They are worth revisiting afterwards. "B904", # `raise ... from` inside except. Correct, but it is 5 edits in # the API's error handling, which is not where to be careless. "B905", # zip(strict=). strict=True can newly RAISE on length mismatch; # strict=False documents intent but changes nothing. Either way # it is a behaviour decision per call site, not a bulk fix. "B007", # unused loop variable — usually an unpack kept for readability "E741", # `l` as a name. Ugly; renaming is churn with no benefit. "RUF007", # itertools.pairwise over zip "RUF046", # redundant int() cast "RUF059", # unused unpacked variable ] [tool.ruff.lint.per-file-ignores] # Prompts and pipelines print a lot; f-strings in logging calls are fine. "main.py" = ["E402"] # sys.stdout is reconfigured before the heavy imports [tool.ruff.format] quote-style = "double" indent-style = "space" [tool.pytest.ini_options] testpaths = ["tests"] # Quiet by default; -v when you want the list. addopts = "-q --strict-markers" markers = [ "slow: needs real media or a model, skipped in CI", ] - README.md (52260 bytes)
<div align="center"> **English** · [简体中文](docs/readme/README_zh.md) · [日本語](docs/readme/README_ja.md) · [한국어](docs/readme/README_ko.md) · [Türkçe](docs/readme/README_tr.md) · [Русский](docs/readme/README_ru.md) · [Tiếng Việt](docs/readme/README_vi.md) · [ไทย](docs/readme/README_th.md) · [Deutsch](docs/readme/README_de.md) · [Español](docs/readme/README_es.md) · [Français](docs/readme/README_fr.md) · [Português (Brasil)](docs/readme/README_pt-BR.md) · [العربية](docs/readme/README_ar.md) · [Bahasa Indonesia](docs/readme/README_id.md) · [Italiano](docs/readme/README_it.md) · [हिन्दी](docs/readme/README_hi.md) · [বাংলা](docs/readme/README_bn.md) · [Filipino](docs/readme/README_tl.md) · [اردو](docs/readme/README_ur.md) </div> # Clips Kitty: the free, open-source Opus Clip alternative for chatting, gaming and sports, on your own PC <a href="https://apps.microsoft.com/detail/9NB6XT7DSQZZ"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://get.microsoft.com/images/en-us%20dark.svg"> <img alt="Get Clips Kitty from the Microsoft Store" src="https://get.microsoft.com/images/en-us%20light.svg" height="52"> </picture> </a> [](https://github.com/ColinGPT9/clips-studio/releases) [](https://github.com/ColinGPT9/clips-studio/releases) [](LICENSE) [](https://github.com/ColinGPT9/clips-studio/stargazers) [](https://github.com/ColinGPT9/clips-studio) [](CONTRIBUTING.md) [](docs/DOCKER.md) <a href="https://youtu.be/BVlAzSWb-d8"><img src="docs/demo/clips-kitty-gta6-setup-guide.gif" alt="Watch the Clips Kitty setup guide on YouTube: turn a gaming stream into a captioned vertical clip" width="100%"></a> <a href="https://youtu.be/g1dMqrq-YaI"><img src="https://github.com/user-attachments/assets/2a1bdd85-723e-46d6-b74b-00579c71b6a7" alt="Clips Kitty turning an NBA broadcast into a 9:16 clip that follows the ball" width="100%"></a> ### Star and share Clips Kitty ⭐ Goal: 550 [](https://github.com/ColinGPT9/clips-studio/stargazers) **of 550** **Please star the repo and share it.** It costs you nothing, and there is no marketing budget behind this, so word of mouth is the only way the next small YouTuber or streamer finds out a free option exists. **I made it for YouTubers, Twitch streamers, and content creators** who cannot afford the other options or an editor. It is a free Opus Clip, Vizard and Klap alternative that runs **Google Gemma** on your own PC: no server cost, no per-clip fee, no AI usage fee. > **Clips Kitty was called Clips Studio until version 1.1.3**, renamed to meet > Microsoft Store naming requirements. Same app, same repository, same data. <img src="docs/brand/mascot.png" alt="" width="150" align="right"> **Turn long streams and videos into ready-to-post Shorts, Reels, and TikToks, entirely on your own machine.** Clips Kitty is a free, open-source AI clip generator and video editor for creators: paste a YouTube, Twitch, or Kick link and it finds the best moments, crops them to 9:16 with the speaker kept centred, burns in word-synced captions, and writes titles, descriptions, and hashtags. No cloud AI by default. No subscription. No per-clip fees. Your footage never leaves your computer, unless you ask it to publish a clip to your own YouTube channel. Local first, cloud when you need it: if your PC is too old to run the AI, a cloud provider runs it instead, billed to your own account. **OpenRouter is the recommended cloud path**: sign in with your OpenRouter account (OAuth, no key to copy) for many models (Claude, GPT, Gemini, Muse Spark and more). In my testing, a 2-hour video cost about 11 cents on OpenRouter, with Gemma (Gemma 4 26B-A4B) picking the clips and Whisper (large-v3-turbo) transcribing; what you pay depends on the models you pick and the length of the video. OpenAI, Claude, Gemini, xAI, Meta, DeepSeek and Qwen also work directly, on your own API key. See [docs/AI-BACKENDS.md](docs/AI-BACKENDS.md). **Why it exists:** most creators growing a channel are doing all of it themselves: filming, streaming, editing, posting. Clipping is how people find you and it is usually the first thing that gets dropped: an editor is a cost most channels cannot justify yet, clipping tools charge per video or per month, and cutting them by hand takes a day you needed elsewhere. This runs on hardware you already own, so it costs nothing to use and there is no cap on how many clips you make. <img src="docs/store-screenshots/fs-dashboard-202754.png" width="100%" alt="The Clips Kitty dashboard: paste a YouTube, Twitch or Kick URL, pick caption and format options, and a finished video row showing 38 clips produced in 19m 38s, with live CPU, RAM, GPU and storage readouts beside it."> <sub>38 clips from one video in 19 minutes, on one PC, with nothing uploaded.</sub> <a href="docs/GAMING.md"><img src="docs/store-screenshots/gaming-layout.png" width="100%" alt="Clips Kitty's Gaming / Reaction layout picker, with adjustable Webcam and Game boxes, layout presets, and a vertical clip preview"></a> <sub>Frame your webcam and game, then preview the vertical clip before generating it.</sub> --- ## Contents - [What it does](#what-it-does) - [How it compares to Opus Clip, Klap, Vizard and CapCut](#how-it-compares-to-opus-clip-klap-vizard-and-capcut) - [How it works](#how-it-works) - [Features](#features) - [What it's built for](#what-its-built-for) - [Requirements](#requirements) - [Install and run](#install-and-run) - [Pick your AI model](#pick-your-ai-model) - [Tested hardware and performance](#tested-hardware-and-performance) - [Supported platforms](#supported-platforms) - [Supported languages](#supported-languages) - [Publish to YouTube](#publish-to-youtube) - [Publish to every platform at once](#publish-to-every-platform-at-once) - [GPU acceleration](#gpu-acceleration) - [Command line use](#command-line-use) - [Write a plugin](#write-a-plugin) - [Build on it](#build-on-it) - [Architecture](#architecture) - [Contributing](#contributing) - [Roadmap](#roadmap-detail) - [License](#license) --- ## What it does You give Clips Kitty a long video. It gives you back a folder of finished vertical clips you can post, plus a studio to review and fix them before you do. It is built for the content most clipping tools handle worst: **live streams, IRL and talking-head footage, podcasts, and gym/fitness content**: long, loosely structured videos where the good moments are buried and a transcript alone won't find them. ## How it compares to Opus Clip, Klap, Vizard and CapCut Clips Kitty is an open-source alternative to cloud clipping tools. The difference that matters is where the work happens: your video is never uploaded, so there is nothing to meter and nothing to charge for. | Tool | Price, billed monthly | Free tier | Finds the clips for you | Watermark | Uploads your video | Open source | | ---- | --------------------- | --------- | ----------------------- | --------- | ------------------ | ----------- | | **Clips Kitty** | Free, no paid tier | The whole app | Yes | Never | No, runs on your PC | Yes, AGPL-3.0 | | Opus Clip | $15 Starter, $29 Pro | 1080p, watermarked, expires in 3 days | Yes | On the free tier | Yes | No | | Klap | $29 Pro, $14 Basic billed yearly | One video | Yes | Not stated | Yes | No | | Vizard | $29 Creator, $14.50 billed yearly | 60 credits a month, 720p | Yes | On the free tier | Yes | No | | quso.ai | $29 to $49 | 75 credits a month, 720p | Yes | Not stated | Yes | No | | Ssemble | $15 to $60, $6 to $24 billed yearly | None listed | Yes | Not stated | Yes | No | | Descript | $16, $24, $50 | 60 minutes a month, 720p | Partly, it is a transcript editor | On Free and Hobbyist | Yes | No | | CapCut | Free to use, paid tiers at capcut.com | The whole editor | **No, you cut by hand** | Varies by feature | No on the desktop app | No | **CapCut is the one worth understanding**, because it is the free answer everyone gives. It is a good editor and it costs nothing, but you find every moment and crop every clip yourself. Clips Kitty is also free, and that first pass is the part it does for you. **Opus Clip, Klap or Vizard are the better choice** if you don't have a Windows PC with a graphics card, you clip occasionally, or you need to work from a phone. Clips Kitty is built for clipping at volume on hardware you already own. **[Full comparison of all seven, with what each one is best at](https://colingpt9.github.io/clips-studio/alternatives.html)** <sub>Prices checked on each company's own pricing page, 17 September 2026. CapCut publishes no pricing page we could read, so no price is quoted for it. Plans change; this table won't update itself.</sub> ## How it works ```mermaid flowchart TD W["Watched channel<br/>YouTube's RSS feed spots a new video"] --> I I["Video input<br/>YouTube · Twitch VOD · Kick VOD · local file"] --> T T["Transcription<br/>faster-whisper, word-level timestamps<br/>local, or OpenRouter, signed in with OAuth"] --> A A["AI video analysis<br/>audio · visual · reaction signals in 1-second bins"] --> S S["Clip detection and scoring<br/>an LLM scores the transcript with those signals<br/>local (Ollama) by default, or OpenRouter, signed in with OAuth"] --> E E["Video editing<br/>9:16 crop on whoever is speaking (YOLOv8 + TalkNet),<br/>on the ball for a match, webcam + game for Gaming / Reaction"] --> C C["Captions<br/>word-synced, burned in, fully editable"] --> M M["Multilingual<br/>translate, subtitle or dub into 19 languages"] --> X X["Export<br/>organized folders, clean filenames, metadata"] --> P P["Publishing<br/>YouTube, TikTok, Instagram: now, on a schedule,<br/>or hands-off for a watched channel"] ``` **Every stage runs on your computer unless you choose otherwise:** OpenRouter, signed in with your account, for the AI or the transcription, and publishing, which uploads only the clips you send. Otherwise the only thing that touches the network in a normal run is downloading the source video. ### The stages in detail **Transcription: faster-whisper.** Word-level timestamps, running on CUDA where available and falling back to CPU. Word timing is what makes captions land on the right syllable and clips start on a real sentence boundary. **AI clip scoring.** The transcript is one signal among five. Audio gives loudness spikes, burst density (what laughter and applause look like), and silence→explosion payoff shapes. Visual gives scene cuts, motion, and face-area changes. A reaction signal fuses those. Each is normalized *within that video*, so a quiet podcast and a screaming stream both produce meaningful peaks. Where real audience data exists (Twitch chat replay measured by unique chatters, YouTube's most-replayed heatmap), it adds a small capped bonus. The local LLM sees the transcript **plus a timeline of those events**, so it can weigh "the words are mild but the room exploded" correctly. Because small models score everything in a narrow band, the finalists then get compared head-to-head in one rerank call: relative judgment is much easier for a 7B model than absolute scoring. **Creator Profiles.** The app builds a knowledge base per creator across their videos: recurring topics, series, collaborators, running jokes, and ongoing storylines. That context helps it write accurate titles and spot callbacks. It is deliberately conservative: a catchphrase has to actually repeat before it counts, knowledge that stops being mentioned goes dormant, and every score contribution from it is additive, capped, and can be switched off. **Video editor.** Non-destructive: trims, internal cuts, mutes, muted words, volume, fades, speed, hook text, music, and watermark are stored as operations and applied at render time. There's an AI edit chat too (*"make it 5 seconds longer"*, *"the caption says gost, it should say ghost"*), where the model proposes and validated code applies. **Long-form processing.** An opt-in 16:9 path using the same analysis: horizontal clips, X/Twitter-length cuts, a best-of highlight reel, or the full stream with dead air removed. **Multilingual pipeline and AI dubbing.** Captions are translated and shown for review *before* anything is written or burned, so a bad line gets fixed while it's still fixable. A glossary protects channel names, sponsors, and in-jokes from being translated. Dubbing uses local TTS with auditionable voices. **Local GPU acceleration.** CUDA for detection and transcription, hardware video encoding via NVENC/AMF/QSV (auto-detected, with CPU fallback), and the LLM on GPU through Ollama. ## Features - **Multimodal clip detection**: moments scored 0–100 by fusing what's said, audio excitement, visual activity, on-screen reactions, and hook/payoff strength. Every clip clearing the quality bar is kept, with no arbitrary cap. - **Speaker-aware face tracking**: YOLOv8 pose detection keeps the subject centred, and in group footage the camera follows **whoever is actually speaking**, decided by TalkNet active-speaker detection from the face and the audio together rather than from movement. When the speaker changes the framing **cuts** to them instead of panning across, the way an editor would. Crop-only framing: never stretched, never distorted. - **Podcast mode**: for multi-camera footage, each shot gets its own steady crop on whoever TalkNet hears talking, changing on the very frame the camera cuts, with no panning. In a long shot it cuts to a new speaker who talks for a second or more; a quick laugh doesn't. - **Vertical Live**: for a livestream that was already vertical when it was streamed (a YouTube vertical live, Twitch Dual Format, Streamlabs Dual Output, a downloaded Instagram or TikTok live). Keeps the stream's own 9:16 layout, skips face tracking, and still picks the moments the usual way. See [docs/VERTICAL-LIVE.md](docs/VERTICAL-LIVE.md). - **Sports: Soccer and Basketball**: every big moment of a match or game as its own clip, with its build-up and reaction. - **Soccer**: goals, saves, cards and big chances. Goals are read off the score box and dated by the crowd, and the 9:16 crop follows the ball. Pick All goals, Best saves, Cards and more, and have the moments joined into a match recap and team or player reels. - **Basketball** - requirements-build.txt (256 bytes)
# Build-only dependencies — needed to PRODUCE the Windows installer, not to # run Clips Kitty. Install these on the build machine: # # pip install -r requirements-build.txt # # The app's own dependencies are in requirements.txt. pyinstaller>=6.21.0
- requirements-chatgpt.txt (574 bytes)
# Optional: the ChatGPT plan sign-in (Settings → AI → OpenAI / ChatGPT plan). # # OpenAI's Codex SDK, which pins its own runtime (openai-codex-cli-bin, about # 400 MB installed). Kept out of requirements.txt on purpose: it isn't in the # installer or the Microsoft Store build yet, and without it the option simply # doesn't appear. Running from source: # # pip install -r requirements-chatgpt.txt # # The SDK is 0.x and moves fast, so the ceiling is the next minor: raise it on # purpose, after signing in and clipping a video with it. openai-codex>=0.157.0,<0.158
- requirements-narration.txt (109 bytes)
# Optional, CPU-only host narration. Install in a separate environment. kokoro-onnx==0.6.1 soundfile==0.13.1
- requirements.txt (4365 bytes)
# Floors AND ceilings, on purpose. # # These were all bare `>=` once, which quietly means "whatever is newest on # the day someone installs". That is not a stable base for a release: at the # time of writing a fresh install pulled opencv-python 5.x and psutil 7.x, # two major versions nobody had ever run the app against. # # The upper bounds pin the MAJOR only. Patches and minors still flow in, so # real fixes are not blocked; a new major is a deliberate decision with a # real video tested afterwards, not something a new contributor discovers by # accident. # # Raising one of these ceilings is a normal thing to do. Do it on purpose, # and process an actual video before merging it. # Extractors break whenever a site changes, so this is the one dependency # you always want newest. No ceiling. yt-dlp>=2026.8.19 # YouTube's player JavaScript solver scripts for the YouTube cookies fallback # (run by the bundled QuickJS, scripts/fetch_quickjs.py). Bump with yt-dlp: it # expects the version in yt_dlp/extractor/youtube/jsc/_builtin/vendor/_info.py. yt-dlp-ejs==0.8.0 # ---- transcription and vision -------------------------------------------- # MFCC features for the active-speaker model. Tiny and pure Python; the # model was trained against this library's defaults, so it is not # interchangeable with another MFCC implementation. python_speech_features>=0.6,<1 faster-whisper>=1.2.1,<2 # faster-whisper's engine (CTranslate2) is a CUDA 12 build and needs cuBLAS 12 # on an NVIDIA GPU; PyTorch's CUDA 13 build only has cuBLAS 13. Without this, # Whisper falls back to the CPU (issue #111). Windows only: that's where the # app ships, and the Linux image runs on the CPU. nvidia-cublas-cu12>=12.4,<13 ; sys_platform == "win32" ultralytics>=8.4.116,<9 # <5 matters. opencv-python 5.0 is out and pip takes it without this. opencv-python>=4.14.0.94,<5 # On-screen text in a gaming stream (analysis/game_text.py): PaddleOCR's # models on the onnxruntime Piper already brings. 1.4 carries its models in # the wheel; the later "rapidocr" package downloads them at first use, which # an installed copy must never do. rapidocr-onnxruntime>=1.4.4,<1.5 # Runs the two voice models behind the second speaker's caption colour # (analysis/voice_turns.py). Already installed as a dependency of # faster-whisper and Piper; pinned now that it is used directly. onnxruntime>=1.26,<2 # ---- API server ------------------------------------------------------------ fastapi>=0.141.1,<1 uvicorn[standard]>=0.52.4,<1 # ---- everything else ------------------------------------------------------- requests>=2.34.2,<3 # Connected Accounts uses the OS keychain outside Windows. On Windows it goes # through core/secrets (DPAPI), so keyring is neither installed nor bundled # there (clips-studio.spec excludes it). Fails closed without an unlocked # keychain; never plaintext. keyring>=25.6,<26 ; sys_platform != "win32" # Desktop Twitch EventSub uses a host-owned WebSocket, not a public webhook. # Already installed through uvicorn[standard] and yt-dlp; 16 is what the # build PC runs, so the ceiling sits above it rather than downgrading it. websockets>=14,<17 # Remote rendering's gateway makes its own HTTPS certificate (remote_render/tls.py). # Already installed as a dependency of google-auth; pinned now that it is used directly. cryptography>=48,<49 # Parses YouTube's RSS feed without expanding entity declarations. defusedxml>=0.7.1,<1 PyYAML>=6.0.3,<7 google-api-python-client>=2.198.0,<3 google-auth-oauthlib>=1.4.0,<2 # 0.x: a minor bump here can break, so the ceiling is tighter than the rest. curl_cffi>=0.15.0,<0.16 # <6, not <8: 6 and 7 exist but have never run here. Same rule as the rest — # the ceiling sits at the major that is actually tested. psutil>=5.9.8,<6 nvidia-ml-py>=12.575.51,<14 # ---- AI dubbing -------------------------------------------------------- # Bundled since 1.1.3. It was previously left out on purpose so an installed # copy simply reported dubbing unavailable, because the code shelled out to # `sys.executable -m piper` — which is api.exe in a frozen build and cannot # run `-m module`. multilingual/dub.py now calls Piper's Python API # in-process, so it can ship. # # Voices are NOT bundled: ~60 MB each across 15 languages, downloaded on # demand into data/voices/ the first time a language is dubbed. piper-tts>=1.5.0,<2
- ROADMAP.md (3683 bytes)
# Roadmap Where Clips Kitty is going, and what is already done. Anything marked shipped is in a release you can install today, not a plan. --- ## Now: stable releases Clips Kitty is a stable release. Below is what has shipped and what is still outstanding. **Shipped** - Windows installer that carries everything: the app, the engine, FFmpeg, the AI runtime, and the tracking and transcription models - GitHub releases with notes written for creators - [Changelog](CHANGELOG.md) and [Known issues](KNOWN-ISSUES.md) - Bug reports and feature requests: issue templates, plus an in-app Feedback Hub that needs no GitHub account - GitHub Discussions - Security scanning: CodeQL, Dependabot, secret scanning, a [security policy](SECURITY.md) - A container image for anyone who wants to work on the engine without installing PyTorch - Website with per-platform pages, sitemap, Open Graph and structured data **Outstanding** - Code signing, so Windows stops warning that the installer is unsigned - Screenshots and a demo video on the website - Progress reporting for the scoring stage and for update downloads, so working cannot be mistaken for hung - Automated Windows builds, so a dependency bump is provably safe rather than hoped-for ## Next: community Making it worth someone's time to contribute. - Good first issues and help-wanted issues with enough context to pick up cold - Contributor recognition - Example workflows, community showcases, performance benchmarks - Voting on feature requests Contributions are wanted in features, bug fixes, documentation, translations, AI improvements and performance work. Start with [CONTRIBUTING.md](CONTRIBUTING.md) and [ARCHITECTURE.md](ARCHITECTURE.md). ## Later None of this holds up a release. - **More sports**: Soccer and Basketball are in; Cricket is next (the Sport menu already lists it as coming soon) - **Text updates when things finish**: a text message when a video's clips are ready, when they're posted, and when something fails - **Reaction videos that understand the video being reacted to**, not just the words spoken over it (the split layout is done: Gaming / Reaction) - **Android companion app, Vertical Live only**, so it stays lightweight: Twitch, Kick and local files, within Google Play's policies - Plugin architecture and community extensions - Creator analytics - More models, better AI workflows, more platforms where they make sense --- ## Long-term Clips Kitty is a **local-first** creator platform. That word carries the whole design: your footage, your workflows and your data stay on your machine, and the app gets better as your hardware does rather than as someone's subscription tier does. Consumer AI hardware is improving quickly. The architecture is kept modular so newer models and faster hardware can be adopted without a redesign. As machines like the NVIDIA RTX Spark and future AI-accelerated GPUs arrive, the same application should be able to run larger local models, infer faster, analyse video better and automate more. The goal is an open-source creator platform that becomes more capable over time because local AI hardware does. ## What success looks like - Creators can install it easily - It is stable and reliable - Developers can understand the codebase - Contributors can submit improvements confidently - Security follows current GitHub practice - Documentation answers the common questions - The website makes the value obvious - People report bugs, propose features, and send code In that order, because each one depends on the one before it. Reliable software earns a good experience, which earns contributors, which earns a community.
- SECURITY.md (4717 bytes)
# Security Policy ## Supported versions Clips Kitty has not had a stable release yet. Security fixes land on `main`, and the latest commit is the only supported version. Once releases begin, this table will list the supported ones. ## Reporting a vulnerability **Please do not open a public issue for a security problem.** Report it privately through GitHub's [private vulnerability reporting](../../security/advisories/new). That opens a discussion visible only to you and the maintainers. Please include: - what the vulnerability lets an attacker do - the steps to reproduce it - the affected file or component, if you know it - your OS, Python version, and app commit You can expect an acknowledgement within a week. If a fix is needed, we'll agree a disclosure timeline with you and credit you in the advisory unless you'd rather stay anonymous. ## What is in scope Clips Kitty is a **local desktop application**. The interesting attack surface is mostly about untrusted input and local exposure: - **The local API** (`server/`) binds to `127.0.0.1:8765`. Anything that lets a remote or cross-origin page reach it, or that widens that binding, is in scope. - **Path handling**: export folders, media serving, and file imports. Path traversal out of the data directory is in scope. - **Untrusted video and transcript content** reaching a shell, an FFmpeg argument, or a file path. Command injection through a video title or filename is in scope. - **Prompt content reaching disk or the shell.** The LLM's output is treated as untrusted and validated before it is applied; a way around that validation is in scope. - **Credential handling**: YouTube OAuth tokens and `config/client_secret.json`. - **The feedback relay** (`feedback-relay/`) and the diagnostics attached to in-app reports. Diagnostics are redacted before leaving the reporter's machine; a leak of secrets or personal data through that path is in scope and important. - **Dependency vulnerabilities** that are actually reachable from Clips Kitty code. ## What is out of scope - Attacks needing an attacker who already has local code execution or admin rights on the machine. A local desktop app can't defend against that. - Vulnerabilities in Ollama, FFmpeg, yt-dlp, or the AI models themselves: report those upstream. If Clips Kitty *uses* one of them unsafely, that part is in scope. - The quality, bias, or content of AI-generated clips, titles, or translations. Those are bugs or feature requests, not security issues. - Denial of service by feeding the app a deliberately enormous video. - Anything requiring a user to change a default setting to something clearly unsafe. ## Known dependency advisories `npm audit` currently reports findings, most of which are in the tooling that *builds* the installer rather than in anything shipped to a user. [docs/DEPENDENCY-SECURITY.md](docs/DEPENDENCY-SECURITY.md) goes through them one at a time: what ships, what does not, what is reachable from app code, and what is already mitigated. Please read it before reporting one, and do report anything it does not cover. ## A note on what this app does Clips Kitty downloads videos with yt-dlp and processes them locally. It never uploads your footage anywhere unless you ask it to. The only outbound network traffic in a normal run is fetching the source video, an optional Twitch chat-replay request, model downloads you ask for, and (only if you submit one) an in-app feedback report. The exception is publishing. If you switch on **Settings → Publish to YouTube**, connect your own Google account and press Upload, the clip goes from your machine straight to YouTube. That is the whole point of the feature, and it never happens on its own: it is off by default, it needs your own Google Cloud API key, and every upload is one you pressed a button for. Nothing is proxied through a Clips Kitty server, because there isn't one. Two things worth knowing if you turn it on: - **The local API has no authentication** (see above). While an account is connected, anything that can reach `127.0.0.1:8765` can publish to your channel. That is the same trust boundary as the rest of the API (it can already read your footage) but the consequence is more visible here. - **Your credentials stay on your machine.** The OAuth token and your Google client secret are stored with Windows DPAPI, encrypted against your Windows account, so copying the data folder to another PC or user does not carry them over. On Linux and in the Docker image there is no DPAPI, and they fall back to an owner-only (0600) file, which protects against other users on the box, not against someone who already has your account.
- SKILL.md (2836 bytes)
--- name: clips-kitty description: Turn long videos and streams into short vertical clips with Clips Kitty, running locally on this computer. Use when someone wants clips, Shorts, Reels or TikToks made from a YouTube, Twitch or Kick link or a video file, or wants to read back or export clips Clips Kitty already made. --- # Clips Kitty Clips Kitty finds the moments worth posting in a long video, crops them to 9:16 with the speaker kept in frame, burns in captions and writes titles. Everything runs on the user's own machine: no account, no upload, no API key. ## Connecting The tools come from Clips Kitty's MCP server, which talks to the engine on `127.0.0.1:8765`. ```bash claude mcp add clips-kitty -- python main.py mcp # from a source checkout ``` In an installed build the engine's binary is `api.exe`, inside the app's `resources\backend` folder, so the command is `api.exe mcp`. **Clips Kitty has to be running**, either the app itself or `python main.py serve`. If a tool reports the engine is not answering, say so and stop: starting it is the user's call, not something to work around. ## How to work 1. **Queue** with `queue_video` (a link) or `queue_local_file` (a file on disk). 2. **Check back** with `job_status`. Processing a two-hour stream takes tens of minutes on a good machine. Never sit in a polling loop: report the job id, and check when the user next asks or after a long wait. 3. **Read the results** with `list_clips`, which gives each clip's id, score and title. 4. **Export** with `export_clip` when the user names a folder. ## Things that will otherwise catch you out - **A queued job is not guaranteed.** `queue_video` returns "Not queued" when that video was processed before, because redoing it costs an hour and duplicates its clips. Offer `list_clips` on the existing video, or `force` if they really meant it. - **An unknown video id looks like a video with no clips**, because the API returns an empty list either way. Check `list_videos` before concluding a video produced nothing. - **A paused queue looks like a broken app.** `queue_status` reports it. Check it before telling anyone nothing is happening. - **Scores are the model's ranking, 0-100**, not a promise. The user picks what to post. - **Gaming uses its own framing.** Set `gaming=true` for its webcam/game layout and existing layout detector. Omit `gaming_layout` for automatic detection, or pass the user's selected layout and boxes. An installed game-specific detector is chosen with `pipeline={"id":"publisher/name"}`; discovery and permissions remain in the app. ## What it will not do There is no cloud tier, no upload of the user's footage anywhere, and no posting to social platforms except YouTube, which the user switches on themselves with their own Google key. Do not offer those. - skills/clips-kitty/SKILL.md (2836 bytes)
--- name: clips-kitty description: Turn long videos and streams into short vertical clips with Clips Kitty, running locally on this computer. Use when someone wants clips, Shorts, Reels or TikToks made from a YouTube, Twitch or Kick link or a video file, or wants to read back or export clips Clips Kitty already made. --- # Clips Kitty Clips Kitty finds the moments worth posting in a long video, crops them to 9:16 with the speaker kept in frame, burns in captions and writes titles. Everything runs on the user's own machine: no account, no upload, no API key. ## Connecting The tools come from Clips Kitty's MCP server, which talks to the engine on `127.0.0.1:8765`. ```bash claude mcp add clips-kitty -- python main.py mcp # from a source checkout ``` In an installed build the engine's binary is `api.exe`, inside the app's `resources\backend` folder, so the command is `api.exe mcp`. **Clips Kitty has to be running**, either the app itself or `python main.py serve`. If a tool reports the engine is not answering, say so and stop: starting it is the user's call, not something to work around. ## How to work 1. **Queue** with `queue_video` (a link) or `queue_local_file` (a file on disk). 2. **Check back** with `job_status`. Processing a two-hour stream takes tens of minutes on a good machine. Never sit in a polling loop: report the job id, and check when the user next asks or after a long wait. 3. **Read the results** with `list_clips`, which gives each clip's id, score and title. 4. **Export** with `export_clip` when the user names a folder. ## Things that will otherwise catch you out - **A queued job is not guaranteed.** `queue_video` returns "Not queued" when that video was processed before, because redoing it costs an hour and duplicates its clips. Offer `list_clips` on the existing video, or `force` if they really meant it. - **An unknown video id looks like a video with no clips**, because the API returns an empty list either way. Check `list_videos` before concluding a video produced nothing. - **A paused queue looks like a broken app.** `queue_status` reports it. Check it before telling anyone nothing is happening. - **Scores are the model's ranking, 0-100**, not a promise. The user picks what to post. - **Gaming uses its own framing.** Set `gaming=true` for its webcam/game layout and existing layout detector. Omit `gaming_layout` for automatic detection, or pass the user's selected layout and boxes. An installed game-specific detector is chosen with `pipeline={"id":"publisher/name"}`; discovery and permissions remain in the app. ## What it will not do There is no cloud tier, no upload of the user's footage anywhere, and no posting to social platforms except YouTube, which the user switches on themselves with their own Google key. Do not offer those.