Home Docs SRE Handbook Incidents Incidents
owner Stefan Coetzee created 2026-10-09 updated 2026-10-09 not reviewed 1 min read
Vault note, not reviewed against the source. Written in the knowledge vault on 2026-10-09 by models working with Stefan Coetzee and published as it stands, with private addresses, e-mail addresses and an employer name redacted. Check claims against the primary source before relying on them.
Incident, cause and mitigation records from the vault, written up from trap files and session logs. Each record names its source.
Child pages Authenticated Cookie Jar for Gated Posts An exported cookies.txt lets yt-dlp list and fetch gated posts. Back Up SQLite With the Backup Command Use sqlite3 .backup, or checkpoint the WAL first, to get a restore point that holds every write. Bracket Pattern Process Check Use pgrep -f "[b]atchtranscribe.py" or the ps | awk form. Compare Videos Found Against the In-App Count The owner's in-app count minus videosfound measures the gap. Compare Videos Found Run to Run First fix for the short scrape: treat any drop in videosfound between runs as suspect. Differential Test in the Same Minutes Run the failing path and its nearest working variants side by side, within the same few minutes, changing one factor at a time. Eliminate Concurrent Saves Before Blaming the Scraper Any 'scrape found N, collection holds M' gap must first rule out that the owner saved something mid-run. Enumeration Retry With Unauthenticated Fallback Enumeration tries the cookie jar twice, then unauthenticated mode three times with backoff, and logs which path succeeded. Expired Bot-Manager Cookie Returns Empty Body yt-dlp sends expired entries from a Netscape cookie jar as they are. File Copy of a WAL-Mode Database Misses Recent Writes A SQLite database in WAL mode keeps recent writes in the -wal file. Filter Expired Cookies Before Each Run cookieargs() writes a filtered copy of the cookie jar with expired entries dropped, logs each drop, and passes the filtered file to yt-dlp. Hand-Check Output Against the Primary Source Compare a sample of generated summaries or extracted figures with the source they came from. Handle-Based Ad Filter Matches Artist Accounts The creator rule \.(de|official)$ was written for brand and shop accounts. Incident 2026-02-27 Dead Collection Short Links Two collections stopped scraping because their stored short links died. Incident 2026-07 Summaries Inverted Irony One sarcastic recommendation was summarised as a sincere warning, one summary named the wrong person, one was selectively pessimistic against its source. Incident 2026-07-14 Backup Missed WAL Writes A pre-change backup copied the main database file only, while the database was in WAL mode with a multi-megabyte WAL file. Incident 2026-07-14 Short Scrape Looked Like a Quiet Feed Six runs in a row reported 19 videos found; a later run reported 32, and the owner's own count was closer than the database's. Incident 2026-07-19 Pending Status on Finished Videos Twelve videos sat at statuspending while holding full transcripts and summaries. Incident 2026-07-21 Ad Filter Removed Artist Videos Ten videos from artists with .official handles were rejected by the ad filter before transcription and looked like unprocessed backlog. Incident 2026-08-08 Empty-Transcript Videos Invisible to Review A reel made of on-screen text returned no transcript. Incident 2026-08-08 False Enumeration Bug Finding A scrape found 46 videos and a re-enumeration minutes later found 48. Incident 2026-09-11 Three-Morning Cron Crash The 06:15 cron died three mornings running with CalledProcessError. Incident 2026-09-13 Ghost Still-Running Job A process check over SSH reported the batch job as still running after it had exited. Incident 2026-09-16 Server Cannot List the Collection From 2026-09-16 the server could not enumerate the collection from any client or egress, while the laptop could with the same yt-dlp version through the same egress. No-Transcript Bucket in the Review Tool Planned: have the review tool surface rows with a NULL review and an empty transcript as their own 'needs a manual look' bucket, so they stop disappearing whether or not OCR ever lands. Probe Pending Rows Before Calling Them Backlog A pending count is unclassified until the rows are looked at. Probe the Artifact Before Reasoning From Counts When a count looks wrong, fetch one real item and read the error. Process Check Matches Its Own Shell pgrep -f <pattern> run over SSH matches the remote shell whose command line contains the pattern. Publish Date Treated As Time In Collection A video's publish date says nothing about when it was saved to a collection. Pull the Full Row List After Any Scrape Do not take the review tool's count as the batch size. Re-Scrape Relinks Processed Rows As Pending Re-scraping a collection re-links videos that were already processed and sets them back to statuspending without flipping them to complete again. Review Tool Lists Only Rows With a Transcript reviewstatus.py reports on rows where length(transcript) > 0. Sensitivity-Gated Posts Vanish From Unauthenticated Listing TikTok refuses to serve age-restricted or sensitivity-flagged posts to an unauthenticated client. Short Link Redirects to the Homepage Stored vm.tiktok.com short links for two collections now redirect to the site homepage. Single Video URL As the Collection Argument A single video URL works as the --collection-url argument, so individual videos can be reprocessed without a working collection link. Small Local Model Misreads Ironic Register A 7B local summarizer renders sarcasm as sincere statement, swaps subjects, or turns selectively pessimistic against its own source. Summaries Are a Triage Index Use summaries to decide what to read, never as evidence. Swallowed Stderr Hides the Cause subprocess.run(..., checkTrue, captureoutputTrue) captures the child's stderr and raises CalledProcessError. Transcriber Incident Register Every failure of the video transcriber pipeline since February 2026, as typed atoms. Verify Transcript Length Never Status Treat status as a report. Check length(transcript) and length(summary) per row. folder Linked from From the knowledge vault, SRE/incidents/ . Built from scripts/docs by build_docs.py .