Analytics Tracking Watchdog
Analytics tracking breaks silently — you find out three weeks later when a chart looks odd. This drives a real browser every night, reads the beacons, and complains before anyone else notices.
- Role
- Monitoring engine, agent, and web app
- Context
- RCS MediaGroup
- Period
- 2026
- Stack
-
- Python
- Playwright
- FastAPI
- React
- Gemini
- Vertex AI
- YAML
Deterministic — decides if an alert fires
- Drive a real browser
- Parse the beacons
- Assert against baseline
- Non-zero exit
LLM — proposes, never asserts
- Onboard a new page
- Triage a failed run
- Edit scenario YAML config only, never engine code
Objective
Analytics tracking is load-bearing for reporting and monetisation, and it fails quietly. A release drops a variable, a consent change reorders a call, a template stops firing a page view — and nothing alerts, because the pipeline downstream is perfectly healthy. It is faithfully processing data that is now wrong.
The goal was to catch that within a day, without a human clicking through pages with the browser dev tools open.
Approach
The engine is deliberately dumb and deterministic: drive a real browser, handle the consent dialog, intercept the analytics beacons, parse them into fields, and assert them against an expected baseline written in YAML. One scenario per flow, per device. Exit non-zero on failure. That part must never be probabilistic, because it is the thing that decides whether an alert fires.
The LLM sits strictly around that core, doing the jobs where judgement helps and errors are cheap: onboarding a new page by browsing it and proposing a scenario, and triaging a failed run into a readable explanation. The agent can read and write scenario YAML and trigger runs — it cannot touch engine code. That boundary is the design.
A second check runs at aggregate level: pull volumes per flow and device from the analytics API and compare them against a seasonal baseline. Field-level assertions catch a broken parameter; only volumes catch a page that stopped reporting entirely.
Architecture
A standalone monitoring package (browser runner, beacon parser, assertion engine, consent handling, storage), a scenario store whose authoritative copy lives in object storage so the agent can edit it without a redeploy, a web app for runs, expected-versus- observed diffs, history and the agent chat, and a watchdog entry point that chains the volume check into an agentic investigation of whatever looks anomalous.
What I owned
All of it: the browser automation, the parser for nested tracking payloads, the assertion engine, the agent tool surface, and the web app.
Outcome
Tracking regressions are now found by a scheduled run instead of by an analyst noticing a weird chart weeks later. The unexpected benefit was the baseline itself: writing down what each page is supposed to send surfaced several inconsistencies that had been live long enough to look intentional.