Shadow Web

PyPI version PyPI downloads CI Python License: MIT Stars MCP Badge

Shadow Web

Compress web pages 4–6× before your LLM sees them — then extract structured data.
Open-source Python SDK + MCP + standalone shadow-web exec agent: flattens Shadow DOM, builds a typed Action Map, indexes rendered text on demand, heals broken selectors, and turns tables/forms/lists into clean JSON — Camoufox by default, no cloud required.

v0.4.4 — the extraction core moved out of the tree: tables/forms/lists are parsed by the standalone schemasnap package (installed automatically by pip install shadow-web), and shadow_web.schema_snap is a 64-line facade over it — same imports, one implementation. Running this suite against the shared core surfaced four correctness bugs in it (phantom empty tables, a swallowed ARIA header row, dropped single-column grids, zero-padded ids rewritten as integer) — all fixed, all in 0.1.9. 0.4.3 was the verified-review pass — the scanner finally reaches script/link/integrity (raw_html), field classification no longer routes profile values into the wrong field, MCP sessions cannot recycle keys, and an agent_run timeout actually stops the work.

from shadow_web.compressor import process_html

clean_html, actions, groups = process_html(raw_html)
# ✅ actions = [{"id":"1","type":"button","label":"Buy Now","group":"Checkout"}, ...]
# ✅ 164 → 46 tokens on a typical page

from shadow_web.schema_snap import parse_page

data = parse_page(clean_html)
# ✅ tables: [{columns, types, rows, total_rows}]
# ✅ forms:  [{action, method, fields: [{name, type, required, label}]}]
# ✅ lists:  [{type, items, total}]

---

Pain point

AI agents need to see web pages. But raw HTML is full of <script>, <style>, inline CSS, and interactive elements buried in Shadow DOM trees that Playwright can't reach. A typical Wikipedia page costs 99K tokens raw. Your LLM bill doesn't need that.

Shadow Web is what runs between the browser and the LLM: a compression layer that keeps only what matters — interactive elements, their labels, and a clean DOM skeleton. SchemaSnap then takes that clean HTML and turns it into structured data agents can actually use: table rows, form fields with validation, list items.

---

What you get

Feature	Raw HTML	Playwright locators	Shadow Web
Token cost (Wikipedia)	99,343	—	16,462 (−83%)
Token cost (GitHub Trending)	167,875	—	37,833 (−77%)
Long-form via Content Index	full page	—	~600t outline → fetch blocks
Shadow DOM readable	❌	❌ partial	✅ flattened
Semantic groups	❌	❌	✅ Login / Cart / Nav
Self-healing selectors	❌	❌	✅ local + LLM fallback
Consent + lazy hydration	❌	manual	✅ readiness + scroll-until-content
Catalog / feed outline ranking	❌	❌	✅ cards / feeds before chrome
Tables → JSON columns+rows	❌	❌	✅ SchemaSnap
Forms → fields with validation	❌	❌	✅ SchemaSnap
Lists → typed items	❌	❌	✅ SchemaSnap
Works offline	✅	✅	✅
PyPI package	—	playwright	shadow-web

---

Who this is for

You're building …	Why Shadow Web
A browser-based AI agent	Action Map + self-healing + SchemaSnap = fewer failures, structured data
A coding agent that needs one-shot browsing	shadow-web exec — one shell call, one JSON answer (transcript stays inside)
An MCP tool for Cursor/Claude	Built-in MCP server with 27 tools, one-command setup
A Playwright scraper that breaks on every deploy	heal_local.py catches DOM drift without LLM cost
A Shadow DOM-heavy app (Web components, Lit, Angular)	Read-only flatten — no React/Vue breakage
An agent that needs data from web pages	SchemaSnap parses tables, forms, and lists into clean JSON
Long articles, catalogs, social feeds	Rendered Text Index + ranked content_outline (cards / feeds)
Attack surface / security recon	security_scan.py + CLI — forms, links, headers, cookies, page_class (not pentest)
Competitor monitoring & SEO content	Shallow multi-site scans with token-bounded Action Map
SaaS onboarding automation	AgentOps form fill — schema knows fields, LLM only picks values

---

Quick install

pip install shadow-web
playwright install chromium

Extras:

pip install "shadow-web[mcp]"          # Cursor/Claude MCP server
pip install "shadow-web[server]"        # FastAPI heal API
pip install "shadow-web[agent]"         # standalone exec agent loop
pip install "shadow-web[all]"           # everything

Run the optional API locally:

export DEEPSEEK_API_KEY="..."
export SHADOW_WEB_API_KEYS="local-secret"
shadow-web-server

Remote and production requests are rejected unless SHADOW_WEB_API_KEYS is configured.

Standalone agent (shadow-web exec)

Hand a bounded browser task to the built-in observe→act→verify loop. One command in, one JSON object out — browsing transcript stays inside the sub-agent. Same thick entrypoint over MCP: agent_run(goal).

This is for answering a question / short flow — not a full-site scrape. Large volumes stay on thin tools + pagination (schema_session, content_outline(offset=…)). The envelope always reports honesty fields: complete, truncated, stop_reason, optional next_hint.

Unlike a11y-tree browser wrappers, observations are HTML Action Maps + SchemaSnap (works on any HTML, not only Playwright CDP). Default browser is Camoufox (Firefox fingerprint); Chromium is opt-in.

# 30-second demo (Chromium path — no camoufox fetch)
pip install "shadow-web[agent]" && playwright install chromium
export DEEPSEEK_API_KEY="..."   # or OPENAI_API_KEY / OPENROUTER_API_KEY
shadow-web exec "What is the heading on https://example.com?" \
  --browser chromium --model deepseek-chat

# one-liner from a clone:
# bash scripts/demo_30s.sh

# Default production path (Camoufox fingerprint):
# camoufox fetch && shadow-web exec "…" --model deepseek-chat

An explicit --browser chromium needs no extra flag: the Camoufox→Chromium fallback switch (SHADOW_WEB_ALLOW_CHROMIUM_FALLBACK=1) only governs what happens when Camoufox is requested but unavailable, and by default that is a hard failure rather than a silent fallback.

{
  "ok": true,
  "answer": "The page title is \"Example Domain\".",
  "steps": 2,
  "usage": {"prompt_tokens": 1821, "completion_tokens": 144, "total_tokens": 1965},
  "url": "https://example.com/",
  "browser": "chromium",
  "stop_reason": "done",
  "complete": true,
  "truncated": false
}

Flag	Default	Notes
--model	deepseek-v4-flash	Any OpenAI-compatible model id (or SHADOW_WEB_LLM_MODEL)
--base-url	auto from env	DeepSeek / OpenRouter inferred from API key
--api-key	env	Else DEEPSEEK_API_KEY / OPENAI_API_KEY / OPENROUTER_API_KEY
--browser	camoufox	chromium for CDP/a11y supplement path
--max-steps	20	Model turns budget
--max-tokens	15000	Cumulative LLM token budget, 0 = unlimited (or SHADOW_WEB_MAX_TOKENS)
--session	—	Persistent profile ~/.shadow-web/sessions/<name>
--headed	off	Show the browser window
--heal-url	env	Optional self-heal API (SHADOW_WEB_HEAL_URL)
--transcript	off	Include step list in JSON
--jev	off	TypeSafe Jev pre-selects cheap steps (needs TYPESAFE_API_KEY); low confidence still goes to the model
--jev-threshold	0.6	Confidence gate for Jev steps (SHADOW_WEB_JEV_THRESHOLD)

Dense pages (action_count > 40) auto-return minimal observations + a hint to use shadow_query / schema_page / content_outline then done — same terse pipeline as MCP, not site-specific. SPA catalogs with empty SchemaSnap should use Rendered Text Index (content_outline → content_blocks).

Programmatic: from shadow_web import run_agent_task.
Model env help: shadow-web models.

---

Demo

Golden path (recommended — run locally)

Full agent loop with token counts at each step:

pip install -e ".[mcp]"
playwright install chromium
python examples/golden_path/demo.py

Output: raw HTML vs navigate(minimal) + schema_session_json + shadow_query — side-by-side token table.
Playbook: examples/golden_path/CASE.md

Attack surface security scan

Automated surface mapping (not penetration testing): forms, links, page_class, HTTP security headers (HSTS, CSP, XFO, nosniff, CORS), cookie flags (Secure, HttpOnly, SameSite), Markdown/JSON reports.

pip install -e ".[mcp]"
playwright install chromium

# Single URL
python scripts/security_surface_scan.py https://example.com

# Shallow same-domain crawl + reports (--no-headers / --no-cookies to skip checks)
python scripts/security_surface_scan.py https://yoursite.com \
  --crawl-depth 1 --max-pages 20 \
  --json report.json --markdown report.md

Rule engine (importable without browser):

from shadow_web.security_scan import analyze_surface, render_markdown_report

result = analyze_surface(
    "https://app.example.com/login",
    raw_html=await page.content(),  # SRI / mixed content / form fields need the served DOM
    clean_html=clean_html,          # compressor output: labels, action map, tokens
    action_map=actions,
    page_class="Static",
)
# findings: FORM_PASSWORD_GET, HEADER_MISSING_CSP, COOKIE_MISSING_HTTPONLY, ...
# Passing only clean_html still works, but adds SCAN_ON_SANITIZED_HTML: the
# compressor strips <script>/<link>/<meta> and integrity/rel/id/required, so
# SRI_MISSING / MIXED_* / form-field findings would be false negatives.

Example output: examples/security_scan/localpdf-full-report.md (20 pages, 0 critical/high on public marketing layer). Header-only sample: localpdf-headers-only.json.

Does not test: XSS, SQLi, auth bypass, or deep TLS/cipher analysis. Use only on authorized targets.

AgentOps form fill

See the full Form Fill section below. Quick start:

python scripts/form_fill_demo.py https://httpbin.org/forms/post \
  --profile examples/form_fill/profile.json --json plan.json

Competitor intelligence scan

Token-bounded weekly audit for programmatic SEO / compare pages:

python scripts/localpdf_competitor_scan.py --json reports/scan.json

Playbook: examples/localpdf/CASE.md

Smoke test (install + unit tests + one live site):

bash scripts/smoke_install.sh

Compress a page (3 lines)

from shadow_web.compressor import process_html, generate_grouped_xml_map

clean_html, actions, groups = process_html(open("page.html").read())
xml_map = generate_grouped_xml_map("https://example.com", "Example", groups)
print(xml_map)

Playwright + Shadow DOM flatten

from playwright.sync_api import sync_playwright
from shadow_web.wrapper import ShadowPage

with sync_playwright() as p:
    page = p.chromium.launch(headless=True).new_page()
    page.goto("https://example.com")
    shadow = ShadowPage(page)
    _, xml_map = shadow.refresh()
    print(shadow.capture_stats)  # shadow_hosts, iframes, a11y supplement

shadow_grep — send only what the LLM needs

result = shadow.query("intent:login", fmt="terse")
# @1 button Sign in [Login Form]
# @2 input[email] Email [Login Form]

Stream a delta after clicking

shadow.refresh()               # full baseline
shadow.click("3")              # navigate
_, delta_xml = shadow.refresh(diff=True)  # only what changed

SchemaSnap — extract tables, forms, and lists

from shadow_web.schema_snap import parse_page, parse_tables

# Parse everything from a page
data = parse_page(raw_html)
# {
#   "tables": [
#     {
#       "columns": ["Product", "Price", "Stock"],
#       "types": ["string", "currency", "integer"],
#       "rows": [["Widget A", "$19.99", "150"], ...],
#       "total_rows": 12,
#       "column_count": 3
#     }
#   ],
#   "forms": [
#     {
#       "action": "/checkout",
#       "method": "POST",
#       "fields": [
#         {"tag": "input", "type": "email", "name": "email",
#          "required": true, "label": "Email Address"},
#         {"tag": "select", "name": "country", "options": [
#           {"value": "US", "label": "United States"}, ...]}
#       ]
#     }
#   ],
#   "lists": [
#     {"type": "unordered", "items": ["Apples", "Bananas"], "total": 2}
#   ]
# }

# Or export table rows directly
from shadow_web.schema_snap import export_table_json, export_table_csv

records = export_table_json(clean_html, max_rows=50)
# [{"Name": "Alice", "Age": 30}, ...]

csv_text = export_table_csv(clean_html)
# "Name,Age\nAlice,30\n..."

SchemaSnap in MCP

From HTML string (no browser):

Tool	Output
schema_table(html)	columns + types + rows
schema_form(html)	fields + validation
schema_list(html)	ul/ol/standalone select
schema_page(html)	all of the above
schema_json(html)	[{column: value}, ...]
schema_csv(html)	"col1,col2\n..."

From browser session (after navigate / snapshot):

Tool	Output
schema_session()	tables + forms + lists
schema_session_json()	JSON records
schema_session_csv()	CSV string
get_page_html(max_chars=50000)	clean HTML (truncated by default)

All table tools accept max_rows=50 (default). Set max_rows=0 for full data.

---

How it works

Browser (live DOM)
    │
    ├─ capture_ready — cookie consent dismiss + wait + scroll-until-content
    │              ↓
    ├─ [default] DOM capture — flatten Shadow DOM + same-origin iframes (read-only)
    │              ↓
    ├─ [optional] a11y CDP supplement — catch closed Shadow DOM elements
    │              ↓
    ├─ [Chrome 145+] WebMCP bridge — page exposes document.modelContext.getTools()
    │
    └─→ compressor.py → Action Map (data-sid, type, label, group)
                          ↓
                    content_index.py → ranked outline (p0, p1…; cards/feeds) + fetch
                          ↓
                    query.py (shadow_grep) → filter Action Map before LLM
                          ↓
                    schema_snap.py → structured data (tables, forms, lists)
                          ↓
                    security_scan.py → attack surface rules (forms, links, headers, cookies)
                          ↓
                    heal_local.py → fuzzy selector recovery (no LLM)
                          ↓
                    FastAPI /v1/heal → LLM fallback + verification

No live DOM mutation. Shadow Web reads your page; it never writes back. React/Vue/Svelte listeners stay intact.

---

Architecture

shadow_web/
├── compressor.py      # DOM strip + Action Map + semantic groups
├── content_index.py   # Rendered Text Index — outline + on-demand fetch
├── capture_ready.py   # Consent dismiss + readiness wait + scroll-until-content
├── dom_capture.py     # Shadow DOM / iframe flatten (in-browser, read-only)
├── grouping.py        # Semantic groups (forms, nav, modals)
├── schema_snap.py     # Tables, forms, lists → JSON/CSV (facade over the `schemasnap` package)
├── form_fill.py       # AgentOps form fill (auto_fill / ask / handoff)
├── security_scan.py   # Attack surface rules (forms, links, headers, cookies)
├── heal_local.py      # Local selector heal + ~/.shadow-web/heal_cache.json
├── query.py           # shadow_grep (type:, intent:, label~, AND)
├── webmcp.py          # WebMCP bridge (Chrome 145+)
├── diff.py            # Page diff (skeleton + delta XML)
├── a11y_capture.py    # CDP Accessibility dual capture + page_class
├── verified_heal.py   # Playwright selector verification
├── wrapper.py         # ShadowPage (sync Playwright)
├── browser_use.py     # AsyncShadowPage (browser-use / async Playwright)
├── mcp/server.py      # Cursor / Claude MCP tools (27)
└── server.py          # FastAPI (/health, /v1/compress, /v1/heal)

scripts/
├── security_surface_scan.py   # CLI: crawl + JSON/Markdown security reports
├── form_fill_demo.py          # AgentOps form fill demo (plan + execute + wizard)
├── localpdf_competitor_scan.py # Multi-site competitor snapshot
├── smoke_install.sh           # Install + pytest + one live navigate
└── cursor-setup.sh            # MCP one-command setup

deploy/
└── oci.sh                     # Oracle Cloud deployment script

---

Examples

Path	What it demonstrates
examples/golden_path/	Full agent loop + token budget
examples/security_scan/	Attack surface scan reports (JSON + Markdown)
examples/localpdf/	Competitor intel playbook + sample scan
examples/form_fill/	AgentOps form fill playbook + profile sample
examples/browser_use/	browser-use integration

---

Benchmarks

Page	Raw HTML (tokens)	Grouped XML (tokens)	Actions	Reduction
Hacker News	8,637	6,704	227	−22% (1.3×)
Wikipedia (Web Scraping)	99,343	16,462	501	−83% (6.0×)
GitHub Trending	167,875	37,833	1,290	−77% (4.4×)

Full compression benchmark: python benchmarks/run.py

Read this before quoting the numbers. The table measures one thing: raw page HTML → grouped XML Action Map, on three pages, with tiktoken. It is not the token cost of an agent run. The project's own end-to-end agent benchmark (below) measures 1.6× total across 10 tasks, with overhead on 4 of 10 small pages, because the agent's system prompt and tool loop cost more than dumping a small page. The win is real on pages where naive HTML is ≳5–10k tokens, and that is the honest claim.

The Rendered Text Index (content_outline(max_tokens=…)) is a retrieval budget, not a compression ratio: it returns at most the requested number of tokens, so a "99% smaller" framing of it would be true of any budget and therefore says nothing. Use it when you cannot afford the whole page, and read coverage/mode/signals in its response to see what was left out.

Agent Task Benchmark (end-to-end, v0.4.1)

Measured 2026-08-01 with deepseek-chat on 10 bounded public-page tasks.
Fair baseline: Playwright Chromium HTML → one LLM call (same model/goal, HTML capped at 24k tokens).
Shadow Web: shadow-web exec agent loop.
Success uses shared answer checks (expected_contains / expected_any / regex) — not “any non-empty string”.

DEEPSEEK_API_KEY=sk-... python benchmarks/agent_tasks.py
# python benchmarks/agent_tasks.py --only mdn-404 --limit 3

Task	Page type	Baseline LLM tok	SW tok	vs baseline	Steps	Success (BL/SW)
Page heading	static	225	5,652	overhead²	3	✓ / ✓
Wikipedia fact	text-heavy	26,466	12,456	2.1×	5	✓ / ✓
HN top story	table	12,000	9,671	1.2×	4	✓ / ✓
GitHub trending	SPA	25,622	7,176	3.6×	3	✓ / ✓
Form fields	form	24,848	6,756	3.7×	3	✓ / ✓
Books first price	catalog	10,340	6,273	1.6×	3	✓ / ✓
Quotes count	list	2,965	9,042	overhead²	4	✓ / ✓
Login fields	form	750	3,678	overhead²	2	✓ / ✓
MDN HTTP 404	text-heavy	25,350	13,644	1.9×	5	✓ / ✓
Table Due (Smith)	table	1,899	6,233	overhead²	3	✓ / ✓
Total	—	130,465	80,581	1.6× (6/10 wins)	—	10/10

² Tiny / already-small pages: agent system prompt + tool loop costs more than dumping HTML.
Advantage shows on pages where naive HTML ≥ ~5–10k tokens.
Raw JSON: benchmarks/results/agent_tasks_live_20260801_094423_corrected.json.

---

MCP for Cursor / Claude

One-command setup:

bash scripts/cursor-setup.sh

Or manually:

{
  "mcpServers": {
    "shadow-web": {
      "command": "shadow-web-mcp"
    }
  }
}

All 27 tools

Category	Tool	What it does
Browse	navigate	Open URL → snapshot (detail: minimal / terse / xml / full). http/https on the public web only
	snapshot	Refresh page; diff=true for delta only
	click, fill	Interact by data-sid
Agent	agent_run	Thick observe→act→verify loop for one bounded task; always returns an envelope (stop_reason, complete, truncated, next_hint)
AgentOps	form_fill_plan	Profile JSON → fill plan (auto_fill / ask / handoff)
	form_fill_execute	Run plan (+ answers); validation_error feedback
Filter (control plane)	shadow_query	grep-style filter on live session
	query_page	Alias for shadow_query (json output)
	shadow_grep_html	Filter raw/clean HTML without browser
Compress (offline)	compress_html	Strip + Action Map + groups
	compress_html_to_xml	Grouped XML from HTML
Data (SchemaSnap)	schema_table	Table columns + types + rows
	schema_form	Form fields + validation
	schema_list	Lists + standalone selects
	schema_page	All structured data at once
	schema_json	Table → JSON records
	schema_csv	Table → CSV string
	schema_session	Structured data from browser session
	schema_session_json	JSON records from session
	schema_session_csv	CSV from session
	get_page_html	Clean HTML (max_chars default 50000)
Content	content_outline	Ranked rendered-text index (cards / feeds + coverage)
	content_blocks	Fetch selected text blocks by compact IDs
Search	web_search	Brave Search (no API keys)
WebMCP	webmcp_list_tools	Chrome 145+ page tools
	webmcp_execute_tool	Execute WebMCP tool by name

Recommended MCP workflow

Browse + extract:

navigate(url, detail="minimal")     # ~200 tokens — action_count, page_class
schema_session_json(max_rows=50)    # data plane — table records
shadow_query("intent:login")        # control plane — what to click
click(sid) → snapshot(diff=true)    # delta only after action

Long-form content:

navigate(url, detail="minimal")
content_outline(max_tokens=600)              # p0, p1… + coverage/mode/signals/cards|feeds
content_blocks(ids="p3,p7", max_tokens=2000) # selected text only

The Rendered Text Index covers semantic articles and div/span-heavy apps without site-specific selectors. Every readable text node belongs to one bounded block; headings and paragraphs act as boundary signals rather than an allowlist. Before the first capture, readiness dismisses cookie consent and scrolls under-hydrated first paints until content appears. On catalog pages, priced cards are ranked into the first outline page; on feed-like pages, repeating mid-size items are ranked without requiring prices. The summary reports coverage, extraction mode, retained price signals, cards, and feeds so a large reduction cannot hide missing content.

Form fill (AgentOps):

form_fill_plan(profile='{"email":"...","name":"..."}', url="https://...")
form_fill_execute(plan='...', answers='{}')

See examples/golden_path/CASE.md for the full playbook.

---

browser-use Integration

Shadow Web provides out-of-the-box integration with browser-use (the popular agentic framework). It drops token usage by up to 90% and allows the agent to interact with elements inside Shadow DOM and iframes using a single line setup.

pip install "shadow-web[browser-use]"

from browser_use import Agent
from shadow_web import ShadowTools

# Default format is "terse" (compact). Use format="xml" in get_xml_action_map when needed.
tools = ShadowTools(
    heal_api_url="http://localhost:8000/v1/heal",  # Optional: LLM fallback self-healing API
)

agent = Agent(task="...", llm=llm, tools=tools)

get_xml_action_map also accepts query (e.g. intent:login) and format (terse | xml) per call.

See examples/browser_use/ for a complete working implementation.

---

Self-healing chain

click("3") → binding path → element not found?
    ↓
Playwright-native locators (getByRole, getByLabel, getByPlaceholder, getByText) →
  extract stable CSS, cache — cross-browser, no LLM cost
    ↓
local heal (fuzzy label + stable attr match, 85% threshold) → no LLM, no cost
    ↓
LLM heal (DeepSeek / OpenAI via /v1/heal) → generates candidate selector
    ↓
selector verified in headless Chromium → cached to ~/.shadow-web/heal_cache.json

---

SchemaSnap — token-aware data extraction

SchemaSnap is the data plane complement to the Action Map control plane:

The extraction core itself lives in the standalone schemasnap package — it is a runtime dependency, installed automatically by pip install shadow-web. shadow_web.schema_snap (64 lines) re-exports it and maps its source vocabulary back (native → table, aria → aria_grid), so parse_tables/parse_page/schema_* behave exactly as before while fixes land in one place instead of two forks.

Layer	Question	Tools
Control	What can I click?	navigate, shadow_query, click, fill
Data	What data is on the page?	schema_session_json, schema_session, schema_csv

Default max_rows=50 per table. Set max_rows=0 for full export when needed.

Type inference: string, integer, number, currency, percentage, date, email, url.

---

AgentOps Form Fill

Give a URL + JSON profile. Shadow Web reads the form schema, fills safe fields, asks on ambiguity, and hands off on blockers — without sending 50k tokens of HTML to your LLM.

Classic browser agents: entire page → LLM guesses selectors → silent validation failures.
Shadow Web AgentOps split:

Data plane:   schema_form  → { name, type, required, minlength, options }
Control plane: Action Map → fill(sid), click(sid)
LLM job:      profile → values only (not “where to click”)

Execution modes

Mode	When	What happens	Example
auto_fill	Safe, typed profile fields	fill(sid, value) immediately	email, name, company, role, phone
ask	Ambiguous or missing data	Structured question returned in plan.questions	textarea bio, <select> country, “How did you hear about us?”
handoff	Human-only UI	Stop with status: handoff — no blind retries	CAPTCHA, OAuth, file upload, custom datepicker, page_class: Anti-bot

Design principle: never promise full autonomy. Pilots survive because blockers surface early instead of failing silently.

Status codes

status	Meaning
completed	Auto-fills ran (+ optional submit click)
questions_pending	Operator must answer questions[] then re-run with answers
handoff	Human step required (blockers[])
validation_error	Client validation failed after fill — see errors[] with sid + field

Validation feedback loop

After every fill():

1. snapshot(diff=true) — catch appeared error messages
2. HTML5 checkValidity() on live DOM
3. Preflight checks (e.g. email format before fill)

{
  "status": "validation_error",
  "errors": [{
    "sid": "2",
    "field": {"name": "email", "kind": "email"},
    "message": "Please include an '@' in the email address",
    "source": "constraint_validation"
  }]
}

Quick start (Python)

from shadow_web.form_fill import (
    validate_profile,
    plan_from_session,
    apply_question_answers,
    execute_form_fill_plan_async,
    execute_form_fill_plan_multi_step_async,
)

validation = validate_profile(profile)  # warns: unknown keys like "companny"
plan = plan_from_session(
    url=shadow.last_url,
    clean_html=shadow.clean_html,
    action_map=shadow.action_map,
    profile=profile,
    page_class=shadow.page_class,
)

result = await execute_form_fill_plan_async(shadow, plan, validate=True)

# Multi-step wizard (enterprise onboarding)
result = await execute_form_fill_plan_multi_step_async(
    shadow, profile, max_steps=3, answers={"form0_field4": "Senior engineer"}
)

CLI demo

pip install -e ".[mcp]"
playwright install chromium

# Plan only
python scripts/form_fill_demo.py https://httpbin.org/forms/post \
  --profile examples/form_fill/profile.json --json plan.json

# Execute safe fills
python scripts/form_fill_demo.py URL --profile examples/form_fill/profile.json --execute

# Multi-step wizard
python scripts/form_fill_demo.py URL --profile profile.json --execute --multi-step

MCP workflow

form_fill_plan(profile='{"email":"[email protected]","name":"Ada","company":"Acme"}', url="https://...")
form_fill_execute(plan='...', answers='{"form0_field4":"..."}')
# Wizard: form_fill_execute(profile='...', multi_step=true, max_steps=3)

Playbook: examples/form_fill/CASE.md

---

Known limitations

Limitation	Workaround
Anti-bot / Cloudflare headless	page_class: Anti-bot — MCP server uses Camoufox (Firefox fingerprint) to bypass common anti-bot; if still blocked, use headed browser or manual step
Cookie / empty content shell	page_class: SparseShell — readiness + scroll did not hydrate; don't index blindly
colspan / rowspan tables	Column alignment may drift; verify row shape
JS-rendered grids (AG Grid, React Table)	May not use <table> — use shadow_query + Action Map instead
Closed Shadow DOM	navigate(..., capture_mode="dual") or "a11y" (Chromium only; Firefox/Camoufox uses DOM-only fallback)
Cross-origin iframes	Not accessible — page_class: Iframe-heavy
Token bombs	Never default to detail="full", get_page_html(max_chars=0), or max_rows=0 unless debugging
Security scan scope	Surface mapping — forms, links, HTTP headers, cookie flags; no XSS/SQLi; SPAs may need capture_mode=dual
Security scan authorization	Run only on systems you own or have explicit permission to test
Form fill custom widgets	MUI Select, date pickers → handoff; use capture_mode=dual for Shadow DOM forms

---

When NOT to use Shadow Web

• You need one document.querySelector — use Playwright directly.
• You're building a static site scraper with no interaction.
• The page is plain HTML with no Shadow DOM — overhead isn't worth it.

---

License

MIT. Free for anything.

---

Stars are the oxygen of open-source. If Shadow Web saved you tokens or debugging time, ★ the repo.