shinygen

Generate, evaluate, and refine Shiny apps using LLM agents in Docker sandboxes

shinygen turns a text prompt into a production-quality Shiny dashboard — generated, screenshot-tested, and quality-scored — all inside an isolated Docker sandbox.


Core Capabilities

Multiple LLM Agents

Leverages Claude Code (via Anthropic), Codex CLI (via OpenAI), Gemini CLI (via Google), and OpenCode (via OpenCode Go) to autonomously write clean and professional app layouts.

Docker Isolation

Generations run in isolated containers using Inspect AI, protecting your host system from side-effects.

Visual Self-Correction

The sandbox agent takes local Playwright screenshots, reviews layout rendering, and dynamically corrects visual styling before submission.

Multi-Judge Evaluation

An external, host-side LLM panel scores code quality, visual design, functionality, and UX, feeding back critiques for refinement.

Iterative Refinement

Runs automated generate-judge-refine cycles until code meets the quality threshold, up to --max-iterations.

Custom Skills Injection

Pass custom guidelines and component directories to enforce branding rules, library preferences, and modern coding standards.

A/B Benchmark Arms

Compare treatment (--skills) vs control (--no-skills) to measure exactly how much your skill rules improve generation scores.

Web Search Fetching

Allows agents to fetch up-to-date documentation and examples from the live web. Can be disabled with --no-web-fetch.


Installation & Requirements

Quick Install

Install shinygen via pip in editable mode:

git clone https://github.com/karangathe/shinygen.git
cd shinygen
pip install -e .

If you plan to use screenshot-based visual evaluation and host-side verification, install the playwright extras:

pip install -e ".[screenshot]"
playwright install chromium
Tip

Docker sandbox images containing R, Python, Playwright, and headless Chromium are pulled and built automatically on your first generation run.

System Prerequisites

Make sure the following dependencies are available on your system before running shinygen:

System Prerequisites
Requirement Details
Python 3.10+ (Recommended: 3.12)
Docker Docker Desktop daemon running
API Key ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, or OPENCODE_GO_API_KEY
export ANTHROPIC_API_KEY='sk-ant-...', export OPENAI_API_KEY='sk-...', export GEMINI_API_KEY='...', or export OPENCODE_GO_API_KEY='sk-...'

Quick Start Guide

You can run shinygen either through the CLI command shinygen generate or via the programmatic Python API.

# Generate a Python-based Shiny app with Claude Sonnet
shinygen generate \
    --prompt "Create a sales dashboard with filters by region and product category" \
    --model claude-sonnet \
    --output ./my-dashboard
# Generate an R-based Shiny app (shiny-r)
shinygen generate \
    --prompt "Create an interactive mtcars data explorer with ggplot2 charts" \
    --model claude-sonnet \
    --framework shiny-r \
    --output ./my-r-dashboard
import shinygen

result = shinygen.generate(
    prompt="Create a clinical trials dashboard with enrollment trends",
    model="claude-sonnet",
    output_dir="./trials-app",
    framework="shiny-python",
    data_csv="./trials_data.csv",
    screenshot=True,
    judge_model="openai/gpt-5.6-luna",
    max_iterations=3,
)

print(f"Generated app path: {result.app_dir}")
print(f"Score: {result.score:.2f} (Passed: {result.passed})")

Evaluate multiple models or prompts sequentially. Create a JSON config file mapping jobs:

batch.json
[
  {
    "prompt": "Create a clinical trials dashboard with enrollment trends",
    "model": "claude-sonnet",
    "output_dir": "./batch/sonnet",
    "csv_file": "./trials_data.csv",
    "screenshot": true,
    "judge_model": "openai/gpt-5.6-luna"
  },
  {
    "prompt": "Create a clinical trials dashboard with enrollment trends",
    "model": "gpt-5.6-luna",
    "output_dir": "./batch/gpt56",
    "csv_file": "./trials_data.csv",
    "screenshot": true,
    "judge_model": "openai/gpt-5.6-luna"
  }
]

Then run the batch config:

shinygen batch --config batch.json

Architecture & Sandbox Deep Dive

The Execution Loop

shinygen orchestrates a closed feedback loop: it runs the coding agent, spins up the application, takes screenshots, rates the visual output via an external LLM, and refines the code iteratively.

%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'Inter, system-ui, -apple-system, sans-serif', 'fontSize': '15px', 'lineColor': '#475569', 'edgeLabelBackground': '#ffffff' }, 'flowchart': { 'padding': 8, 'nodeSpacing': 30, 'rankSpacing': 35, 'curve': 'linear', 'htmlLabels': true, 'useMaxWidth': true }}}%%
flowchart TD
    %% Color Palette Definitions
    classDef input fill:#e0f2fe,stroke:#0284c7,stroke-width:2px,color:#0c4a6e
    classDef setup fill:#f8fafc,stroke:#64748b,stroke-width:2px,color:#334155
    classDef agent fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#581c87
    classDef test fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#78350f
    classDef judge fill:#ffe4e6,stroke:#e11d48,stroke-width:2px,color:#881337
    classDef loop fill:#f1f5f9,stroke:#475569,stroke-width:2px,color:#1e293b
    classDef done fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d

    %% 1. User Input & Setup
    User(["User Request<br/>CSV data + prompt"]):::input
    User --> Setup["shinygen Setup<br/>Python/R, model, flags"]:::setup

    %% 2. Docker Sandbox Generation
    subgraph Sandbox["Docker Sandbox"]
        direction TB
        Skills["Design Skills<br/>layouts, palettes, schema"]:::agent
        Coder["AI Coding Agent<br/>writes Shiny code"]:::agent
        Skills --> Coder
    end
    Setup --> Skills

    %% 3. Automated Health Check & Visuals
    subgraph Verify["Live Verification"]
        direction TB
        Launch["Health Check<br/>boot app, read logs"]:::test
        Snap["Screenshots<br/>full-page capture"]:::test
        Launch --> Snap
    end
    Coder --> Launch

    %% 4. Multimodal AI Judge
    subgraph Quality["Quality Evaluation"]
        direction TB
        Judge["AI Judge<br/>design + code<br/>scores 1 to 10"]:::judge
        Check{"Score OK?"}:::judge
        Judge --> Check
    end
    Snap --> Judge

    %% 5. Self-Healing Refinement Loop or Success
    Feedback["Auto-Fix Loop<br/>retry with feedback"]:::loop
    Success(["Dashboard Ready<br/>app code, screenshots<br/>cost + quality report"]):::done
    Launch -- crashed --> Feedback
    Check -- no --> Feedback
    Check -- yes --> Success
    Snap -- no judge --> Success
    Feedback -. retry .-> Coder

Docker Sandboxing Mechanics

Every generation job occurs inside an isolated Docker sandbox managed via Inspect AI. - Images Used: - Python: ghcr.io/karangattu/shinygen-sandbox-python:latest - R: ghcr.io/karangattu/shinygen-sandbox-r:latest - Isolation Boundaries: The agent cannot see or modify files on your host machine. It can only read and write inside /home/user/project inside the container. - Auto-dependency Installation: On startup, shinygen executes the framework’s install command (pip install ... or install.packages(...)) to ensure all necessary rendering and plotting libraries (like plotly, bslib, faicons, ggplot2) are hot-loaded in the sandbox. - Web Search: The agent has internet access to look up API docs and download styling examples. Disable this by passing --no-web-fetch (which blocks the web search tool).


Visual QA & Validation

Hands-on Agent Self-Evaluation

When --screenshot is enabled, the sandbox agent leverages the visual-qa skill: 1. The agent starts the Shiny app inside the container as a background daemon (using nohup ... &). 2. It runs screenshot_helper.py to capture a Playwright screenshot on http://localhost:8000. By default, it pauses for 7 seconds (configurable) to allow JavaScript/charts to load completely. 3. The agent reviews the screenshot, identifies design flaws (e.g., overlapping text, unstyled charts, missing headers), modifies the code, and takes another screenshot until it is satisfied. 4. It terminates the background daemon before submitting the final draft.

Geographic Task Coverage

The visual-qa v1.3 skill requires a useful map when location materially helps answer the user’s questions and the data contains reliable coordinates or mappable geographic areas. For example, an Airbnb dashboard should let users locate listings and relate neighborhood prices and guest quality to geography. A location column alone does not require a map for an unrelated task, and missing geographic data must not be invented.

The map must provide recognizable geographic context, fit the area of interest, and have enough space to interpret its records or aggregates. Neighborhood bars, a bare latitude/longitude scatter, an unreadable map strip, or markers lost in an overly broad extent do not satisfy a spatial comparison task. A clearly labeled, usable map tab is acceptable when it preserves relevant filter context; a duplicate overview map is not required.

Geographic obstacle Review priority Visual quality ceiling
An applicable map is missing or materially inadequate while other analysis remains useful Medium 7/10
A broken or unusable map blocks the primary spatial workflow High 3/10

Apply the lowest applicable ceiling, including existing render-state and label-readability ceilings, and group related map defects into one task-level finding. These are review rules, not validated predictions of human preference.

For screenshot-only reviews, inspect all supplied views before reporting a missing map. Distinguish absence from the supplied evidence from confirmed absence in a running app, and mark unpictured map tabs and untested filter linkage as unverified. An archived loading state does not establish a persistent runtime failure.

Host-Side Verification & Logging

Once the agent completes its run, shinygen performs a host-side runtime check: - It launches the app on the host machine using a dynamically allocated free port (validated range 1-65535). - It captures startup logs. If the app crashes on launch, shinygen catches the traceback and feeds it directly into the next iteration’s refinement prompt. - It takes host-side Playwright screenshots to pass to the external LLM judge.


Quality Evaluation & Value-Based Scoring

Multi-Judge Panel

If --judge-model is specified, shinygen routes code and host-side screenshots to an external LLM judge (or a panel of judges, if the flag is repeated).

NoteJudge Validation & Retries

The judge model must be fully qualified starting with anthropic/, openai/, or google/ (e.g., openai/gpt-5.6-luna). Unknown model providers will raise validation errors. Judge calls use automatic retry with exponential backoff (up to 3 attempts) for robustness against rate limits.

The judges evaluate the dashboard across four strict criteria:

  1. Requirement Fidelity: Does the app implement all requested tabs, charts, filters, and datasets?
  2. Code Maintainability: Is the code modular, DRY (Don’t Repeat Yourself), cleanly formatted, and easy to modify?
  3. Visual & UX Quality: Evaluates human-perceived visual quality from rendered screenshots — whether the dashboard looks clear, useful, cohesive, trustworthy, and complete at first glance. Key standards include:
    • One-Screen Usefulness (5-Second Rule): The landing view must immediately communicate the subject, 3–4 concise KPIs, a prominent primary visualization, and an actionable analytical story. Extra tabs add depth but cannot repair a weak first impression.
    • Render-State Gate: Visible screenshot evidence overrides code intent. Repeated empty cards, stuck spinners, tiny placeholder marks, or unresolved outputs cap visual_ux_quality at 3 even when surrounding chrome is polished.
    • Layout & Structure: Consistent grid alignment, comfortable spacing, and clear visual hierarchy (primary KPIs prominent, secondary views subordinated) without wasted whitespace or cramped tiles.
    • Typography & Color: Refined type scales (distinct header, section, body, caption weights), readable line-height, restrained color palettes (2–4 semantic accents), and high-contrast elements (WCAG AA).
    • Component & Chart Legibility: Grouped controls with clear labels, value boxes with meaningful icons/units/deltas, zebra-striped tables with aligned numerics, and charts with explicit titles, labeled axes, units, and restrained palettes (no default plotly rainbow or raw unstyled axes).
    • State Polish & Graceful Errors: Handled empty and loading states without raw tracebacks or broken layouts across viewports.
  4. Code Robustness: Graceful error handling (no raw tracebacks visible to users), empty states handled cleanly, and stable reactive bindings.

Visual & UX Quality Calibration Scale

Visual & UX Quality Judge Rubric
Score Assessment
1–3 Broken, loading, or repeatedly empty visible outputs.
4 Confusing, cramped, patchwork, or substantially incomplete dashboard.
5 Mixed visual hierarchy, awkward density, weak chart choice, or poor coherence.
6 Tidy but basic, sparse, weakly prioritized, or visually generic (typical default layout).
7 Clear, trustworthy dashboard with strong information density and legible charts.
8 Strong professional composition, coherent palette, polished components, and immediately useful landing view.
9 Exceptional human favorite: cohesive, information-rich, beautifully styled, and complete.
10 Essentially unreachable (flawless in every dimension with zero possible improvement).

Value-Based Scoring Formula

Instead of raw quality alone, shinygen optimizes for value-adjusted scores to penalize wasteful API calls:

\[\text{value\_score} = \text{quality\_score} - \text{iteration\_penalty} - \text{cost\_penalty}\]

Where: - Iteration Penalty: Deducts \(0.35\) for each extra generation iteration. - Cost Penalty: Deducts \(1.00\) per \(\$1.00\) of API generation cost (capped at a maximum deduction of \(2.50\)).

This formula ensures models that get the dashboard right in the first iteration at lower costs score higher than models that take many expensive correction steps.


Custom Skills Injection & A/B Benchmarks

Injecting Custom Skills

“Skills” represent styling guides, code templates, and coding standards injected into the agent’s workspace. - Standard Skills: shinygen dynamically fetches the official app-authoring skills at runtime and caches the latest successful download: shiny-for-python from posit-dev/py-shiny and shiny-for-r from rstudio/shiny. Set SHINYGEN_PY_SHINY_SKILL_REF or SHINYGEN_R_SHINY_SKILL_REF to use specific tags or commits instead of main. - Custom Skills: You can inject your own guidelines (e.g. company styling themes, database query layers) using --skills-dir <path>.

Creating a Custom Skill Directory

To write your own skill, create a folder with the following structure:

my-custom-skills/
├── SKILL.md                 # REQUIRED: Contains skill description and YAML frontmatter
└── references/              # OPTIONAL: Supporting markdown guides
    ├── cards.md
    └── theming.md

Your SKILL.md must start with a YAML frontmatter block:

---
name: corporate-theme-skill
description: Guidelines for styling dashboards using the corporate HSL color palette.
metadata:
  version: "1.0"
---

# Corporate Styling Guidelines
- Use CSS variables for color branding.
- Avoid absolute positioning.

Inject your custom skill using:

shinygen generate --prompt "Sales App" --skills-dir ./my-custom-skills/

Control vs Treatment Arms (A/B Testing)

You can benchmark skill effectiveness using the control/treatment flags: - Treatment Arm (--skills): Injects the framework skill plus any custom skills. - Control Arm (--no-skills): Bypasses skill loading entirely. The agent receives only the raw prompt.

shinygen tracks the arm used (skills or vanilla) inside the resulting run_summary.json to calculate the lift in quality score.


Model Ecosystem

Model Types

shinygen routes across three distinct tiers of LLMs:

  1. Frontier Models: Run in Docker containers (Claude Code, Codex CLI, Gemini CLI).
  2. OpenCode Go Models: Routed via the OpenCode agent in Docker containers. API calls are authenticated using OPENCODE_GO_API_KEY and target https://opencode.ai/zen/go/v1.
  3. Local Models via LM Studio: Run unmetered local evaluations (e.g., gemma-4-26b-a4b or qwen3.6-27b):
    • Runs entirely on the host (Docker check is bypassed).
    • Bypasses API key verification.
    • Sets pricing to $0.00 so value-based scores are not penalized by token usage.
    • Targets the local server on http://localhost:1234/v1 (configurable via LMSTUDIO_BASE_URL).

Supported Models Reference

The table below outlines the model aliases resolved automatically by shinygen:

Supported Models & Agents
Alias Agent Model ID Provider
claude-opus-5 Claude Code anthropic/claude-opus-5 Anthropic
claude-sonnet-5 Claude Code anthropic/claude-sonnet-5 Anthropic
claude-haiku-4-5 Claude Code anthropic/claude-haiku-4-5 Anthropic
gpt-5.6-sol Codex CLI openai/gpt-5.6-sol OpenAI
gpt-5.6-terra Codex CLI openai/gpt-5.6-terra OpenAI
gpt-5.6-luna Codex CLI openai/gpt-5.6-luna OpenAI
gemini-3.7-flash Gemini CLI google/gemini-3.7-flash Google
gemini-3.7-pro Gemini CLI google/gemini-3.7-pro Google
gemini-3.7 Gemini CLI google/gemini-3.7 Google
glm-5.3 OpenCode openai-api/opencode-go/glm-5.3 OpenCode Go
glm-5.3-flash OpenCode openai-api/opencode-go/glm-5.3-flash OpenCode Go
kimi-k3 OpenCode openai-api/opencode-go/kimi-k3 OpenCode Go
kimi-k2.7-code OpenCode openai-api/opencode-go/kimi-k2.7-code OpenCode Go
deepseek-v4-pro OpenCode openai-api/opencode-go/deepseek-v4-pro OpenCode Go
deepseek-v4-flash OpenCode openai-api/opencode-go/deepseek-v4-flash OpenCode Go
mimo-v2.5-pro OpenCode openai-api/opencode-go/mimo-v2.5-pro OpenCode Go
mimo-v2.5 OpenCode openai-api/opencode-go/mimo-v2.5 OpenCode Go
minimax-m3 OpenCode anthropic/opencode-go/minimax-m3 OpenCode Go (Anthropic Bridge)
qwen3.8-max OpenCode anthropic/opencode-go/qwen3.8-max OpenCode Go (Anthropic Bridge)
qwen3.8-flash OpenCode openai-api/opencode-go/qwen3.8-flash OpenCode Go
grok-4.5 OpenCode openai-api/opencode-go/grok-4.5 OpenCode Go
gemma-4-26b-a4b OpenCode openai/google/gemma-4-26b-a4b-qat Local (LM Studio)
qwen3.6-27b OpenCode openai/qwen/qwen3.6-27b Local (LM Studio)

Pricing Comparison

Pricing per 1 million tokens for commercial models. Note that local models (LM Studio) run at $0.00.

Cost per 1M Tokens
USD pricing by model
Model Input ($/1M tokens) Output ($/1M tokens)
claude-opus-5 $5.00 $25.00
claude-sonnet-5 $3.00 $15.00
claude-haiku-4-5 $1.00 $5.00
gemini-3.7-flash $0.15 $0.60
gemini-3.7-pro $1.25 $5.00
gpt-5.6-sol $5.00 $30.00
gpt-5.6-terra $2.50 $15.00
gpt-5.6-luna $1.00 $6.00
deepseek-v4-pro $1.74 $3.48
deepseek-v4-flash $0.14 $0.28
glm-5.3 $1.40 $4.40
glm-5.3-flash $0.20 $0.80
kimi-k3 $0.95 $4.00
minimax-m3 $0.60 $2.40
qwen3.8-max $2.50 $7.50

Data Inputs Configuration

shinygen provides convenient mappings to inject CSVs or custom files:

Data Input Options
Option CLI Options Python API Keys Notes
Single CSV --csv-file data_csv Convenient shorthand; is mapped directly by basename.
Multiple files --data-file (repeatable) data_files Accepts any text-based data format (JSON, TXT, CSV).
Note: If both options supply the same file name, --csv-file takes precedence.

Environment Variables

Set the following variables in your terminal or a .env file at the project root to authenticate:

Environment Variable Purpose Example
ANTHROPIC_API_KEY Authenticate Anthropic models (e.g. Claude Sonnet/Opus) export ANTHROPIC_API_KEY='sk-ant-...'
OPENAI_API_KEY Authenticate OpenAI models (e.g. GPT-5.6) export OPENAI_API_KEY='sk-...'
GEMINI_API_KEY Authenticate Google Gemini models (e.g. Gemini 3.7 Flash) export GEMINI_API_KEY='AIzaSy...'
OPENCODE_GO_API_KEY Authenticate OpenCode Go models export OPENCODE_GO_API_KEY='sk-...'
LMSTUDIO_BASE_URL Base URL endpoint for local LM Studio servers export LMSTUDIO_BASE_URL='http://localhost:1234/v1'

Local Development & Testing

We maintain a complete suite of unit tests. Run them locally to verify sandbox configurations, config parsers, and API bindings:

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

DashSwipe: Interactive Dashboard User Testing

To help align dashboard styles, this repository includes DashSwipe, an interactive user-testing survey application located in the user_testing_app/ directory.

  • Purpose: Collects human feedback on LLM-generated layouts. It serves screenshots anonymously, lets users tier-rank layouts (S/A/B/C/D/F), prompts for 2-word critiques, and writes evaluation statistics directly to a Supabase database.

  • Run Locally:

    python3 -m http.server 8080 --directory user_testing_app

    Open http://localhost:8080/ in your browser.

  • Database Schema: Refer to user_testing_app/README.md to set up the Postgres schema.


License

This project is licensed under the MIT License.