| System Prerequisites | |
| Requirement | Details |
|---|---|
| Python | 3.10+ (Recommended: 3.12) |
| Docker | Docker Desktop daemon running |
| API Key | ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, or OPENCODE_GO_API_KEY |
export ANTHROPIC_API_KEY='sk-ant-...', export OPENAI_API_KEY='sk-...', export GEMINI_API_KEY='...', or export OPENCODE_GO_API_KEY='sk-...' |
|
shinygen
Generate, evaluate, and refine Shiny apps using LLM agents in Docker sandboxes
shinygen turns a text prompt into a production-quality Shiny dashboard — generated, screenshot-tested, and quality-scored — all inside an isolated Docker sandbox.
Core Capabilities
Multiple LLM Agents
Leverages Claude Code (via Anthropic), Codex CLI (via OpenAI), Gemini CLI (via Google), and OpenCode (via OpenCode Go) to autonomously write clean and professional app layouts.
Docker Isolation
Generations run in isolated containers using Inspect AI, protecting your host system from side-effects.
Visual Self-Correction
The sandbox agent takes local Playwright screenshots, reviews layout rendering, and dynamically corrects visual styling before submission.
Multi-Judge Evaluation
An external, host-side LLM panel scores code quality, visual design, functionality, and UX, feeding back critiques for refinement.
Iterative Refinement
Runs automated generate-judge-refine cycles until code meets the quality threshold, up to --max-iterations.
Custom Skills Injection
Pass custom guidelines and component directories to enforce branding rules, library preferences, and modern coding standards.
A/B Benchmark Arms
Compare treatment (--skills) vs control (--no-skills) to measure exactly how much your skill rules improve generation scores.
Web Search Fetching
Allows agents to fetch up-to-date documentation and examples from the live web. Can be disabled with --no-web-fetch.
Installation & Requirements
Quick Install
Install shinygen via pip in editable mode:
git clone https://github.com/karangathe/shinygen.git
cd shinygen
pip install -e .If you plan to use screenshot-based visual evaluation and host-side verification, install the playwright extras:
pip install -e ".[screenshot]"
playwright install chromiumDocker sandbox images containing R, Python, Playwright, and headless Chromium are pulled and built automatically on your first generation run.
System Prerequisites
Make sure the following dependencies are available on your system before running shinygen:
Quick Start Guide
You can run shinygen either through the CLI command shinygen generate or via the programmatic Python API.
# Generate a Python-based Shiny app with Claude Sonnet
shinygen generate \
--prompt "Create a sales dashboard with filters by region and product category" \
--model claude-sonnet \
--output ./my-dashboard# Generate an R-based Shiny app (shiny-r)
shinygen generate \
--prompt "Create an interactive mtcars data explorer with ggplot2 charts" \
--model claude-sonnet \
--framework shiny-r \
--output ./my-r-dashboardimport shinygen
result = shinygen.generate(
prompt="Create a clinical trials dashboard with enrollment trends",
model="claude-sonnet",
output_dir="./trials-app",
framework="shiny-python",
data_csv="./trials_data.csv",
screenshot=True,
judge_model="openai/gpt-5.6-luna",
max_iterations=3,
)
print(f"Generated app path: {result.app_dir}")
print(f"Score: {result.score:.2f} (Passed: {result.passed})")Evaluate multiple models or prompts sequentially. Create a JSON config file mapping jobs:
batch.json
[
{
"prompt": "Create a clinical trials dashboard with enrollment trends",
"model": "claude-sonnet",
"output_dir": "./batch/sonnet",
"csv_file": "./trials_data.csv",
"screenshot": true,
"judge_model": "openai/gpt-5.6-luna"
},
{
"prompt": "Create a clinical trials dashboard with enrollment trends",
"model": "gpt-5.6-luna",
"output_dir": "./batch/gpt56",
"csv_file": "./trials_data.csv",
"screenshot": true,
"judge_model": "openai/gpt-5.6-luna"
}
]Then run the batch config:
shinygen batch --config batch.jsonArchitecture & Sandbox Deep Dive
The Execution Loop
shinygen orchestrates a closed feedback loop: it runs the coding agent, spins up the application, takes screenshots, rates the visual output via an external LLM, and refines the code iteratively.
%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'Inter, system-ui, -apple-system, sans-serif', 'fontSize': '15px', 'lineColor': '#475569', 'edgeLabelBackground': '#ffffff' }, 'flowchart': { 'padding': 8, 'nodeSpacing': 30, 'rankSpacing': 35, 'curve': 'linear', 'htmlLabels': true, 'useMaxWidth': true }}}%%
flowchart TD
%% Color Palette Definitions
classDef input fill:#e0f2fe,stroke:#0284c7,stroke-width:2px,color:#0c4a6e
classDef setup fill:#f8fafc,stroke:#64748b,stroke-width:2px,color:#334155
classDef agent fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#581c87
classDef test fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#78350f
classDef judge fill:#ffe4e6,stroke:#e11d48,stroke-width:2px,color:#881337
classDef loop fill:#f1f5f9,stroke:#475569,stroke-width:2px,color:#1e293b
classDef done fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
%% 1. User Input & Setup
User(["User Request<br/>CSV data + prompt"]):::input
User --> Setup["shinygen Setup<br/>Python/R, model, flags"]:::setup
%% 2. Docker Sandbox Generation
subgraph Sandbox["Docker Sandbox"]
direction TB
Skills["Design Skills<br/>layouts, palettes, schema"]:::agent
Coder["AI Coding Agent<br/>writes Shiny code"]:::agent
Skills --> Coder
end
Setup --> Skills
%% 3. Automated Health Check & Visuals
subgraph Verify["Live Verification"]
direction TB
Launch["Health Check<br/>boot app, read logs"]:::test
Snap["Screenshots<br/>full-page capture"]:::test
Launch --> Snap
end
Coder --> Launch
%% 4. Multimodal AI Judge
subgraph Quality["Quality Evaluation"]
direction TB
Judge["AI Judge<br/>design + code<br/>scores 1 to 10"]:::judge
Check{"Score OK?"}:::judge
Judge --> Check
end
Snap --> Judge
%% 5. Self-Healing Refinement Loop or Success
Feedback["Auto-Fix Loop<br/>retry with feedback"]:::loop
Success(["Dashboard Ready<br/>app code, screenshots<br/>cost + quality report"]):::done
Launch -- crashed --> Feedback
Check -- no --> Feedback
Check -- yes --> Success
Snap -- no judge --> Success
Feedback -. retry .-> Coder
Docker Sandboxing Mechanics
Every generation job occurs inside an isolated Docker sandbox managed via Inspect AI. - Images Used: - Python: ghcr.io/karangattu/shinygen-sandbox-python:latest - R: ghcr.io/karangattu/shinygen-sandbox-r:latest - Isolation Boundaries: The agent cannot see or modify files on your host machine. It can only read and write inside /home/user/project inside the container. - Auto-dependency Installation: On startup, shinygen executes the framework’s install command (pip install ... or install.packages(...)) to ensure all necessary rendering and plotting libraries (like plotly, bslib, faicons, ggplot2) are hot-loaded in the sandbox. - Web Search: The agent has internet access to look up API docs and download styling examples. Disable this by passing --no-web-fetch (which blocks the web search tool).
Visual QA & Validation
Hands-on Agent Self-Evaluation
When --screenshot is enabled, the sandbox agent leverages the visual-qa skill: 1. The agent starts the Shiny app inside the container as a background daemon (using nohup ... &). 2. It runs screenshot_helper.py to capture a Playwright screenshot on http://localhost:8000. By default, it pauses for 7 seconds (configurable) to allow JavaScript/charts to load completely. 3. The agent reviews the screenshot, identifies design flaws (e.g., overlapping text, unstyled charts, missing headers), modifies the code, and takes another screenshot until it is satisfied. 4. It terminates the background daemon before submitting the final draft.
Geographic Task Coverage
The visual-qa v1.3 skill requires a useful map when location materially helps answer the user’s questions and the data contains reliable coordinates or mappable geographic areas. For example, an Airbnb dashboard should let users locate listings and relate neighborhood prices and guest quality to geography. A location column alone does not require a map for an unrelated task, and missing geographic data must not be invented.
The map must provide recognizable geographic context, fit the area of interest, and have enough space to interpret its records or aggregates. Neighborhood bars, a bare latitude/longitude scatter, an unreadable map strip, or markers lost in an overly broad extent do not satisfy a spatial comparison task. A clearly labeled, usable map tab is acceptable when it preserves relevant filter context; a duplicate overview map is not required.
| Geographic obstacle | Review priority | Visual quality ceiling |
|---|---|---|
| An applicable map is missing or materially inadequate while other analysis remains useful | Medium | 7/10 |
| A broken or unusable map blocks the primary spatial workflow | High | 3/10 |
Apply the lowest applicable ceiling, including existing render-state and label-readability ceilings, and group related map defects into one task-level finding. These are review rules, not validated predictions of human preference.
For screenshot-only reviews, inspect all supplied views before reporting a missing map. Distinguish absence from the supplied evidence from confirmed absence in a running app, and mark unpictured map tabs and untested filter linkage as unverified. An archived loading state does not establish a persistent runtime failure.
Host-Side Verification & Logging
Once the agent completes its run, shinygen performs a host-side runtime check: - It launches the app on the host machine using a dynamically allocated free port (validated range 1-65535). - It captures startup logs. If the app crashes on launch, shinygen catches the traceback and feeds it directly into the next iteration’s refinement prompt. - It takes host-side Playwright screenshots to pass to the external LLM judge.
Quality Evaluation & Value-Based Scoring
Multi-Judge Panel
If --judge-model is specified, shinygen routes code and host-side screenshots to an external LLM judge (or a panel of judges, if the flag is repeated).
The judge model must be fully qualified starting with anthropic/, openai/, or google/ (e.g., openai/gpt-5.6-luna). Unknown model providers will raise validation errors. Judge calls use automatic retry with exponential backoff (up to 3 attempts) for robustness against rate limits.
The judges evaluate the dashboard across four strict criteria:
- Requirement Fidelity: Does the app implement all requested tabs, charts, filters, and datasets?
- Code Maintainability: Is the code modular, DRY (Don’t Repeat Yourself), cleanly formatted, and easy to modify?
- Visual & UX Quality: Evaluates human-perceived visual quality from rendered screenshots — whether the dashboard looks clear, useful, cohesive, trustworthy, and complete at first glance. Key standards include:
- One-Screen Usefulness (5-Second Rule): The landing view must immediately communicate the subject, 3–4 concise KPIs, a prominent primary visualization, and an actionable analytical story. Extra tabs add depth but cannot repair a weak first impression.
- Render-State Gate: Visible screenshot evidence overrides code intent. Repeated empty cards, stuck spinners, tiny placeholder marks, or unresolved outputs cap
visual_ux_qualityat 3 even when surrounding chrome is polished. - Layout & Structure: Consistent grid alignment, comfortable spacing, and clear visual hierarchy (primary KPIs prominent, secondary views subordinated) without wasted whitespace or cramped tiles.
- Typography & Color: Refined type scales (distinct header, section, body, caption weights), readable line-height, restrained color palettes (2–4 semantic accents), and high-contrast elements (WCAG AA).
- Component & Chart Legibility: Grouped controls with clear labels, value boxes with meaningful icons/units/deltas, zebra-striped tables with aligned numerics, and charts with explicit titles, labeled axes, units, and restrained palettes (no default plotly rainbow or raw unstyled axes).
- State Polish & Graceful Errors: Handled empty and loading states without raw tracebacks or broken layouts across viewports.
- Code Robustness: Graceful error handling (no raw tracebacks visible to users), empty states handled cleanly, and stable reactive bindings.
Visual & UX Quality Calibration Scale
| Visual & UX Quality Judge Rubric | |
| Score | Assessment |
|---|---|
| 1–3 | Broken, loading, or repeatedly empty visible outputs. |
| 4 | Confusing, cramped, patchwork, or substantially incomplete dashboard. |
| 5 | Mixed visual hierarchy, awkward density, weak chart choice, or poor coherence. |
| 6 | Tidy but basic, sparse, weakly prioritized, or visually generic (typical default layout). |
| 7 | Clear, trustworthy dashboard with strong information density and legible charts. |
| 8 | Strong professional composition, coherent palette, polished components, and immediately useful landing view. |
| 9 | Exceptional human favorite: cohesive, information-rich, beautifully styled, and complete. |
| 10 | Essentially unreachable (flawless in every dimension with zero possible improvement). |
Value-Based Scoring Formula
Instead of raw quality alone, shinygen optimizes for value-adjusted scores to penalize wasteful API calls:
\[\text{value\_score} = \text{quality\_score} - \text{iteration\_penalty} - \text{cost\_penalty}\]
Where: - Iteration Penalty: Deducts \(0.35\) for each extra generation iteration. - Cost Penalty: Deducts \(1.00\) per \(\$1.00\) of API generation cost (capped at a maximum deduction of \(2.50\)).
This formula ensures models that get the dashboard right in the first iteration at lower costs score higher than models that take many expensive correction steps.
Custom Skills Injection & A/B Benchmarks
Injecting Custom Skills
“Skills” represent styling guides, code templates, and coding standards injected into the agent’s workspace. - Standard Skills: shinygen dynamically fetches the official app-authoring skills at runtime and caches the latest successful download: shiny-for-python from posit-dev/py-shiny and shiny-for-r from rstudio/shiny. Set SHINYGEN_PY_SHINY_SKILL_REF or SHINYGEN_R_SHINY_SKILL_REF to use specific tags or commits instead of main. - Custom Skills: You can inject your own guidelines (e.g. company styling themes, database query layers) using --skills-dir <path>.
Creating a Custom Skill Directory
To write your own skill, create a folder with the following structure:
my-custom-skills/
├── SKILL.md # REQUIRED: Contains skill description and YAML frontmatter
└── references/ # OPTIONAL: Supporting markdown guides
├── cards.md
└── theming.md
Your SKILL.md must start with a YAML frontmatter block:
---
name: corporate-theme-skill
description: Guidelines for styling dashboards using the corporate HSL color palette.
metadata:
version: "1.0"
---
# Corporate Styling Guidelines
- Use CSS variables for color branding.
- Avoid absolute positioning.Inject your custom skill using:
shinygen generate --prompt "Sales App" --skills-dir ./my-custom-skills/Control vs Treatment Arms (A/B Testing)
You can benchmark skill effectiveness using the control/treatment flags: - Treatment Arm (--skills): Injects the framework skill plus any custom skills. - Control Arm (--no-skills): Bypasses skill loading entirely. The agent receives only the raw prompt.
shinygen tracks the arm used (skills or vanilla) inside the resulting run_summary.json to calculate the lift in quality score.
Model Ecosystem
Model Types
shinygen routes across three distinct tiers of LLMs:
- Frontier Models: Run in Docker containers (Claude Code, Codex CLI, Gemini CLI).
- OpenCode Go Models: Routed via the OpenCode agent in Docker containers. API calls are authenticated using
OPENCODE_GO_API_KEYand targethttps://opencode.ai/zen/go/v1. - Local Models via LM Studio: Run unmetered local evaluations (e.g.,
gemma-4-26b-a4borqwen3.6-27b):- Runs entirely on the host (Docker check is bypassed).
- Bypasses API key verification.
- Sets pricing to $0.00 so value-based scores are not penalized by token usage.
- Targets the local server on
http://localhost:1234/v1(configurable viaLMSTUDIO_BASE_URL).
Supported Models Reference
The table below outlines the model aliases resolved automatically by shinygen:
| Supported Models & Agents | |||
| Alias | Agent | Model ID | Provider |
|---|---|---|---|
claude-opus-5 |
Claude Code | anthropic/claude-opus-5 |
Anthropic |
claude-sonnet-5 |
Claude Code | anthropic/claude-sonnet-5 |
Anthropic |
claude-haiku-4-5 |
Claude Code | anthropic/claude-haiku-4-5 |
Anthropic |
gpt-5.6-sol |
Codex CLI | openai/gpt-5.6-sol |
OpenAI |
gpt-5.6-terra |
Codex CLI | openai/gpt-5.6-terra |
OpenAI |
gpt-5.6-luna |
Codex CLI | openai/gpt-5.6-luna |
OpenAI |
gemini-3.7-flash |
Gemini CLI | google/gemini-3.7-flash |
|
gemini-3.7-pro |
Gemini CLI | google/gemini-3.7-pro |
|
gemini-3.7 |
Gemini CLI | google/gemini-3.7 |
|
glm-5.3 |
OpenCode | openai-api/opencode-go/glm-5.3 |
OpenCode Go |
glm-5.3-flash |
OpenCode | openai-api/opencode-go/glm-5.3-flash |
OpenCode Go |
kimi-k3 |
OpenCode | openai-api/opencode-go/kimi-k3 |
OpenCode Go |
kimi-k2.7-code |
OpenCode | openai-api/opencode-go/kimi-k2.7-code |
OpenCode Go |
deepseek-v4-pro |
OpenCode | openai-api/opencode-go/deepseek-v4-pro |
OpenCode Go |
deepseek-v4-flash |
OpenCode | openai-api/opencode-go/deepseek-v4-flash |
OpenCode Go |
mimo-v2.5-pro |
OpenCode | openai-api/opencode-go/mimo-v2.5-pro |
OpenCode Go |
mimo-v2.5 |
OpenCode | openai-api/opencode-go/mimo-v2.5 |
OpenCode Go |
minimax-m3 |
OpenCode | anthropic/opencode-go/minimax-m3 |
OpenCode Go (Anthropic Bridge) |
qwen3.8-max |
OpenCode | anthropic/opencode-go/qwen3.8-max |
OpenCode Go (Anthropic Bridge) |
qwen3.8-flash |
OpenCode | openai-api/opencode-go/qwen3.8-flash |
OpenCode Go |
grok-4.5 |
OpenCode | openai-api/opencode-go/grok-4.5 |
OpenCode Go |
gemma-4-26b-a4b |
OpenCode | openai/google/gemma-4-26b-a4b-qat |
Local (LM Studio) |
qwen3.6-27b |
OpenCode | openai/qwen/qwen3.6-27b |
Local (LM Studio) |
Pricing Comparison
Pricing per 1 million tokens for commercial models. Note that local models (LM Studio) run at $0.00.
| Cost per 1M Tokens | ||
| USD pricing by model | ||
| Model | Input ($/1M tokens) | Output ($/1M tokens) |
|---|---|---|
claude-opus-5 |
$5.00 | $25.00 |
claude-sonnet-5 |
$3.00 | $15.00 |
claude-haiku-4-5 |
$1.00 | $5.00 |
gemini-3.7-flash |
$0.15 | $0.60 |
gemini-3.7-pro |
$1.25 | $5.00 |
gpt-5.6-sol |
$5.00 | $30.00 |
gpt-5.6-terra |
$2.50 | $15.00 |
gpt-5.6-luna |
$1.00 | $6.00 |
deepseek-v4-pro |
$1.74 | $3.48 |
deepseek-v4-flash |
$0.14 | $0.28 |
glm-5.3 |
$1.40 | $4.40 |
glm-5.3-flash |
$0.20 | $0.80 |
kimi-k3 |
$0.95 | $4.00 |
minimax-m3 |
$0.60 | $2.40 |
qwen3.8-max |
$2.50 | $7.50 |
Data Inputs Configuration
shinygen provides convenient mappings to inject CSVs or custom files:
| Data Input Options | |||
| Option | CLI Options | Python API Keys | Notes |
|---|---|---|---|
| Single CSV | --csv-file |
data_csv |
Convenient shorthand; is mapped directly by basename. |
| Multiple files | --data-file (repeatable) |
data_files |
Accepts any text-based data format (JSON, TXT, CSV). |
Note: If both options supply the same file name, --csv-file takes precedence. |
|||
Environment Variables
Set the following variables in your terminal or a .env file at the project root to authenticate:
| Environment Variable | Purpose | Example |
|---|---|---|
ANTHROPIC_API_KEY |
Authenticate Anthropic models (e.g. Claude Sonnet/Opus) | export ANTHROPIC_API_KEY='sk-ant-...' |
OPENAI_API_KEY |
Authenticate OpenAI models (e.g. GPT-5.6) | export OPENAI_API_KEY='sk-...' |
GEMINI_API_KEY |
Authenticate Google Gemini models (e.g. Gemini 3.7 Flash) | export GEMINI_API_KEY='AIzaSy...' |
OPENCODE_GO_API_KEY |
Authenticate OpenCode Go models | export OPENCODE_GO_API_KEY='sk-...' |
LMSTUDIO_BASE_URL |
Base URL endpoint for local LM Studio servers | export LMSTUDIO_BASE_URL='http://localhost:1234/v1' |
Local Development & Testing
We maintain a complete suite of unit tests. Run them locally to verify sandbox configurations, config parsers, and API bindings:
# Install development dependencies
pip install -e ".[dev]"
# Run tests
pytestDashSwipe: Interactive Dashboard User Testing
To help align dashboard styles, this repository includes DashSwipe, an interactive user-testing survey application located in the user_testing_app/ directory.
Purpose: Collects human feedback on LLM-generated layouts. It serves screenshots anonymously, lets users tier-rank layouts (S/A/B/C/D/F), prompts for 2-word critiques, and writes evaluation statistics directly to a Supabase database.
Run Locally:
python3 -m http.server 8080 --directory user_testing_appOpen http://localhost:8080/ in your browser.
Database Schema: Refer to user_testing_app/README.md to set up the Postgres schema.
License
This project is licensed under the MIT License.