# Architecture

## High-Level System Overview

dpg is a **code generation pipeline** that accepts a text prompt (or wireframe images) and produces a fully runnable application. It chains five LLM-powered steps:

```
User Input
  (prompt / images)
       │
       ▼
[Step 01] Input Ingestion          → normalise inputs into IngestedInputs
       │
       ▼
[Step 02] PRD Generation           → generate PRD.md + schema.sql via LLM
       │                               (sub-step 02c: Mantara schema + Dalfin JSON)
       ▼
[Step 03] Backend Generation       → generate FastAPI Python project + api_manifest.json
       │
       ▼
[Step 04] IR Generation            → detect pages (Opus vision) + generate IRBundle per page
       │
       ▼
[Step 05] React Generation         → generate TSX per page + deterministic App.tsx router
       │
       ▼
runs/outputs/<slug>/               → assembled project: backend/ + frontend/ + ir/
```

Every step writes intermediate artifacts to `pipeline/<step>/output/<run_id>/` so the state of any step can be inspected in isolation. The master orchestrator in `pipeline/master-pipeline/pipeline/orchestrator.py` wires them in sequence and captures a telemetry `RunLog`.

---

## Full Annotated Folder Structure

```
dpg/
├── main.py                    # CLI: parses args, calls generate_project()
├── pyproject.toml             # uv project, Python version, dependencies
├── .env / .env.example        # AWS + Bedrock config (not committed)
│
├── shared/                    # Imported by ALL steps — no step-specific imports
│   ├── config.py              # All paths, model IDs, timeouts (env-var overrides)
│   ├── bedrock_client.py      # build_chat_model() → ChatBedrockConverse (LangChain)
│   ├── bedrock_raw_client.py  # BedrockLLMClient → boto3.client("bedrock-runtime")
│   ├── anthropic_client.py    # build_bedrock_client() → AnthropicBedrock (edit only)
│   ├── extractors.py          # extract_json_object(), extract_code_block()
│   ├── llm_logging.py         # timed_llm_call() context manager + token logging
│   ├── logging.py             # configure_logging(), get_logger()
│   ├── timing.py              # log_timed_step() context manager
│   ├── run_artifacts.py       # new_run_id(), step_output_dir()
│   ├── run_log.py             # RunLog, StepEvent, LLMCall — telemetry accumulator
│   ├── redis_config.py        # REDIS_ENABLED, REDIS_URL, TTLs
│   ├── exceptions.py          # NoRunsFoundError, RunNotFoundError
│   ├── media/
│   │   ├── images.py          # EncodedImage dataclass + encode_image()
│   │   └── prd_loader.py      # validate_and_load_prd() — PDF/PPTX/DOCX/MD/TXT
│   └── schemas/               # Pydantic models shared between steps
│       ├── ir_bundle.py       # IRBundle (12 sections) — the core UI spec schema
│       ├── app_plan.py        # AppPlan, PageNode, DesignSystem
│       ├── app_ir.py          # AppIRBundle, IRPage, MultiPageModelConfig
│       ├── multi_page_bundle.py # MultiPageBundle, PageBundle
│       └── plan.py            # ExecutionPlan, PlanStep
│
├── pipeline/
│   ├── master-pipeline/pipeline/
│   │   ├── orchestrator.py    # generate_project() — wires steps 01–05
│   │   ├── master.py          # StepResult, MasterResult — debug aggregation
│   │   ├── reporter.py        # render_console_summary(), render_markdown_report()
│   │   ├── runner.py          # CLI: from-scratch / aggregate / samples / report
│   │   └── pptx_to_pngs.py   # PPTX → PDF → PNG via LibreOffice + PyMuPDF
│   │
│   ├── step-01-input-ingestion/pipeline/
│   │   └── ingestion.py       # ingest() → IngestedInputs dataclass
│   │
│   ├── step-02-prd-generation/pipeline/
│   │   ├── prd_generator.py   # generate_from_prompt / generate_prd / generate_prd_from_ddl
│   │   └── ddl_to_cir.py      # LLM: DDL → enriched CIR dict
│   │
│   ├── step-03-backend-generation/pipeline/backend_gen/
│   │   ├── orchestrator.py    # Orchestrator.run() — 9 internal sub-steps
│   │   ├── requirement_analyzer.py  # LLM: raw input → structured RequirementAnalysis
│   │   ├── fastapi_code_generator.py # LLM: analysis → per-module FastAPI files
│   │   ├── manifest_generator.py    # AST: generated files → api_manifest.json
│   │   ├── markdown_generator.py    # LLM: → BACKEND_SPEC.md
│   │   ├── readme_generator.py      # LLM: → README.md
│   │   ├── project_file_generator.py# deterministic: → pyproject.toml
│   │   ├── output_writer.py         # writes all files to output_dir
│   │   └── validation_runner.py     # AST-validates generated Python syntax
│   │
│   ├── step-04-ir-generation/pipeline/ir_pipeline/
│   │   ├── services/multi_page_service.py    # generate_app_ir() orchestrator
│   │   ├── services/page_detection_service.py # detect_pages() via Opus vision
│   │   ├── services/ir_generation_service.py  # generate_ir_bundle_for_page()
│   │   └── prompts/                           # prompt templates for IR and page detection
│   │
│   └── step-05-react-generation/pipeline/
│       ├── react_gen.py         # generate_react_pages() — TSX generation orchestrator
│       ├── scaffolder.py        # scaffold_frontend(), write_generated_files()
│       └── react_pipeline/services/react_generation_service.py
│           # generate_page_react_code()
│
├── runs/outputs/               # Assembled projects written here
└── specs/                      # Drop-in input directory (images/, ddl/)
```

---

## Layer Communication Map

```
CLI (main.py)
    │  args: { user_prompt, images_dir, output, name, api_base_url }
    │  via:  argparse.Namespace
    ▼
orchestrator.generate_project()
    │  in:  user_prompt: str|None, images_dir: Path|None, output_dir: Path, project_name: str|None
    │  out: Path  (assembled project root)
    ▼
Step 01 — ingestion.ingest()
    │  in:  inputs_dir: Path, user_prompt: str|None, images_dir: Path|None
    │  out: IngestedInputs { user_prompt, images_dir, image_paths }
    ▼
Step 02 — prd_generator.generate_from_prompt() or generate_prd()
    │  in:  user_prompt: str  OR  images_dir: Path
    │  LLM calls: BedrockLLMClient (claude-3-5-sonnet) × 2 (PRD, DDL)
    │  out: LoadedPRD { path, text, images, ddl_path }
    ▼
Step 02c — Mantara + Dalfin (optional)
    │  in:  prd_text + ddl_text
    │  out: mantara_schema.json + dalfin.json  (written to s02_out/)
    ▼
Step 03 — backend_gen.orchestrator.Orchestrator.run()
    │  in:  ddl_file, prd_file, prompt_text, image_files
    │  LLM calls: BedrockLLMClient (claude-3-5-sonnet) × multiple
    │  out: GenerationResult { system_name, api_manifest, module_names, endpoint_count, file_count }
    │       api_manifest: dict written to project_root/api_manifest.json
    ▼
Step 04 — multi_page_service.generate_app_ir()
    │  in:  images_dir, user_prompt, api_manifest, prd_text, prd_images
    │  LLM call 1: ChatBedrockConverse (Opus) → AppPlan (page detection)
    │  LLM call N: ChatBedrockConverse (Opus) × N_pages → IRBundle per page
    │  out: AppIRBundle { app_plan, ir_pages, run_id, encoded_images }
    ▼
Step 05 — react_gen.generate_react_pages()
    │  in:  AppIRBundle, api_manifest
    │  LLM call N: ChatBedrockConverse (Sonnet) × N_pages → TSX code
    │  deterministic: generate_router_code(app_plan) → App.tsx
    │  deterministic: generate_context_code(app_plan) → AppContext.tsx
    │  out: MultiPageBundle { app_plan, pages, router_code, context_code, run_id }
    ▼
scaffolder.write_generated_files()
    │  in:  frontend_dir, router_code, context_code, pages: dict[filename, tsx_code]
    │  out: files written to frontend_dir/src/
    ▼
runs/outputs/<project_slug>/
    ├── backend/        (FastAPI Python project)
    ├── frontend/       (Vite + React + Ant Design TSX project)
    └── ir/             (IRBundle JSON files + app_plan.json)
```

### What data crosses each boundary

| Boundary | Data | Type | File handling the crossing |
|----------|------|------|---------------------------|
| CLI → orchestrator | prompt + paths | `argparse.Namespace` | [main.py:80](../main.py#L80) |
| orchestrator → step-01 | `inputs_dir`, `user_prompt`, `images_dir` | `Path \| str \| None` | [orchestrator.py:132](../pipeline/master-pipeline/pipeline/orchestrator.py#L132) |
| step-01 → step-02 | `IngestedInputs` | dataclass | [orchestrator.py:151](../pipeline/master-pipeline/pipeline/orchestrator.py#L151) |
| step-02 → step-03 | `LoadedPRD.path`, `LoadedPRD.ddl_path` | `Path` | [orchestrator.py:215](../pipeline/master-pipeline/pipeline/orchestrator.py#L215) |
| step-03 → step-04 | `api_manifest: dict` | JSON dict | [orchestrator.py:262](../pipeline/master-pipeline/pipeline/orchestrator.py#L262) |
| step-04 → step-05 | `AppIRBundle` | dataclass | [orchestrator.py:295](../pipeline/master-pipeline/pipeline/orchestrator.py#L295) |
| step-05 → disk | `MultiPageBundle` | dataclass → `.tsx` files | [orchestrator.py:304](../pipeline/master-pipeline/pipeline/orchestrator.py#L304) |

---

## State Management Strategy

There is no shared mutable state between steps at runtime. State is passed **forward only** via:

1. **Function return values** — each step returns a typed dataclass or dict
2. **Filesystem artifacts** — each step writes intermediate files to `<step>/output/<run_id>/`
3. **Module-level singleton** — `shared/run_log.py` keeps one `RunLog` instance active per run via `set_active()` / `get_active()`

The `RunLog` is the only global side-channel. It accumulates LLM call metrics from any step via `record_llm_call()` and is finalized (writing `run_log.json` + `run_log.md`) in the `finally:` block of `generate_project()`.

---

## Routing Strategy

There is no HTTP routing in the pipeline itself. "Routing" here means **model routing** — which Bedrock model is used at each step:

| Step | Model | Rationale |
|------|-------|-----------|
| Step 02 (PRD + DDL) | `claude-3-5-sonnet-20241022` | Cost-efficient for text generation |
| Step 03 (backend) | `claude-3-5-sonnet-20241022` | Code generation, good cost/quality |
| Step 04a (page detection) | `claude-opus-4-6` | Vision + highest-stakes structural decision |
| Step 04b (IR generation) | `claude-opus-4-6` | Complex 12-section JSON schema |
| Step 05 (React TSX) | `claude-sonnet-4-5` | Long TSX files, text-only |

All model IDs are configurable via environment variables — see [ARCHITECTURE.md#environment-variables].

---

## Error Handling Strategy

Errors propagate **upward** through the call stack without being swallowed:

| Layer | What it does on error |
|-------|-----------------------|
| `timed_llm_call()` | logs `[LLM] FAILED` then re-raises |
| `log_timed_step()` | logs `failed | duration_ms=...` then re-raises |
| `RunLog.step()` context manager | sets `ev.status = "failed"`, records `ev.error = repr(exc)`, then re-raises |
| `orchestrator.generate_project()` | `try/finally` block always calls `run_log.finalize()` even on failure |
| `main.cmd_generate()` | catches `FileNotFoundError`, `ValueError` → prints to stderr, returns exit code 2; catches `KeyboardInterrupt` → returns 130 |

```
┌──────────────────────────────────────────────────────┐
│ ERROR PATH (e.g. Bedrock throttling in step-04)       │
├──────────────────────────────────────────────────────┤
│ INPUT                                                  │
│   from:    boto3 / LangChain raises ThrottlingException│
│   data:    exception instance                          │
│   via:     exception propagation                       │
├──────────────────────────────────────────────────────┤
│ PROCESS                                                │
│   timed_llm_call() logs FAILED + elapsed_ms, re-raises│
├──────────────────────────────────────────────────────┤
│ OUTPUT                                                 │
│   to:      generate_ir_bundle_for_page() caller        │
│   triggers: multi_page_service propagates up           │
├──────────────────────────────────────────────────────┤
│ ERROR PATH                                             │
│   RunLog.step() catches: sets status="failed"          │
│   generate_project() finally: run_log.finalize()       │
│   main.py: catches generic Exception if unhandled      │
│   shown:   printed to stderr; exit code 2              │
└──────────────────────────────────────────────────────┘
```

---

## Environment Variables

| Variable | Purpose | Example / Default |
|----------|---------|-------------------|
| `AWS_ACCESS_KEY_ID` | AWS auth key | `AKIA...` |
| `AWS_SECRET_ACCESS_KEY` | AWS auth secret | `wJalrXUt...` |
| `AWS_SESSION_TOKEN` | Temporary session token | *(optional)* |
| `AWS_PROFILE` | Named AWS profile (instead of explicit keys) | `default` |
| `BEDROCK_AWS_REGION` | Primary Bedrock region | `us-east-1` |
| `AWS_REGION` | Fallback region | `us-east-1` |
| `BEDROCK_MODEL_ID` | Default Sonnet model ID | `global.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `BEDROCK_OPUS_MODEL_ID` | Default Opus model ID | `global.anthropic.claude-opus-4-6-v1` |
| `BEDROCK_HAIKU_MODEL_ID` | Default Haiku model ID | `global.anthropic.claude-haiku-4-5-20251001-v1:0` |
| `BEDROCK_IR_MODEL_ID` | Model for IR generation (step-04b) | falls back to Opus |
| `BEDROCK_REACT_MODEL_ID` | Model for React TSX generation (step-05) | falls back to Sonnet |
| `BEDROCK_PAGE_DETECT_MODEL` | Model for page detection (step-04a) | falls back to Opus |
| `BEDROCK_ROUTING_MODEL` | Model for routing decisions | falls back to Haiku |
| `AGENT_EDIT_MODEL` | Model for agentic edit (step-06) | falls back to Opus |
| `BACKEND_BEDROCK_MODEL` | Model for backend generation (step-03) | `anthropic.claude-3-5-sonnet-20241022-v2:0` |
| `BEDROCK_CONNECT_TIMEOUT_SECONDS` | boto3 connect timeout | `30` |
| `BEDROCK_READ_TIMEOUT_SECONDS` | boto3 read timeout | `600` |
| `BEDROCK_MAX_ATTEMPTS` | boto3 retry count | `2` |
| `BEDROCK_TEMPERATURE` | LLM temperature for all steps | `0.0` |
| `AGENT_MAX_ITERATIONS` | Max agentic edit iterations | `40` |
| `AGENT_MAX_TOKENS` | Max tokens per agent response | `8096` |
| `API_BASE_URL` | Base URL for generated frontend's API calls | `http://localhost:8000` |
| `LOG_LEVEL` | Python logging level | `INFO` |
| `REDIS_ENABLED` | Enable Redis session cache | `false` |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379/0` |
| `SESSION_TTL` | Redis session TTL (seconds) | `7200` |

---

## Related source files

- [shared/config.py](../shared/config.py) — all constants and env-var resolution
- [pipeline/master-pipeline/pipeline/orchestrator.py](../pipeline/master-pipeline/pipeline/orchestrator.py) — step wiring
- [main.py](../main.py) — CLI and error boundary
- [shared/run_log.py](../shared/run_log.py) — telemetry accumulation