Starflow Method

Starflow is an optional guided methodology layer that helps you plan and implement data pipelines step-by-step. While Starlake Skills give you direct access to every CLI command, Starflow provides a structured workflow with specialized agent personas that guide you through the full lifecycle, from domain discovery to production deployment.

Like the core skills, Starflow works with Claude Code, GitHub Copilot, and Gemini CLI: it is plain markdown plus a small Python config resolver, and the installer links it into all three assistants by default. Claude Code offers the richest experience (parallel subagents for the adversarial code review); on assistants without subagent support, the review falls back to generated reviewer prompts you run separately.

When to Use Starflow

• Greenfield projects: starting a new data platform from scratch
• Complex migrations: moving from legacy ETL to Starlake
• Team onboarding: structured approach for teams new to Starlake
• Architecture reviews: systematic evaluation of existing pipelines

For quick, targeted tasks (loading a file, writing a transform), use the CLI skills directly.

Workflow Phases

Starflow organizes work into four phases, each with dedicated skills:

        1. Discovery
             │
             ▼
        2. Architecture
             │
             ▼
  ┌──▶  3. Pipeline Design
  │          │
  │          ▼
  │     4. Implementation
  │          │  sprint → dev → review → retro
  └──────────┘
        next epic

  Quality Review (cross-cutting, run at any phase):
    • starflow-data-quality-review
    • starflow-lineage-review

Phase 1: Discovery

Map your data landscape before writing any configuration.

Skill	Description
starflow-domain-discovery	Identify and document data domains, sources, and ownership
starflow-source-analysis	Deep-dive into source schemas, quality, volume, and extraction strategies

Phase 2: Architecture

Design the platform and schemas that will support your pipelines. See also: Schema Management.

Skill	Description
starflow-create-data-architecture	Design layers (raw, staging, mart), engines, storage, and governance
starflow-schema-design	Design Starlake-compatible table schemas with types, constraints, privacy, and expectations

Phase 3: Pipeline Design

Specify pipelines end-to-end before implementation. See also: Ingestion, Transformation, Orchestration.

Skill	Description
starflow-create-pipeline-spec	Create complete pipeline specifications covering extract, load, transform, and orchestrate
starflow-transform-design	Design SQL transformations with quality checks and dependency management
starflow-orchestration-design	Design DAGs, schedules, and retry/timeout policies for pipeline execution
starflow-semantic-model-design	Design a business semantic model (dimensions, facts, metrics, relationships) over pipeline tables, written to metadata/semantic/

Phase 4: Implementation

Build, test, deploy, and reflect on your pipelines. See also: Ingestion, Transformation.

Skill	Description
starflow-sprint-planning	Break down pipeline work into sprint-sized tasks with dependency ordering
starflow-dev-pipeline	Generate Starlake configuration files (YAML + SQL) from specifications
starflow-code-review	Adversarial parallel review (Winston + Amelia + Quinn) before deployment
starflow-retrospective	End-of-epic retrospective that checks follow-through on the previous retro's action items

Quality Review

Cross-cutting skills for validating pipelines at any phase. See also: Data Quality, Lineage.

Skill	Description
starflow-data-quality-review	Review expectations coverage and identify gaps across pipelines
starflow-lineage-review	Trace and document data lineage across pipeline stages

What Starflow Produces

Each skill writes a markdown artifact you can read, version, and iterate on. The default output layout (configurable; see Layered Configuration below) is:

{project-root}/starflow-output/
├── planning-artifacts/
│   ├── domain-discovery-*.md       # Phase 1
│   ├── source-analysis-*.md        # Phase 1
│   ├── data-architecture-*.md      # Phase 2
│   └── schema-design-*.md          # Phase 2
└── implementation-artifacts/
    ├── pipeline-spec-*.md          # Phase 3 (step-file workflow)
    ├── transform-design-*.md       # Phase 3
    ├── orchestration-design-*.md   # Phase 3
    ├── semantic-model-design-*.md  # Phase 3 (model YAML goes to metadata/semantic/)
    ├── sprint-plan-*.md            # Phase 4
    ├── *-implementation/           # Phase 4 (generated YAML + SQL)
    ├── review-*.md                 # Phase 4 (adversarial review report)
    └── retrospective-epic-*.md     # Phase 4 (end-of-epic retro)

Artifacts produced by step-file workflows (pipeline-spec-*.md, review-*.md, retrospective-epic-*.md) carry a stepsCompleted: [...] list in their frontmatter so the workflow can resume across sessions.

Agent Personas

Talk to a specialized agent for guided assistance. Each agent coordinates multiple workflow skills and brings domain expertise:

Skill	Agent	Specialty
starflow-data-analyst	Lea	Domain discovery, source analysis, business requirements
starflow-data-architect	Winston	Architecture, schemas, pipeline design, Starlake configuration
starflow-data-engineer	Amelia	ETL pipeline development, SQL transformations, orchestration
starflow-data-quality-engineer	Quinn	Expectations framework, data profiling, privacy compliance
starflow-platform-engineer	Max	Infrastructure, orchestration deployment, CI/CD

You: /starflow-data-architect Design a data platform for our e-commerce analytics

How Starflow Works

Step-File Workflows

The heavier workflows (starflow-create-pipeline-spec, starflow-code-review, and starflow-retrospective) use a step-file architecture. Each step lives in its own steps/step-NN-*.md file with explicit halt-for-input checkpoints, and progress persists in a stepsCompleted: [...] list in the output document's frontmatter. A workflow can therefore resume cleanly across sessions or context windows: rerun the skill on the same output file and it picks up at the next uncompleted step.

Adversarial Parallel Code Review

starflow-code-review spawns three independent persona subagents in parallel: Winston (architecture), Amelia (engineering), and Quinn (data quality). Each has a focused prompt and the same code under review. Findings are then deduplicated and triaged into BLOCKER / WARNING / SUGGESTION / APPROVED. The independence is the point: each reviewer applies a different lens, surfacing issues a single-pass review would miss.

For small diffs, the workflow offers an explicit light pass: one reviewer covering all three lenses, faster and cheaper, never chosen silently. On assistants without subagent support, the workflow generates the three reviewer prompts as files for you to run in separate sessions and paste back.

Scale-Adaptive Depth

Pipeline specs carry a scale field (light / standard / deep) classified at the start of the workflow from simple heuristics (sources, tables, transforms, environments, SLA). Light pipelines skip optional ceremony and merge confirmation checkpoints; deep pipelines get extra rigor prompts (failure modes, backfill strategy, SLA math, per-column privacy review). Standard is the workflow exactly as written.

Unattended Mode

Setting unattended: true in the layered config (or asking for it per run) lets step-file workflows run without halting at confirmation checkpoints: each checkpoint documents a default, the workflow takes it and logs the decision to an autoDecisions: list in the output frontmatter. Questions that gather facts (names, connections, credentials) still halt, and unattended specs finish as ready-for-review, never ready-for-dev, so a human always reviews the decision log before implementation starts.

Extending Starflow

starflow-builder scaffolds your own extensions without forking the bundle: new agent personas (coached toward the same philosophy-and-voice style as the built-in five, written into the team config layer), new workflow skills (flat or step-file, from templates), and customizations of existing skills through the layered config.

Adaptive Help

starflow-help is more than a menu. It reads the skill manifest at .agents/starflow/_config/starflow-help.csv (with phase, after, before, required, output-location, and outputs columns) and scans your artifacts directory to detect which steps are already done. It then recommends the next required skill based on dependencies, not a hard-coded order, so it works whether you started with discovery or jumped in mid-stream.

You: /starflow-help What should I work on next?

Layered Configuration

Starflow defaults are resolved from three layers (highest wins):

1. Base: .agents/starflow/config/starflow.yaml (installer-managed, treat as read-only)
2. Team: .agents/starflow/config/custom/starflow.yaml (committed to your repo)
3. Personal: .agents/starflow/config/custom/starflow.user.yaml (gitignored)

The same model applies to each skill's customize.yaml. Skills resolve config at runtime via python3 .agents/starflow/scripts/resolve_config.py --starflow-root <path>, which deep-merges mappings and merges sequences of mappings by their code or id key, so a team can override a single agent's description without re-listing the others. See .agents/starflow/config/README.md in the skills repo for full merge semantics.

For example, to set the team's default engine to Snowflake and tighten Winston's voice without re-declaring the other personas, drop this in .agents/starflow/config/custom/starflow.yaml:

default_engine: snowflake
target_engines:
  - snowflake
  - bigquery

agents:
  - code: data-architect
    description: "Channels Kimball at the whiteboard. Always names a recommended option, never just trade-offs."

The agents list merges by code: only data-architect is touched; Lea, Amelia, Quinn, and Max keep their base values.

Getting Started

1. Run /starflow-help to assess your project state and get recommendations
2. Begin with /starflow-domain-discovery to map your data landscape
3. Follow the phases in order, or jump to the phase you need
4. After shipping an epic, run /starflow-retrospective to capture lessons and check follow-through on the previous retro
5. Each Starflow skill references the relevant Starlake CLI skills for implementation details