# Workflows — End-to-End Scenarios

These workflows assume a workspace already exists (`lumo init <tenant-id>` has been run).

## Workflow 1: ETL Pipeline — Database to Data Warehouse

**Scenario:** Extract data from a PostgreSQL database, transform with DuckDB SQL, load to DW.

```bash
# 1. Choose an agent that has network access to your DB
lumo info agent                 # find an online agent_id

# 2. Create database credential
lumo new credential --name "Prod Postgres"
# Edit credentials/<created>.yaml: set host, port, database, user, password, tokenId (agent_id)
lumo push credential:<id>

# 3. Verify connection
lumo credential test credential:<id>
lumo credential schema credential:<id> --schema public

# 3b. Read the credential KEY — this is what goes in the extract node's `Credential:`
lumo list credential            # the `key:` column, e.g.  1251  MySQL  key:QZYcr8rxt9
# The key is an opaque server-generated string. It is NOT the numeric id and NOT the
# nome. Writing the nome there makes the agent fail to resolve the connection, and the
# node fails without a useful message. The key is also in the credential YAML (`key:`,
# read-only) and in `lumo info credential:<id>`.

# 4. Create ETL flow
lumo scaffold flow duckdb       # check the shape/nodes first
lumo new flow --name "Carga Vendas"
# Edit flows/<created>.yaml:
#   - Set credential key in extract node
#   - Write SQL query in extract node
#   - Write DuckDB transformation SQL in SQLProcessor
#   - Configure output table name in InsertDatawarehouse
lumo lint flows/<created>.yaml
lumo push flow:<id>

# 5. Test single node first
lumo flow run-node flow:<id> --node ext01 --wait
# Check data preview from that node
lumo flow preview-node flow:<id> --node ext01

# 6. Run full flow to populate DW
lumo flow run flow:<id> --context Total --wait
# Streams live progress: node start/finish, row counts, duration

# 7. Refresh local state to see the new table columns
lumo fetch table:<table-id>
lumo columns table:<table-id>
```

## Workflow 2: BI Dashboard from REST API

**Scenario:** Fetch data from IBGE API, transform, create BI dashboard.

```bash
# 1. Create HTTP extract flow
lumo new flow --name "IBGE Municipios"
# Edit flow YAML (see `lumo scaffold flow http` for reference):
#   - Extract node: GET https://servicodados.ibge.gov.br/api/v1/localidades/municipios
#   - SQLProcessor: transform JSON response into structured table
#   - InsertDatawarehouse: write to DW table "dim_municipios"
lumo push flow:<id>

# 2. Run flow to populate DW
lumo flow run flow:<id> --context Total --wait

# 3. Create BI application
lumo new app --name "Brasil BI"
# Edit apps/<created>.yaml:
#   - Add the new table ID to apps/_tables.yaml (if split) or body.tables
#   - Set type: dimension (covers both facts and dimensions in BI engine)
lumo push app:<id>

# 4. Get app metadata (column IDs for widgets)
lumo info app:<id> --full

# 5. Add widgets (get -> edit -> push)
# Edit app YAML / dashboards: add widgets with column IDs from the info report
# Use `lumo scaffold widget <type>` to see chart configuration details
lumo push app:<id>

# 6. Verify visual results
lumo app dashboard-preview app:<id>
lumo app export app:<id> --image-format png
```

## Workflow 3: Joining Multiple Data Sources

**Scenario:** Combine database data with API data using DuckDB.

```bash
# 1. Create credentials for each source
lumo new credential --name "Postgres"
lumo new credential --name "MySQL"
# Edit and push both
lumo push credential:<id1>
lumo push credential:<id2>

# 2. Create flow with multiple extracts + Join node
lumo new flow --name "Joined Data"
# Edit flow YAML (see `lumo scaffold flow join` for reference):
#   - Extract node 1: PostgreSQL (customers)
#   - Extract node 2: MySQL (orders)
#   - Join node: JOIN on customer_id
#   - InsertDatawarehouse: write result to DW
lumo push flow:<id>

# DuckDB SQL example for the transform node:
# SELECT
#   c.NAME, c.CITY,
#   COUNT(o.ID) AS ORDER_COUNT,
#   SUM(o.TOTAL) AS TOTAL_REVENUE
# FROM input_0 c                    -- input_0 = first connected node
# JOIN input_1 o ON c.ID = o.CUSTOMER_ID  -- input_1 = second connected node
# GROUP BY c.NAME, c.CITY

# Gotcha — casting a Number column to varchar in DuckDB goes through the decimal
# representation, so `cast(SIT_DD as varchar)` on a Number yields '0.000000000',
# not '0'. That silently breaks a join against a code/legend table keyed by '0'.
# Go through an integer type first:
#   cast(cast(SIT_DD as bigint) as varchar)

# 3. Run and verify
lumo flow run flow:<id> --wait
```

## Workflow 4: Scheduling Automated Pipelines

**Scenario:** Run flows on a daily schedule at 7 AM.

```bash
# 1. Create schedule
lumo new schedule --name "Carga Diaria"
# Edit schedules/<created>.yaml:
#   - Attach flow IDs (must be published flows for production)
#   - Configure trigger: type=Day, period=1, start_when=07:00 (daily at 7 AM)
#   - Note: Not cron format! Uses type/period/start_when
lumo push schedule:<id>

# 2. Verify schedule
lumo info schedule:<id>        # shows next execution times

# 3. Force a test run
lumo schedule run schedule:<id>

# 4. Pause/resume
lumo schedule pause schedule:<id>
lumo schedule resume schedule:<id>
```

## Workflow 5: Published Resource Iteration

**Scenario:** Safely edit a published dashboard.

```bash
# 0. First check whether you need a clone at all. A published app accepts an
#    in-place push when the ONLY changes are finish fields:
#    nome, descricao, icon, color, facts, filtros.
$EDITOR apps/<published>.yaml       # e.g. re-apply the AI indicators
lumo push app:<published-id>        # succeeds; exit 9 names the structural field if not

# 1. For structural changes (tables, expressions, relationships, dashboards,
#    config, tags) the app is read-only. Clone to your desk first.
lumo app clone app:<published-id> --name "Edit Version"
# CLI output tells you where the clone YAML is created
# Note: cloning regenerates every expression id, so any facts[].primaryColumn
# written against the old ids must be re-pointed before publishing.

# 2. Iterate locally
$EDITOR apps/<clone>.yaml
lumo push app:<clone-id>
lumo app dashboard-preview app:<clone-id>

# 3. Publish back to the original desk
lumo app publish app:<clone-id> --desk <target-desk-id>
# This overwrites the original with your clone's body and removes the clone.
```

## Workflow 6: Python Transform with API Enrichment

**Scenario:** Enrich database data with external API calls using Python node.

```python
# In the PythonProcessor node (use `lumo scaffold node PythonProcessor` for shape):
import pandas as pd
import requests

df = input_0.copy()

# Enrich each row with API data
def fetch_cep(cep):
    try:
        r = requests.get(f"https://viacep.com.br/ws/{cep}/json/", timeout=5)
        data = r.json()
        return data.get("localidade", ""), data.get("uf", "")
    except:
        return "", ""

# Apply enrichment
df[["CIDADE", "UF"]] = df["CEP"].apply(lambda x: pd.Series(fetch_cep(x)))

# Column names MUST be UPPERCASE for DW insertion
df.columns = [c.upper() for c in df.columns]
```

> **HTTP node choice — prefer HTTPRequest over Python.** For HTTP/REST use the
> **HTTPRequest** node (`lumo scaffold node HTTPRequest`): it handles auth,
> pagination, retries and JSON flattening declaratively and runs faster than
> row-by-row Python. It even does per-row calls via `inputLoop` (replaces the
> pattern above for most cases). Use a PythonProcessor for HTTP **only** when the
> API needs logic HTTPRequest can't express — response-dependent request graphs,
> custom request signing, etc. Don't default to `requests` for bulk ingestion.

## Workflow 7: Publishing Rules & Cascade Behavior

### What `publish` Actually Does

`publish` is an **override** by default. The published target inherits the clone's identity — `nome`, body, and the workspace filename derived from that `nome`. The source clone is removed; the target keeps its original id but with the clone's content.

Implications:
- If you cloned with `--name "fix-temporal"`, the post-publish file is `<original-slug>-fix-temporal--<original-id>.yaml`. To preserve the original name, clone without `--name`.
- `flow publish` returns `affected: [{kind:'app',...}]` for every app whose column refs were rewritten by the cascade. The CLI re-fetches each so the workspace stays consistent.
- Use `--new` to create a fresh published copy side-by-side instead of replacing the existing one.

### Cascade Rules

| Operation | Cascades to | Why |
|---|---|---|
| `flow clone` | the flow's target table | The table is flow-owned (schema comes from the flow's `InsertDatawarehouse`). A table without its flow is an orphan. |
| `app clone` | nothing | Apps reference tables; they don't own them. The clone-app points at the same table ids as the source. |
| `lumo table clone` | — | **Does not exist.** To clone a table, clone its producing flow. |

### When to Publish (and When Not To)

Build and validate the **whole stack unpublished before publishing anything**. A flow, its target tables, and an app on those tables can all live unpublished side-by-side — only the final cutover requires publish. Order: **flows first, then app**.

1. Build flows (unpublished) → `flow run-node` to capture schema → `new table --from-flow --from-node`.
2. Build the app referencing those unpublished tables → push → validate via `app query`, `app export`, dashboard preview.
3. Iterate freely — every change is a regular `push`, no clone-edit-publish dance.
4. Once the app behaves, publish flows (freezes table/column ids).
5. Publish the app last. The server enforces "no app referencing unpublished tables".

Anti-pattern: publishing a flow just to test the app. Use `app query` against the unpublished table directly.

### change-table Workflow

When you need to rewire an app to a different table (e.g., after cloning a flow that produces a new table):

```bash
lumo app change-table app:<id> --from-table <published-id> --to-table <clone-id>
```

Requirements: the destination table must share `column_name`s with the source table (extras on either side are fine). Translation is by **name match** — column ids on the destination replace the source's wherever they appear (filters, widget dataSources, relationships, facts, expressions).

## Pre-flight Audit

Before every dashboard or ETL task, run a semantic audit to prevent rework:

- **Dashboard tasks**: `lumo info app:<id> --full` → plan widgets BEFORE editing.
- **ETL tasks**: `lumo info flow:<id>` for current config. `lumo flow nodes <id>` to map the DAG. `lumo columns table:<target-id>` if writing to an existing table.

## Bulk Workspace Review

```bash
lumo status                     # what changed locally?
lumo list flow                  # what flows are in this workspace?
lumo diff app:<id> --include-remote  # any drift on the server?
lumo log                        # history of pushes/pulls
```
