# Platform Concepts — Horus BI

## Table of Contents

- [Suites](#suites)
- [Entity Hierarchy](#entity-hierarchy)
- [Multi-Tenancy](#multi-tenancy)
- [Desks (Mesas)](#desks-mesas)
- [Agents](#agents-etl-execution-engines)
- [RBAC](#rbac-role-based-access-control)
- [Authentication Methods](#authentication-methods)
- [Datamarts](#datamarts)
- [Templates](#templates)
- [Variables](#variables)
- [Load Types](#load-types-tipos-de-carga)
- [InsertDatawarehouse Modes](#insertdatawarehouse-modes)
- [Indicators (`facts[]`)](#indicators-facts)

## Suites

Horus BI has four suites; each maps to a set of CLI entities:

| Suite | Purpose | CLI entities |
|-------|---------|-------------|
| **HorusETL** | Extract, transform, load | `flow`, `credential`, `schedule`, `agent` |
| **HorusDW** | Data warehouse storage | `table`, `dw-desk` |
| **DataViz** | Dashboards and BI apps | `app`, `bi-desk`, dashboards/widgets/facts/relationships |
| **HEC** | Users, groups, permissions | (admin via UI; not in CLI scope) |

## Entity Hierarchy

```
Client (Organization)
  └── Tenant (Isolated environment / project)
        ├── Mesas de Dados (Data desks — DW layer)
        ├── Mesas de Aplicacoes (App desks — BI layer)
        ├── Users
        ├── Groups
        ├── Agents (ETL execution engines)
        └── Credentials (Database connections)
```

- **Client** — top-level organization with platform license. Defines white-label (logo, colors, domain).
- **Tenant** — isolated environment within a client. Complete data/user/config isolation. One client can have many tenants (e.g., per department, per customer).
- **User** — global identity. Single credential can access multiple tenants. Assigned to groups for permissions.

## Multi-Tenancy

- `lumo auth login` authenticates the user globally.
- Users can belong to multiple tenants; discover them with `lumo tenants list`.
- Every workspace is scoped to one tenant via `.lumo/workspace.json`, created by `lumo init <tenant-id>`.
- Resource API requests send the workspace tenant context; the server validates user access on each request.
- Resources (flows, tables, apps) are tenant-isolated — IDs are unique per tenant
- Cross-tenant operations use Templates, not direct copy

## Desks (Mesas)

Desks are organizational containers. Two types:

### Mesas de Dados (Data Desks) — CLI kind `dw-desk`
- Organize data warehouse **tables** and **ETL flows**
- Flows and tables are published to data desks
- Common names: "RAW" (raw data), "Refined" (cleaned), "Gold" (business-ready), "Operação"
- Backend table: `dw_desk`
- **Required for** `lumo flow publish` and `lumo table publish`
- CLI commands: `lumo list dw-desk`, `lumo new dw-desk --name <X>`, `lumo info dw-desk:<id>`, `lumo rm dw-desk:<id>`
- Workspace folder: `dw-desks/`

### Mesas de Aplicacoes (Application Desks) — CLI kind `bi-desk`
- Organize **BI dashboards and applications** (DataViz)
- Apps are published to application desks
- Common names: "Comercial", "Financeiro", "RH", "Logistica", "Projetos"
- Backend table: `bi_desk`
- **Required for** `lumo app publish`
- CLI commands: `lumo list bi-desk`, `lumo new bi-desk --name <X>`, `lumo info bi-desk:<id>`, `lumo rm bi-desk:<id>`
- Workspace folder: `bi-desks/`

**Important:** These are two separate entity types stored in different backend tables with disjoint ID spaces. The legacy unified `desk` kind (and `desks/` folder) is kept as an alias for `bi-desk` for backward compatibility, but new work should use the typed forms. Flows cannot be published to a `bi-desk`, and apps cannot be published to a `dw-desk` — the server rejects the mismatch.

### Minha Mesa (My Desk)
- Personal sandbox for every user
- **The only place where you can edit flows and tables**
- Development happens here, then publish to shared desks

### Publishing Rules
1. Develop in Minha Mesa → test → publish to shared desk
2. Publishing a flow automatically publishes its output table
3. To update a published resource: clone to Minha Mesa → edit → republish
4. Publishing options: "Create New" (independent copy) or "Replace" (update existing)

## Agents (ETL Execution Engines)

### Architecture
```
Frontend (UI) → User designs flows
Backend (API) → Orchestrates, manages schedules
Agent (C#)    → Executes data processing, connects to databases
```

### Agent = Physical Machine Gateway

An agent represents a **physical or virtual machine** that has network access to data sources (databases, APIs, files). Every agent-dependent resource must be bound to a specific agent:

| Resource | Binding | Why |
|----------|---------|-----|
| **Credential** | `tokenId` → agent | The agent machine must have network access to the database/API |
| **Flow** | `tokenId` → agent | The agent machine executes the Python/SQL/HTTP processors |
| **Schedule** | inherits from flow | Schedules trigger flows, which run on their bound agent |

**This binding is mandatory.** The ETL does not run on the backend server or the CLI machine; it runs exactly where the agent is installed. A credential for a PostgreSQL database on a private network only works if the assigned agent has network access to that database. A flow executes its queries and Python scripts from the agent's machine.

To change the agent of an existing flow or credential, edit the `tokenId` field in the resource YAML and run `lumo push`.

If `tokenId` is null or 0, the backend may assign a default agent, which might not be the correct one in multi-agent setups.

### Key Facts
- Agents maintain persistent WebSocket connections to the backend
- Backend never initiates inbound connections (firewall-friendly)
- Each agent has a unique 128-char hex token
- One agent can serve 100+ tenants simultaneously
- Recommended parallel executions: 2-4 for standard machines

### Agent Lifecycle
1. Install agent on-premise or cloud (Windows service or Linux/Docker)
2. Configure with token from `lumo info agent:<id>`
3. Agent connects to backend automatically
4. Monitor: `lumo agent restart agent:<id>`

## RBAC (Role-Based Access Control)

### Three-Layer Security Model

| Layer | Question | Controls |
|-------|----------|----------|
| System Functions | "What can I do?" | Create, Edit, Delete, Publish, Block per module |
| Data Access | "What can I see?" | Mesa visibility, table access, datamarts |
| Row-Level Restrictions | "Which rows can I see?" | Column-value filters (e.g., Region = "South") |

### Permission Structure
- **Groups** — named collections of permissions. Users inherit group permissions.
- **Users** can belong to multiple groups (permissions are additive)
- **Block permission** takes precedence over all grants
- **System Functions** per module: flow, app, table, user, group, credential, etc.
- **Actions**: read, create, update, delete, publish

### Access Levels
- **Client Admin** — all tenants, full platform control
- **Tenant Admin** — one tenant, full control of users/permissions/resources
- **User** — limited access defined by groups within one or more tenants

## Authentication Methods

| Priority | Method | Use Case |
|----------|--------|----------|
| 1 | `LUMO_API_TOKEN` env var | CI/CD pipelines |
| 2 | `LUMO_ACCESS_TOKEN` env var | Advanced/proxy scenarios |
| 3 | Interactive login | Local development |

### Token Lifecycle
- Access token: 1 hour TTL
- Refresh token: 30 days (sliding, resets on each refresh)
- API token (static): no expiration, revocable from web dashboard
- CLI auto-refreshes expired tokens transparently

### 2FA Support
- TOTP (Google Authenticator) or email OTP
- Interactive: CLI prompts for OTP code
- Non-interactive: use `--otp <code>` or `LUMO_OTP` env var

## Datamarts

Datamarts bridge physical storage with business access patterns:

| Aspect | Mesa de Dados | Datamart |
|--------|--------------|----------|
| Purpose | Physical storage location | Business-focused grouping |
| Who manages | Technical team | Datamart owner |
| Contains | Actual table files | References to tables from any mesa |
| Governance | Where data is stored | Who can access what |

## Templates

Templates are packaged resource bundles for cross-tenant deployment:
- Bundle desks, flows, tables, apps into a reusable template
- Apply to any tenant: `lumo template apply <id>`
- Use `--dry-run` to preview what will be created
- Templates handle ID remapping automatically (unlike snapshot, which preserves IDs)

## Variables

Tenant-level variables shared across all flows:
- Set: `lumo variable set --key DB_HOST --value "prod.db.com"`
- List: `lumo variable list`
- Use in flow SQL: `{VARIABLE_NAME}` syntax

> **Note:** Schedule triggers use `type: Day | Hour | Month | Minute | Week | CDC` with `period` and `start_when` (`CDC` also needs a `config` block; see yaml-format.md). They are **not** cron expressions. Any schedule can also set `parallel_lane: true` to run outside the agent's normal concurrency limit.
- Common uses: shared connection strings, business rules, org IDs

## Load Types (Tipos de Carga)

Every flow that writes to the Data Warehouse has a **load type** that controls how data is refreshed. This is set at the flow level (`load_type`) and affects how the InsertDatawarehouse processor handles deletions and inserts.

> **Reference:** [ETL Load Types Guide](https://docs.horusbi.com.br/etl/guides/tipos-de-carga.md)

### Total (Full Reload)

Clears the entire target table and replaces all data on every execution.

- **DELETE condition:** `1=1` (delete all rows)
- **Context:** `--context Total`
- **Use case:** Small dimension tables, lookup tables, or when data cannot be incrementally tracked. Simple and safe — no risk of duplicates.
- **Example:** Reference tables (cities, categories), daily snapshots

### Temporal (Date-Range Reload)

Reloads only a specific date range, determined by a partition column.

- **DELETE condition:** `TRUNC("COLUMN_NAME") BETWEEN 'StartDate' AND 'EndDate'`
- **Context:** `--context Temporal` with `--temporal-days N` (N days back)
- **Requires:** `load_type_column` set on the flow (the date column to partition by), and the DW table column marked as `is_partition_column: true`
- **Variables injected:** `{StartDate}`, `{EndDate}` — available in SQL extract queries
- **Use case:** Large fact tables where you only need to refresh recent data. Run daily with `--temporal-days 7` to refresh last 7 days.
- **Example:** Sales transactions (partition by `data_venda`), log events (partition by `created_at`)

#### The Temporal contract is three parts, not one

A Temporal load is only correct when **all three** of these agree. They live in different
places and nothing at runtime ties them together, so getting one wrong corrupts the
published table while the run still reports success.

1. **`load_type: Temporal`** on the flow.
2. **`load_type_column: <date column>`** on the flow. This is the column the agent puts in
   the DELETE. Leave it empty and the predicate falls back to `1=1` — the load **wipes the
   entire table** and reinserts only the window.
3. **The extract SQL filters on `{StartDate}` / `{EndDate}`**, on that same column. The
   DELETE window comes from the run context; the SELECT window comes from your query.
   They are not connected. If the SELECT is **wider** than the DELETE, rows outside the
   deleted range get inserted a second time → **duplicates**. If it is **narrower**, rows
   are deleted and never reinserted → **holes**.

```sql
-- correct: the SELECT window is the same window the DELETE will clear
select * from vendas
where DATA_VENDA >= cast('{StartDate}' as date)
  and DATA_VENDA <= cast('{EndDate}' as date)
```

> **`run-node` does not inject `{StartDate}` / `{EndDate}`.** Those variables are set by
> the flow Runner from the run context, and `run-node` executes a single node through the
> agent's DevRunner, which never goes through that step. The placeholder therefore reaches
> the database **literally**, as the eight characters `{StartDate}`. A query written for a
> Temporal load behaves differently under `run-node` than under `flow run`, which makes
> node-level testing misleading exactly where it matters most. Guard the query if you need
> both paths to work:
>
> ```sql
> where ('{StartDate}' not like '2%' or DATA_VENDA >= cast('{StartDate}' as date))
> --      ^ still the literal placeholder → full extract (run-node)
> --                                        a real date → windowed (flow run)
> ```

> **CRITICAL — the context does not upgrade the flow.** The run context decides the
> extraction window; the flow's `load_type` decides the DELETE predicate. Running a
> `load_type: Total` flow with `--context Temporal` (or binding it that way in a schedule)
> windows the SELECT but leaves the DELETE at `1=1`: the table is emptied and only the
> window comes back. This silently destroyed published fact tables in production. The CLI
> now rejects the combination at `lumo lint`, the API rejects it on push and on
> `flow run`, and the agent aborts the load instead of deleting. Never reach for a
> Temporal context to "make a Total flow incremental" — convert the flow properly, all
> three parts at once.

> **CRITICAL:** The partition column **must be immutable** (never changes after creation). Use `data_emissao` (issue date), `created_at`, or `data_venda` — never `data_vencimento` (due date) or any field that can be updated. If the value changes (e.g., a due date is extended), the DELETE removes the old range but the INSERT puts the record in a new range, causing **duplicates**. For mutable date columns, use Incremental with unique key instead.

### Incremental (Append / Upsert)

Appends new records or upserts (insert + update) based on a tracking column.

- **DELETE condition:** None (no deletion — pure append or upsert)
- **Context:** `--context Incremental`
- **Requires:** `load_type_column` set on the flow (the tracking column, e.g., `updated_at` or auto-increment `id`)
- **Variable injected:** `{LastDataPoint}` — the maximum value of the tracking column from the last execution
- **First execution:** Loads everything (behaves like Total)
- **Subsequent executions:** Only fetches records where tracking column > LastDataPoint
- **Upsert behavior:** If the DW table has `key_type: "unique_key"` with `is_key_column: true` columns, Doris performs UPSERT (replace on key conflict). Otherwise, it's pure APPEND.
- **Use case:** Append-only logs with `created_at`, or mutable records with `updated_at` + unique key.
- **Example:** Orders table with `updated_at` tracking column and `order_id` as unique key

### Choosing the Right Load Type

| Scenario | Load Type | Partition/Track Column | Table Keys |
|----------|-----------|----------------------|------------|
| Small reference table (< 100k rows) | Total | — | — |
| Daily sales (millions of rows) | Temporal | `data_venda` (Date) | — |
| Append-only events/logs | Incremental | `created_at` (Date) | — |
| Mutable records (orders, tickets) | Incremental | `updated_at` (Date) | `order_id` (unique key) |
| Mixed: full refresh weekly + daily delta | Temporal | date column | — |

### DW Table Column Configuration

Tables created via `lumo table create-from-node` auto-detect columns from execution metadata. After creation, configure keys and partitions by editing the table YAML and running `lumo push table:<id>`.

Key column flags:
- **`is_key_column: true`** — Part of the unique key (enables upsert in Incremental mode). Multiple columns can be key.
- **`is_partition_column: true`** — The date column used for Temporal load filtering. Only one column can be partition.
- **`key_type: "unique_key"`** — Set on the table header to enable Doris UNIQUE KEY model (required for upsert)

> **Warning:** Changing key or partition columns triggers a **table recreation** in Doris (data is preserved via rebuild).

### One Flow = One DW Table

Each flow writes to exactly **one** table via its `InsertDatawarehouse` node. Need two tables? Create two flows. Do not attempt to configure multiple `InsertDatawarehouse` nodes in a single flow expecting multiple outputs.

### Post-Creation Table Setup

After `lumo table create-from-node`, the table has auto-generated labels and no masks. **Always configure:**

1. **Labels** — Humanize column names: `CUSTOMER_ID` → `Customer`, `DATA_EMISSAO` → `Issue Date`
2. **Masks** — Format numbers: `MOEDA` (currency), `PCT1` (percentage 1 dec), `N0` (integer), `N2` (2 decimals)
3. **Expressions** — Add calculated metrics for the BI layer:
   - `count(*)` → "Total Records"
   - `sum(VALOR)` → "Total Revenue" (mask: MOEDA)
   - `sum(VALOR) / count(*)` → "Average Ticket" (mask: MOEDA)
   - `count(distinct CUSTOMER_ID)` → "Unique Customers" (mask: N0)

Expressions are virtual columns (not stored in Doris) computed at query time by the Generator.

**Lint rules enforced:**
- String columns cannot have numeric masks (MOEDA, PCT, N0, etc.)
- Number columns should have a mask
- Partition column must be Date type
- All columns should have labels

> **Reference:** [Table Configuration](https://docs.horusbi.com.br/dataviz/02-apps/tables.md)

## InsertDatawarehouse Modes

Every `InsertDatawarehouse` node writes to one of two storages:

| Mode | Storage | Use for |
|---|---|---|
| **Datawarehouse** (default) | Doris database | Final tables for dashboards (gold layer) |
| **Datalake** | S3 parquet files | Raw/intermediate tables reusable by other flows |

### Layered Processing (Raw → Refined → Gold)

1. Flow 1: source → `InsertDatawarehouse(Mode=Datalake)` → publish to "Raw" desk
2. Flow 2: `ExtractDatalake(from Flow 1)` → SQLProcessor → `InsertDatawarehouse(Mode=Datawarehouse)` → publish to "Gold" desk

`ExtractDatalake` can ONLY read from **published** datalake tables.

## Indicators (`facts[]`)

An **Indicator** is the definition of a business metric. It teaches the AI to understand
that concept **and** it appears in the customer's Indicators cockpit, with an icon, color,
goal and traffic light.

The YAML key is `facts` — a historical name, kept because it is what the backend uses.
On screen, in the MCP tool `list_indicators`, and for the customer, the name is
**Indicator**. Same record, one name per surface.

Do not confuse it with a *fact table* in the star schema (`fact_sales`). That is data
modelling; this is a business dictionary. See [Data Modeling](data-modeling.md).

**Without an indicator**: "how much did we receive last month?" forces the AI to guess
table, column, aggregation, filter and date column. With a `titulos` table holding both
payables (CP) and receivables (CR), it may pick `VR_TIT` (gross) instead of `VR_BAIXA`
(received), miss the `TP_TIT=CR` filter, or use `DT_EMISSAO` instead of `DT_BAIXA`.

**With an indicator** named "Contas Recebidas" mapped to value=`VR_BAIXA:SUM`,
filter=`TP_TIT=CR`, date=`DT_BAIXA`, the AI answers correctly in one turn.

### What reaches the AI, and what does not

| Reaches the chat prompt and MCP | Cockpit presentation only |
|---|---|
| `nome`, `prompt`, `primaryColumn`, `alternativeColumns`, `dateColumnId`, `defaultDatePeriod`, `filters` | `apresentacao`, `monitoramento`, `meta` |

Changing an icon or color does not change a single chat answer. Do not spend turns tuning
presentation expecting better AI results.

### Fields

| Field | Required | Description |
|-------|----------|-------------|
| `nome` | yes | Business term users say ("Faturamento", "Contas a Receber") |
| `primaryColumn` | yes | `"<columnId>:AGG"` — SUM, COUNT, AVG, MIN, MAX, DCOUNT. IDs from `lumo info app:<id> --full` |
| `prompt` | no | Extra context for the AI. Use it to **disambiguate** or flag a trap — not to restate the obvious. The whole catalogue is sent on **every** chat message, so filler dilutes context and costs tokens |
| `alternativeColumns` | no | Related metrics sharing the same filters/date |
| `dateColumnId` | no | Date column. Enables "last month", "today". Without it there is no series and no monitoring |
| `defaultDatePeriod` | no | One of 14 values: `Today`, `CurrentMonth`, `MonthToDate`, `PreviousMonth`, `Yesterday`, `CurrentYear`, `NextYear`, `DayBeforeYesterday`, `AllTime`, `PreviousYear`, `ThisWeek`, `PreviousWeek`, `NextWeek`, `Last7Days`. The cockpit screen only offers a subset of these — the YAML accepts all 14 |
| `filters` | no | Fixed filters: `[{ columnId, operator, value }]` |
| `apresentacao.icone` | no | Iconify, e.g. `"ph:chart-line-up"` |
| `apresentacao.cor` | no | hex, e.g. `"#2E7D32"` |
| `apresentacao.formato` | no | `currency` \| `percent` \| `integer` \| `decimal` |
| `apresentacao.direcao` | no | `up_good` \| `down_good` \| `neutral` — which way is good. Cost and churn are `down_good` |
| `monitoramento.grao` | no | `hour` \| `day` \| `week` \| `month` |
| `monitoramento.janelaDias` | no | 1..1095. Default per grain: hour=42, day=365, week=730, month=1095 |
| `monitoramento.farol` | no | `auto` (learned statistical band) or manual bands **in percent**: `{ verde: 5, ambar: 10 }`. `verde` must be smaller than `ambar` — **the lint does not check this** (JSON Schema cannot compare sibling fields). An inverted farol (`verde` > `ambar`) passes lint clean and only shows up wrong later, on the customer's cockpit. Check it yourself before writing it |
| `meta` | no | Goal value, greater than 0 and less than 1e14. **One per indicator**, for the period in `defaultDatePeriod` — changing the period detaches the goal. Requires `dateColumnId` set on the same indicator (unless `meta: null` to erase); without a date column, lint rejects the goal unless nulling it |

### Block semantics

A **missing** block preserves whatever the server has. A **present** block is desired
state — a key omitted inside it **removes** the value.

To drop an icon, include `apresentacao` without `icone`. To touch nothing, omit the block.

`meta` is the exception: it is a loose scalar, not a block. **Absent** preserves state;
**`null`** is the only way to delete it. Omitting the `meta` key does **not** delete
anything — it means "leave this alone."

### Out of scope for the CLI

Following, pinning to the home, notifications and promoting a personal indicator are
end-user actions on screen — they belong to a person, not to the project. The CLI only
reads and writes **curated** indicators.

### Volume

5 to 25 per app. Above that the catalogue dilutes the AI's context and makes every message
more expensive. Prefer few correct indicators over many approximate ones.

Indicators are **inline** in the app YAML (under `facts:`), pushed with the rest of the app.
For the full field reference: `lumo scaffold fact`.

### Example — same table, different concepts

```yaml
facts:
  - nome: "Contas Recebidas"
    primaryColumn: "1054:SUM"          # VR_BAIXA
    dateColumnId: 1060                 # DT_BAIXA
    defaultDatePeriod: CurrentMonth
    filters:
      - { columnId: 1050, operator: "=", value: "CR" }
    apresentacao:
      icone: "ph:hand-coins"
      formato: currency
      direcao: up_good
    meta: 1500000

  - nome: "Inadimplência"
    primaryColumn: "1056:SUM"          # VR_ABERT
    dateColumnId: 1061                 # DT_VENCTO
    filters:
      - { columnId: 1050, operator: "=", value: "CR" }
    apresentacao:
      direcao: down_good               # falling is good
    monitoramento:
      farol: { verde: 5, ambar: 10 }
```
