> Public product help. This page has no customer entries, projects, chat transcripts, API keys, or horizon contents.
> Reading that data requires `Authorization: Bearer` (a signed-in app session JWT, or a Headless access token).
> Do not call Supabase `/rest/v1` with a Headless JWT. Data reads go through `/casc/v1/headless/v1/rest`.

# API Ingestion

API ingestion groups pull JSON records from HTTP endpoints on demand or on a schedule. Configure under **Settings → Ingestion**; run from **Horizon → Data Ingestion**.

Deduplication is configured on the **ingestion group** (not on the API provider panel). See **Group-level deduplication** below and the Deduplication section in the group editor.

## Providers

| Provider | Use case |
| --- | --- |
| **Custom API** | Any REST JSON endpoint you control |
| **XE.net** | Currency rates template; needs `XE_NET_ACCOUNT_ID` and `XE_NET_API_KEY` in **Settings → System** (instance System Settings — not project Agent Settings). See [settings-system-vs-agent](settings-system-vs-agent.md). |

## Custom API configuration

- **Base URL** and **HTTP method** (GET or POST).
- **Auth** — none, bearer token, basic auth, or API key (header or query).
- **Query parameters**, **headers**, **body** — static or templated values.
- **data_path** — JSON path to the array of records (e.g. `data.items`).
- **Field mapping** — each API field maps to a `raw` slug on entries.

## Group-level deduplication

Every ingest into a group (API, Transformation, SQL Transformation, etc.) uses the same
`config.dedupe` settings: a unique key plus one of four modes. Default is **Add**
(always create). New non-duplicate entries are always added in every mode.

- **Field name** — a top-level property on each entry (e.g. `id`).
- **SQL expression** — a Postgres-style expression over `raw`, for composite keys. Allowed forms: `raw->>'field'`, nested `raw->'a'->>'b'`, string literals, `||` concatenation, parentheses, and `COALESCE(...)`. Example: `raw->>'id' || '-' || (raw->>'start')`.

Matching values are stored as `intrinsic.unique_key` (legacy `intrinsic.api_unique_key` is still read).

| Mode | Existing key | New entries | Deletes missing keys |
| --- | --- | --- | --- |
| **Add** | not checked | added | no |
| **Skip Duplicates** | skipped | added | no |
| **Update Duplicates** | raw updated | added | no |
| **Sync** | raw updated | added | yes (hard delete) |

**Migration:** older API `add_only` + key → Skip Duplicates; older API `sync` + key →
Update Duplicates (preserves prior runtime behavior). Opt into true Sync per group when mirroring
(update + delete missing) is intended.

Intra-batch duplicates use last-wins. Sync assumes one primary source per group; multiple sources with Sync can delete another source’s rows.

## Nested arrays and Transformations

Deduplication keeps one entry per API item. If a single item contains a nested array that should become
multiple entries (for example Toggl `time_entries`), use a **Transformation**
group with an Explode step after the API ingest. See [transform-ingestion](transform-ingestion.md).

## Mapping responses to entries

Each array element becomes one entry. Nested values flatten into `raw` via mapping rules. Empty or missing `data_path` yields zero rows — verify with the test harness.

## Test harness

Before enabling schedules:

1. Open **Help → System Test → Ingestion API**, or use the harness in the group editor.
2. Select suffix and group; adjust param overrides if needed.
3. Inspect request debug (URL, auth configured flag, body) and response JSON.
4. Confirm mapped preview rows look correct.
5. Save group config; run manual **Ingest** from Horizon.

## Review and embed

- Entries land in **Data Review** unless **auto-approve** is on.
- Update / Sync on a previously reviewed entry marks it unreviewed again (unless auto-approve).
- Enable **auto-embed** on the group or run **Initiate Embedding** on linked search tools after each ingest — **before** significant LLM chat or workflows that use search. Accepted rows alone are not searchable until embedded. See [tools](tools.md).

## Scheduling

API groups support periodic ingestion (minimum 5 minutes). Configure in the group editor or **Settings → Schedules**. See [schedules](schedules.md).

## Troubleshooting

| Symptom | Likely cause |
| --- | --- |
| Harness works, production empty | Saved config differs from harness overrides; re-save group |
| Auth errors | Token expired, wrong header name, or system secret missing |
| Zero rows, HTTP 200 | Wrong `data_path` or mapping on non-array payload |
| Unexpected deletions | Group Deduplication mode is Sync — switch to Update Duplicates if you only want upserts |