> Public product help. This page has no customer entries, projects, chat transcripts, API keys, or horizon contents.
> Reading that data requires `Authorization: Bearer` (a signed-in app session JWT, or a Headless access token).
> Do not call Supabase `/rest/v1` with a Headless JWT. Data reads go through `/casc/v1/headless/v1/rest`.

# Building an Ingestion Group

An ingestion group is the unit of data organization in Zagger: a slug, display name, source configuration, and membership in one or more horizons. This page walks through creation from empty horizon to embedded, searchable data.

## Prerequisites by role

- **Any user** — access to the active horizon (public horizon, assignment, or ownership). Can create private groups they own.
- **Google sources** — personal Google OAuth connected (Settings → User).
- **API with system secrets** — superadmin configures XE.net or embedding model in Settings → System.
- **Assign group to another horizon** — admin/superadmin or horizon owner, depending on setup.

See [roles-and-access](roles-and-access.md) for full role matrix.

## Step 1 — Choose your horizon

**Settings → User → Active Horizon** sets which suffix you edit and ingest into. All subsequent steps use this horizon unless you explicitly assign the group elsewhere.

## Step 2 — Create the group

**Settings → Ingestion → Add Ingestion Source**

1. Enter a unique **slug** (e.g. `customer_companies`) and **name**.
2. Pick a **source type** (see [ingestion-sources](ingestion-sources.md)) — including **API Source**, **Transformation**, or **SQL Transformation**.
3. Complete type-specific config (spreadsheet id, folder link, API endpoint, transform steps / SQL script, column mapping).
4. Configure **Deduplication** on the group (key + mode) if re-ingests should skip, update, or sync instead of always adding.
5. Set **Auto-approve** / **Auto-embed** if you want hands-off review or embedding.
6. Save.

## Step 3 — Assign horizons

On the ingestion card, click **Assign** and select which data horizons should see this group. Groups only appear in Horizon pages for assigned suffixes.

## Step 4 — Run ingestion

**Horizon → Data Ingestion** → **Ingest** on your group.

- Spreadsheet and folder sources may show an options dialog (row limits, link validation).
- **API / Transformation / SQL Transformation** — re-ingest applies the group’s Deduplication mode (Add, Skip Duplicates, Update Duplicates, Sync). See [api-ingestion](api-ingestion.md).
- **Transformation** — Ingest materializes the pipeline into this group (preview is available in the group editor). See [transform-ingestion](transform-ingestion.md).
- **SQL Transformation** — Ingest runs the SQLite script and writes the final SELECT into this group. See [transform-sql-ingestion](transform-sql-ingestion.md).
- Progress appears on the page; large jobs poll until complete.
- For recurring pulls, configure a schedule (see [schedules](schedules.md)).

## Step 5 — Review and approve

Unless auto-approve is on, open **Horizon → Data Review**. Approve entries you want in production datasets. Rejected or unreviewed rows are excluded from many tool operations.

Processing prompts (Settings → Prompts, page location `review`) can appear as buttons on the review page to enrich entries.

## Step 6 — Create a search tool

**Settings → Tools → Add Tool**

- Kind: **Search**
- Link one or more ingestion groups (must include this group's slug)
- Set `search_field` (e.g. `raw.body`, `raw.description`) and `return_fields`

See [tools](tools.md) and [search-fields](search-fields.md).

## Step 7 — Embed data (required before search / LLM features)

> Ingestion alone does not make rows searchable. After ingest (and after later Sheet/API syncs), run
> **Initiate Embedding** on linked search tools — or enable **Auto-embed** on the group —
> and wait for status **Ready** before using project chat, workflows, or other LLM features that rely on search.

On the tool row, click **Initiate Embedding**, or use **Horizon → Data Embeddings**. Wait for status **Ready** (`finished`). If the UI shows **Needs Re-embed**, data was ingested after the last embed — run Initiate Embedding again. See embedding verification in [tools](tools.md).

## Step 8 — Use in projects

Only after embeddings are Ready should you lean on search tools in chat. Project chat agents automatically receive horizon search tools (plus web search if configured). Attach workflow prompts that reference specific tools for narrower tasks. See [projects](projects.md).

## Reingestion and schema changes

> Changing column mapping or raw schema after data exists may require **reingestion**: run **Ingest** again (sync mode updates matched rows) or **Wipe** group data from Settings → Ingestion and pull fresh. After large schema or data changes, re-run **Initiate Embedding** on linked search tools.

- **Group Deduplication** — Add / Skip / Update / Sync with a unique key (`intrinsic.unique_key`). Sync hard-deletes keys absent from the latest ingest. See [api-ingestion](api-ingestion.md).
- **Spreadsheet / Drive modes** — those sources still have their own file/row sync options for Google-native identity.
- **Wipe** — removes entries in the group; irreversible for that group's data.

## Wipe and delete

- **Wipe data** — clears entries; keeps group config.
- **Delete group** — removes group and cascaded associations (admin-level caution).