> Public product help. This page has no customer entries, projects, chat transcripts, API keys, or horizon contents.
> Reading that data requires `Authorization: Bearer` (a signed-in app session JWT, or a Headless access token).
> Do not call Supabase `/rest/v1` with a Headless JWT. Data reads go through `/casc/v1/headless/v1/rest`.

# Ingestion Source Types

Ingestion groups pull or receive data into **entries** — one row per record, with a `raw` JSON object holding column values and optional file metadata. Configure sources in **Settings → Ingestion**. Run pulls from **Horizon → Data Ingestion**.

## Source type comparison

| UI label | Config `type` | Best for | Scheduled? |
| --- | --- | --- | --- |
| None | `none` | Manual entry only via Horizon → Data Management or API | No |
| Google Spreadsheet | `google_spreadsheet` | Tabular data in Sheets; column mapping to raw slugs | Yes |
| Google Drive Folder | `google_drive_folder` | Files in a folder (PDF, Docs, etc.) with optional recursion | Yes |
| API Source | `api_source` | HTTP JSON feeds (custom or XE.net currency template) | Yes |
| Transformation | `transform` | Explode / join / filter / aggregate across groups into a new group | Yes (via Ingest) |

> **File upload is not a pull source.** Files are attached to individual entries (Horizon → Data Management → Add Entry, or the entries API). Drive folder ingestion *imports* files from Google; manual upload stores files on entries you create yourself.

## Google Spreadsheet

### Configuration

- **Spreadsheet ID** — from the Sheet URL.
- **Sheet name** — optional; first tab if omitted.
- **Column mapping** — each column maps to a `raw` slug (e.g. `company_name`, `revenue`). Link columns can point to URLs validated on ingest.
- **Mode** — `add_only` (append new rows) or `sync` (match and update by intrinsic keys such as Drive file id).

### Before you run

Connect Google under Settings → User (see [google-auth](google-auth.md)). The Horizon status bar shows whether your personal account is connected.

## Google Drive Folder

### Configuration

- **Folder link or ID** — root folder to scan.
- **Recursive** — include subfolders when enabled.
- **Allowed MIME types** — restrict to PDF, Google Docs, etc.
- **Mode** — `add` or `sync` (dedupe by file id).
- **Process mode** — per-file-type handling (extract text vs store file reference).
- **Directory depth columns** — optional path segments as raw fields for nested folders.

Each file typically becomes one entry. Text extraction populates searchable raw fields; binary files use `_file_*` metadata for download and embedding.

## API Source

### Providers

- **Custom API** — define endpoint, GET/POST, auth (none, bearer, basic, API key), query/header/body parameters, JSON `data_path` to the array of records, and field mapping into `raw`.
- **XE.net** — currency rates template; system credentials in Settings → System (`XE_NET_ACCOUNT_ID`, `XE_NET_API_KEY`).

### Deduplication

Configured on the **group** (Deduplication section), not on the API provider panel.
Modes: Add, Skip Duplicates, Update Duplicates, Sync. See [api-ingestion](api-ingestion.md).

### Test before production

Use the **Ingestion API** tab in Help → System Test, or the harness in the ingestion editor, to preview request/response. Saved group config is what scheduled and manual runs use — not one-off harness overrides.

See [api-ingestion](api-ingestion.md) for unique-key details and troubleshooting empty pulls.

## Transformation

A Transformation group runs a visual pipeline over one or more source groups on the same horizon:
explode nested arrays, join on keys, filter/sort with **Zagger Views**, aggregate, and calculate.
Preview without writing; materialize with Horizon → Data Ingestion. See [transform-ingestion](transform-ingestion.md).

## SQL Transformation

A SQL Transformation runs a multi-statement SQLite script against read-only source tables (`raw` JSON).
The final `SELECT` becomes this group’s rows. See [transform-sql-ingestion](transform-sql-ingestion.md).

## Manual entries and file upload

For groups with source type **None**, or to supplement any group:

1. Go to **Horizon → Data Management**.
2. Select the ingestion group and click **Add Entry**.
3. Fill raw fields and attach files; files are stored as `raw._file_<id>` with searchable `raw.file_filename` when configured.

Programmatic alternative: `POST /suffixes/{suffix}/groups/{groupSlug}/entries`.

## Shared group options

- **Auto-approve on ingestion** — skip Data Review; entries marked reviewed and approved immediately.
- **Auto-embed on ingestion** — queue embedding for linked search tools after new/updated rows (search tools only). **Recommended** for Google Sheet and other recurring syncs so new rows do not stay invisible to search. If Auto-embed is off, run **Initiate Embedding** after each ingest before significant LLM / search-tool use. See [tools](tools.md).
- **Private** — group visible only to owner and admins (default is private).
- **Horizon assignment** — which suffixes see this group (Assign on the ingestion card).