Data Sources
A data source is a named collection of structured data that supplies the per-item variables (text values, image URLs, etc.) that ConnectWyze uses to render your creative assets.
What it is
A data source holds a collection of items — structured rows of data. Each item carries an externalId (a stable identifier from the source) and a data JSON object containing all of its fields.
When a workflow runs, it reads the active items from a data source and maps their fields to template layer variables. The data source is the only place item data lives; the template and workflow never store item data directly.
Data sources come in five types:
| Type | Description |
|---|---|
| API | ConnectWyze fetches data from a remote HTTP endpoint on a schedule or on demand. |
| CSV | You upload a CSV file; ConnectWyze parses and stores the rows as items. |
| Manual | You enter and manage rows directly in a table editor in the dashboard. |
| Merged | Combines two or more existing data sources using a shared join key. No raw data of its own. |
| Background Removal | Reads image URLs from a source data source, strips the background, and stores the processed URLs as new fields. |
When you'd use it
- API — your product data lives in an e-commerce platform, PIM, or Google Sheets and exposes a JSON endpoint. You want the data source to refresh automatically.
- CSV — you receive a weekly data export and want to drop it in without building an integration.
- Manual — you have a small, curated dataset (10–200 items) that you maintain directly, or you're prototyping a workflow before connecting a real data source.
- Merged — you need to combine a product catalogue with a pricing table that share a SKU. See Data Source Merging.
- Background Removal — you want product images with backgrounds removed before they reach your template. See Background Removal.
Quick start (user)
Create an API data source
- Go to
/dashboard/creative/data-sourcesand click New Data Source. - Select API as the type.
- Enter the endpoint URL. Set the HTTP method (GET is the default).
- Under Response mapping, enter the JSON path to the array of items (e.g.
data.products). - Set an ID strategy: use a specific field (
field), hash all data (hash), or use row index (index). - Optionally add a cron Schedule (e.g.
0 */6 * * *for every 6 hours) and a timezone. - Click Save, then click Sync Now to run the first sync.
Create a CSV data source
- Click New Data Source and select CSV.
- Click Upload CSV and choose your file.
- Map which column to use as the external ID (or let ConnectWyze hash the row).
- Click Save to import the rows as items.
To update data, return to the data source and upload a new CSV. Existing items are matched by externalId and updated in place; new rows are added; removed rows are marked inactive.
Create a Manual data source
- Click New Data Source and select Manual.
- Define your Schema: add columns, set each column's type (Text, Number, URL, Image, etc.), and mark required columns.
- Click Save to create the data source.
- Open the data source and click Add Item to start entering rows.
Settings reference
Data source settings
| Field | Type | Default | Description |
|---|---|---|---|
| Name | string | — | Display name for the data source. |
| Description | string | — | Optional notes. |
| Type | API | CSV | Manual | Merged | Background Removal |
— | Set at creation; cannot be changed. |
| Schedule | cron string | null |
Cron expression for automatic refresh (API data sources). null = manual only. |
| Timezone | string | UTC |
Timezone for interpreting the schedule. |
| Is Active | boolean | true |
Inactive data sources do not refresh automatically. |
| ID Strategy | field | hash | index |
hash |
How externalId is assigned to each row. |
| ID Field | string | — | Required when ID Strategy is field. |
| Mirror Images | boolean | false |
When enabled, URLs in imageFields are fetched through the URL proxy and copied to workspace storage. Useful for source CDNs that block direct Playwright access. |
| Image Fields | string[] | [] |
Field names containing image URLs to mirror. Only active when Mirror Images is on. |
| Hash Fields | string[] | [] |
Limit change-detection hashing to these top-level fields. Useful to ignore noisy metadata that changes every fetch without real data changes. Empty = hash the entire row. |
API data source settings
| Field | Type | Default | Description |
|---|---|---|---|
| URL | string | — | The remote endpoint to fetch. |
| Method | GET | POST |
GET |
HTTP method. |
| Headers | JSON object | — | Additional request headers (e.g. Authorization). |
| Body | JSON | — | Request body for POST requests. |
| Data Path | string | — | Dot-notation path to the items array in the response (e.g. data.products). Leave empty if the response is the array itself. |
| Total Count Path | string | — | Path to the total item count for pagination (e.g. meta.total). |
| Pagination Type | none | url_path | query_param | cursor | offset_limit |
none |
How the fetcher pages through results. |
| Pagination Config | JSON | — | Pagination parameters; shape depends on type (e.g. { "pageParam": "page" } for query_param). |
Schema columns (Manual data sources)
Manual data sources require you to define a schema before entering data. Each column has:
| Field | Options | Description |
|---|---|---|
| Name | string | Column identifier used in field mapping. |
| Type | TEXT | NUMBER | URL | IMAGE | VIDEO | DATE | BOOLEAN | SELECT |
Data type — drives the editor input and validation. |
| Required | boolean | Whether a value is mandatory on every item. |
| Default | — | Pre-filled value for new items. |
Items
Each item has:
| Field | Description |
|---|---|
externalId |
Stable identifier from the source. Used as the primary key for upsert on re-sync. |
data |
JSON object containing all field values for the item. |
status |
ACTIVE or INACTIVE. Inactive items are skipped by workflows. |
Tags
Tags let you label items for filtering and conditional template selection.
- Create tags (name + colour) at the data source level.
- Assign one or more tags to individual items.
- In a workflow, filter which items to process by tag.
Media
For Manual data sources, image and video columns store files as FeedItemMedia records rather than plain URLs. Each media record holds the storage URL, filename, MIME type, dimensions, and an optional focal point (X/Y percentage) that controls crop centering when the image is placed in a template layer.
Focal point
The focal point is stored per media file. When an image layer in a template has a variable bound to a media column, ConnectWyze uses the focal point to crop the image via CSS object-position. You can override the focal point per-render in the workflow's field mapping.
Mirroring images
When Mirror Images is enabled, ConnectWyze fetches each URL listed in Image Fields through the URL proxy, optimises the image (resizes to 4096 px max, converts to JPEG at quality 95), and uploads it to your workspace storage. The item's data URL is replaced with the workspace storage URL. Subsequent workflows read the mirrored URL, which eliminates Cloudflare bot-management blocks and removes a network dependency at render time.
Mirroring is triggered automatically after each data source sync. Progress is visible in the Image Mirror Runs panel on the data source detail page.
Developer / API
Data source endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/creative/data-sources |
List all data sources in the workspace. |
| POST | /api/creative/data-sources |
Create a data source. |
| GET | /api/creative/data-sources/[id] |
Get data source details. |
| PATCH | /api/creative/data-sources/[id] |
Update settings. |
| DELETE | /api/creative/data-sources/[id] |
Delete data source and all items. |
| POST | /api/creative/data-sources/[id]/fetch |
Trigger a manual sync. |
| GET | /api/creative/data-sources/[id]/fields |
List available field names (used by workflows for field mapping dropdowns). |
Item endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/creative/data-sources/[id]/items |
List items. Supports ?page, ?limit, ?status, ?search, ?tag. |
| POST | /api/creative/data-sources/[id]/items |
Create one or more items (Manual data sources). |
| GET | /api/creative/data-sources/[id]/items/[itemId] |
Get a single item. |
| PATCH | /api/creative/data-sources/[id]/items/[itemId] |
Update item data or status. |
| DELETE | /api/creative/data-sources/[id]/items/[itemId] |
Soft-delete (sets status to INACTIVE). |
| POST | /api/creative/data-sources/[id]/items/bulk |
Bulk create, update, or delete. |
Media endpoints (Manual data sources)
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/creative/data-sources/[id]/items/[itemId]/media |
Upload a media file. |
| GET | /api/creative/data-sources/[id]/items/[itemId]/media |
List media for an item. |
| PUT | /api/creative/data-sources/[id]/items/[itemId]/media |
Reorder media files. |
| PATCH | /api/creative/data-sources/[id]/items/[itemId]/media/[mediaId] |
Update focal point. |
| DELETE | /api/creative/data-sources/[id]/items/[itemId]/media/[mediaId] |
Delete a media file. |
Tag endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/creative/data-sources/[id]/tags |
List tags for the data source. |
| POST | /api/creative/data-sources/[id]/tags |
Create a tag. |
| PATCH | /api/creative/data-sources/[id]/tags/[tagId] |
Update tag name or colour. |
| DELETE | /api/creative/data-sources/[id]/tags/[tagId] |
Delete a tag. |
| POST | /api/creative/data-sources/[id]/items/[itemId]/tags |
Assign tags to an item. |
| DELETE | /api/creative/data-sources/[id]/items/[itemId]/tags |
Remove tags from an item. |
Import/Export
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/creative/data-sources/[id]/import |
Import items from a CSV payload. |
| GET | /api/creative/data-sources/[id]/export |
Export items as CSV. |
ASCII diagram
Data Source (type: API)
│ schedule: "0 */6 * * *"
▼
Sync run ──► remote endpoint (apiUrl)
│ paginate if configured
│ map to items (externalId + data)
▼
items[] ──► status: ACTIVE / INACTIVE
│ ├── data: { product_name, price, image_url, … }
│ ├── media: [FeedItemMedia { url, focalX, focalY }]
│ └── tags: [FeedItemTag → FeedTag { name, colour }]
│
▼
Workflow reads items → field mapping → Template render
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Data source shows 0 items after sync | Data path misconfigured | Check Data Path — verify it points to the array in the response JSON. |
| Items show duplicate entries | ID strategy produces collisions | Switch to hash or index strategy, or pick a truly unique field for field strategy. |
| Images are blocked or return 403 during render | Source CDN (Cloudflare) blocking render worker's IP | Enable Mirror Images for the data source and add the image field names to Image Fields. |
| CSV import produces validation errors | Required column missing or wrong data type | Review the error list shown in the import modal; correct the CSV and re-upload. |
| Focal point not saving | Clicking outside the crop overlay before saving | Click the Save button inside the focal point modal, not outside it. |
| Tags not colouring in the grid | Browser cache | Hard-refresh the page (Cmd+Shift+R / Ctrl+Shift+R). |
| Data source stops auto-refreshing | Set to inactive | Check the Is Active toggle on the settings page. |
| Background-removal data source stuck processing | Source images failing proxy fetch | Check image mirror logs; verify source URLs are reachable. |
Related
- Data Source Merging — combine two or more data sources with join logic.
- Workflows — map data source fields to template variables and run generation.
- Templates — build the visual canvas.
- Background Removal — strip backgrounds from data source images.
- Overview — the full pipeline at a glance.