Data Source Merging

A Merged data source combines two or more existing data sources into one by joining them on a shared key field — giving workflows a single, enriched dataset.

What it is

A Merged data source is a special type that does not hold its own raw data. Instead, it reads items from a primary data source and enriches each row by joining data from one or more secondary data sources on a shared key field. The result is a new set of items whose data object contains fields from all joined data sources, prefixed to avoid name collisions.

Merged data sources refresh automatically when any of their source data sources update (configurable). The merge algorithm is a standard relational join — left join by default, inner join optionally.

When you'd use it

Your product catalogue lives in one data source (name, description, image) and your pricing table in another (price, discount), both keyed by SKU. You want to render images that include both sets of fields.
You have a base dataset enriched by a regional data source (different prices per region) and need a combined view for each workflow.
You want to cross-reference inventory levels from a stock data source against a product data source, and only generate images for items with stock > 0 (using an inner join to exclude non-matching rows).

Quick start (user)

Create a Merged data source

Go to /dashboard/creative/data-sources and click New Data Source.
Select Merged as the type.
Choose the Primary data source — this is the base dataset; every active item from the primary appears in the merge result (for left joins).
Set the Primary join key — the field in the primary used to match rows (e.g. product_id).
Click + Add Join to attach a secondary data source:
- Data Source — the secondary to join.
- Join key — the field in the secondary that matches the primary join key (e.g. sku).
- Join type — left (keep all primary items, NULL for non-matching) or inner (only items that match in both).
- Prefix — a string prepended to all field names from this secondary (e.g. pricing_ → produces pricing_price, pricing_currency).
- Fields (optional) — limit which fields are pulled in from the secondary. Leave empty to include all.
Add more joins if needed. Each join gets its own prefix.
Click Preview to see a sample of the merged output.
Click Save then Merge Now to run the first merge.

Refresh a Merged data source

Automatic refresh: by default, a Merged data source re-merges whenever any of its sources complete a sync. Disable this in the settings by turning off Refresh on source update.
Manual refresh: click Merge Now on the detail page, or call POST /api/creative/data-sources/[id]/fetch.

Settings reference

Merge configuration

Field	Type	Default	Description
Primary Data Source	data source ID	—	The base data source. Its items form the left-hand side of every join.
Primary Join Key	string	—	Field name in the primary used to match rows.
Joins	JSON array	`[]`	One entry per secondary data source. See join fields below.
Refresh on Source Update	boolean	`true`	Re-merge automatically when any source updates.

Join fields

Field	Type	Required	Description
`feedId`	string	Yes	ID of the secondary data source to join. (Field name retains legacy `feed` naming.)
`joinKey`	string	Yes	Field in the secondary used to match against the primary join key.
`type`	`"left"` \| `"inner"`	Yes	Join type. `left` keeps all primary items; `inner` excludes primary items with no match.
`prefix`	string	Yes	Prefix applied to all field names from this secondary in the merged output. Must be unique across joins.
`fields`	string[]	No	Allowlist of field names from the secondary to include. Empty = include all fields.

Join types

Type	Behaviour
Left	All items from the primary are included. If a secondary has no matching row, those fields are `null`.
Inner	Only items that have a matching row in the secondary are included. Items with no match are dropped from the merged output.

Field prefixing

All fields from each secondary data source are prefixed with the join's prefix value to prevent name collisions. If the primary has a field name and the secondary also has a field name, setting prefix: "inventory_" produces inventory_name in the merged output.

The primary data source's fields are included without any prefix.

Example

Primary item:

{ "product_id": "SKU-001", "name": "Air Max 90", "image_url": "https://…" }

Secondary item joined on sku with prefix: "pricing_":

{ "sku": "SKU-001", "price": 120.00, "currency": "USD", "discount": 0.15 }

Merged result:

{
  "product_id": "SKU-001",
  "name": "Air Max 90",
  "image_url": "https://…",
  "pricing_price": 120.00,
  "pricing_currency": "USD",
  "pricing_discount": 0.15
}

Cycle detection

ConnectWyze uses breadth-first search (BFS) to detect circular dependencies before saving a merge configuration. A merge that would reference itself (directly or through a chain of merged data sources) is rejected with a validation error. This prevents infinite refresh loops.

Cascade refresh

When a source data source finishes a sync, ConnectWyze finds all Merged data sources that reference it (directly or transitively) and re-merges each one in order — deepest dependencies first. This is called cascade refresh.

For example, if Source C merges Source B, and Source B merges Source A:

Source A updates
    │
    ▼
Source B (merges A) → re-merged
    │
    ▼
Source C (merges B) → re-merged

Cascade refresh stops if a merged data source has Refresh on Source Update disabled.

If any source in the chain uses Background Removal, the cascade waits for background-removal processing to complete before propagating.

Developer / API

Create a Merged data source

POST /api/creative/data-sources
Content-Type: application/json

{
  "name": "Products + Pricing",
  "type": "MERGED",
  "mergeConfig": {
    "primaryFeedId": "clxyz123…",
    "primaryJoinKey": "product_id",
    "joins": [
      {
        "feedId": "clabc456…",
        "joinKey": "sku",
        "type": "left",
        "prefix": "pricing_",
        "fields": ["price", "currency", "discount"]
      }
    ],
    "refreshOnSourceUpdate": true
  }
}

(API field names like primaryFeedId and feedId retain legacy feed naming for backward compatibility.)

Update merge configuration

PATCH /api/creative/data-sources/[id]
Content-Type: application/json

{
  "mergeConfig": {
    "primaryJoinKey": "product_id",
    "joins": [ … ]
  }
}

Trigger a merge

POST /api/creative/data-sources/[id]/fetch

Preview merge results

POST /api/creative/data-sources/[id]/merge
Content-Type: application/json

{ "preview": true, "limit": 10 }

Returns a sample of up to limit merged items without writing to the database.

Get available fields

GET /api/creative/data-sources/[id]/fields

Returns the full list of field names available from the merged result. Use these names when configuring field mapping in a workflow.

ASCII diagram

Source A (primary)        Source B (secondary)
  { product_id, name }      { sku, price, currency }
         │                          │
         │    primaryJoinKey        │  joinKey
         │    = "product_id"        │  = "sku"
         │    type: "left"          │
         └──────────────────────────┘
                        │
                        ▼
              Merged Data Source (Source C)
          { product_id, name,
            pricing_price, pricing_currency }
                        │
                        ▼
                   Workflow

Troubleshooting

Symptom	Cause	Fix
Merged data source shows 0 items	Join key mismatch — no rows matched	Verify that the primary join key field and secondary join key field contain the same values. Check for whitespace or case differences.
Missing joined fields on some items	Left join with no match in secondary	Expected behaviour for left join. Switch to inner join if you only want rows that have a match in both.
Items disappearing after inner join	Inner join is filtering out primary items	If some primary items don't have a match in the secondary, use left join instead.
Duplicate field names appear	Two joins use the same prefix	Give each join a distinct prefix.
`Circular dependency detected` error	Merge chain contains a cycle	Remove the join that creates the loop. Merged data sources cannot reference themselves or form a chain back to themselves.
Merged data source not refreshing after source update	`refreshOnSourceUpdate` is disabled	Enable Refresh on Source Update in the merge configuration.
Fields are not available in workflow field mapping	Merge has not run yet, or last merge failed	Click Merge Now and check the merge status on the detail page.

Data Sources — create and manage the sources that a Merged data source combines.
Workflows — map merged fields to template variables.
Background Removal — background-removal data sources integrate with the merge cascade.
Overview — the full pipeline at a glance.

Data Source Merging

What it is

When you'd use it

Quick start (user)

Create a Merged data source

Refresh a Merged data source

Settings reference

Merge configuration

Join fields

Join types

Field prefixing

Example

Cycle detection

Cascade refresh

Developer / API

Create a Merged data source

Update merge configuration

Trigger a merge

Preview merge results

Get available fields

ASCII diagram

Troubleshooting

Related