Sync types: full vs incremental
The decision that defines your cost and how up-to-date your data is. With real example and decision guide.
- Understand the difference between full sync and incremental
- Know how to estimate the impact on cost and updates
- Choose the right sync for each source
Every source you connect needs a decision: with each sync, the Nekt pulls all again or just what changed? This seemingly technical choice directly defines how much you pay and how fresh your data is. It's worth understanding before connecting.
The two types
Full sync (complete). At each run, Nekt reads the entire source and rewrites the table. Simple and error-proof: the destination is always an exact portrait of the origin. The cost is that you reprocesses everything, even what has not changed.
Incremental. With each run, Nekt brings only the
new or changed records since the last sync, using a column of
reference (usually updated_at or an increasing id).
Much cheaper and faster on large tables. The cost is that it requires a
reliable column to know "what's changed".
Incremental depends on a column indicating when the record was
updated. If the source changes a record without stir
in updated_at, Nekt does not see the change and the data remains
out of sync. When this column does not exist, the only option that
guarantees data integrity is full sync.
Nekt always defaults to incremental whenever the source allows filters by update date. The exception is banks data, where the user himself must configure the incremental key for each database table.
The trade-off in practice
Full: extracts 5M rows every hour, 24 times per day. Optimal refresh rate, high and rising cost as the table grows.
Incremental by updated_at: just read the
~2 thousand orders that changed in the last hour. Same rate
update, cost an order of magnitude lower, and even stable
when the table reaches 50M.
The rule of thumb: the larger and busier the table, the more the incremental pays off, especially in databases.
An e-commerce pulled the product catalog (8 thousand SKUs, change little) in daily full sync, and orders (growing 20 thousand/day) in incremental hourly. Each source in its type: products simple and always correct, fresh orders without blowing up the cost. A decision of "which is better" must take into account the context and case of specific use.