Efficient triggers and scheduling
Rotating more times leaves the data more updated and spends more credits. O The game is to find the frequency that the business needs and chain the rest together event.
- Balancing updated data and cost in credits
- Chain query and notebook by event, without scheduling everything on the clock
- Save credits with business hours, incremental sync and fewer orphaned pipelines
One trigger is the trigger that triggers an execution of pipeline. It could be because time (turn every hour hour, every day at 6am) or by event (run as soon as the source I depend on finishes running). Each trigger pulls the updated data from the source, goes through the transformation (a query or a notebook) and arrives at the ready table that the team consumes.
It seems like a configuration detail, but this is where you decide whether the data will be up to date at the time of the decision and how much it will cost in credits. Two choices, same button.
The central trade-off: updated data vs. cost
The rule is simple: the more times the pipeline runs, the more current the data stays, and the more credits he spends. Rotating every hour leaves everything almost in real time, but multiplies the executions (and the count). Rotate once a day is cheap, but the data can be up to 24 hours late. Updated data and cost go together, in the same direction.
The curve rises together: more frequency means more current data and counts higher, always in the same direction. There is no "right" frequency in vacuum, there is the frequency that the business needs. A sales dashboard that the team looks at in the morning doesn't gain anything by running every hour night. A fraud alert that needs to react quickly justifies the cost of always running.
Start with the question "how late does the business make the decision?" right?" If the answer is "it'll do in the morning", schedule it for dawn, once a day. Raise the frequency only when someone missing more current data, not as a precaution.
Event chaining: fire at the end of execution, not on the clock
One source, when it finishes rotating, it can fire automatically a query or a notebook. This is event chaining: instead of mark each step on the clock, you say "when this is finished, turn the next." The query only processes data that has just arrived, and the notebook It only runs after the query completes.
The alternative is to schedule everything at a fixed time (source at 6am, query at 6:05 am, notebook at 6:10 am), breaks on the day the source is late: the query runs on top of the old data and propagates the delay throughout the chain. No one notices until the number is wrong. Chaining by event, the chain adjusts itself if something takes time, and you never run over outdated data.
Put the frequency in one place, in the source, and leave query and notebook per event. This way you control how current the data is at a single point, and the rest of the chain follows the rhythm naturally, without Magic schedules to keep in sync.
Credit saving tips
After choosing the frequency, you can cut credits without losing nothing of value. Most of the waste comes from running when no one consumes or reprocesses since it has not changed.
| Tip | Why save |
|---|---|
| Operate only during business hours | If no one consults the data in the early hours of the morning, schedule from 10pm to 6am only burns credit without anyone reading the result. |
| Only run on weekdays | A report that the team looks at during working hours does not need to be updated Saturday and Sunday. There are two days of executions cut by week. |
| Use incremental sync | Brings only what changed since the last run, instead of reprocess the entire base every time. |
| Group runs | Focus a few shots on appointment times (beginning, middle and end of work hours) instead of spreading rounds throughout the day. |
| Chain by event | Query and notebook run when the source ends, not in windows fixed. Nothing goes off waiting for nothing because it hasn't arrived yet. |
| Shut down orphan pipelines | A sync that feeds a dashboard that is already turned off continues to cost dawn without delivering anything. Pausing is pure credit back. |
One orphan pipeline is the one that keeps running agenda, but whose results no one consumes anymore. The data source has been changed, the dashboard was turned off, the project was over, and the sync is still going strong. Because each alone looks cheap, they pile up unnoticed. To find: list the sources and transformations with active trigger, ask for each ready-made table who still reads it (dashboard, MCP or Data API) and, if no one consumes the end of the chain, pause the trigger that feeds without going out. After a while without complaining, you can archive securely.
Before disconnecting, confirm that the table is not input from another transformation. It may appear to be without direct consumption, but food another table per thread. Follow the chain to the end before pause any trigger.
Decide now
A client ran all data sources hourly "just in case", seven days a week, including a financial export that only closed the day at midnight. There were 24 daily executions for a given that changed once. We download this source for 1x a day at dawn on a business day, we turn on incremental sync and chain the query and the notebook per event. The financial panel continued ready before 8 a.m., and executions at that chain dropped significantly. from 160 per week to 5.