Monitoring pipelines and runs
In the event of a failure, the first step is to open the run and read the log and analyst summary that Nekt generates through AI. He already says to cause and what to do. How to diagnose, act and reprocess before the customer notices.
- Open the run and read the log and analyst summary
- Follow recommended actions and reprocess
- Monitor the health and currency of data
Scheduling a pipeline is a promise: the data will be there, updated when someone needs it. Monitoring is ensuring that this promise is fulfilled. The worst way to find out that a pipeline broken is due to the customer asking why the number is outdated. When this happens, you have lost a piece of confidence. This guide is about getting to the glitch before it gets to the customer, and about the most important thing when it happens: at Nekt, you almost never have to guess what went wrong. Each run Leaves a detailed log and AI-written summary that explains the cause and what to do.
Why monitor
Static data breaks trust faster than wrong data. One wrong number sometimes goes unnoticed, but a frozen dashboard on a Monday morning it is noticed right away. And the problem almost never is dramatic: a sync that failed in the early hours of the morning, a credential that expired, a data source that changed schema without warning. All this, if no one is looking, it only appears when it has already become a complaint. Monitoring is transforming these silences into signs that you see first.
The execution history
Each time a pipeline runs, it generates a run: one record of what happened during that execution. The run history is the your health dashboard. Three things count more than the rest:
- Status: ended successfully or failed? It's the first look.
- Duration: How long did it take? A run that usually take 2 minutes and suddenly it takes 40 minutes is a warning, even if it has given "success".
- Volume: how many lines entered? A very large number Out of the ordinary (more or less) deserves a look.
A typical cut of recent rums looks like this:
When faced with a failure: Read the analyst log and summary
Here's the most important part of this guide, and the one that changes your experience the most. day to day. When a run fails, the first measure no is to go out and investigate the source, not to touch the transformation, not to open the provider dashboard. The first measure is open the failed run and read what Nekt has already prepared for you: the detailed log of that execution and the analyst summary, an analysis that Nekt generates automatically by AI for each run.
For critical pipelines, you should always enable alerts to receive notification of failures via email, so you can take action before the problem even broke out.
The analyst summary reads the log for you and writes it, in language Simple, two things: what happened and the recommended actions. In practice, it is already say the cause (e.g. "credential expired, 401 Unauthorized") and what to do to resolve (update credentials, reconnect the source, reprocess). You don't need to decipher stack trace: Nekt translates the technical error in an explanation and action plan.
What happened Your connection to the source failed because the credentials are not more authorized. Source responded 401 Unauthorized when trying authenticate, the run stopped right at the beginning and no lines were processed. Recommended actions 1) Update source credentials 2) Check source permissions connected account 3) Reprocess the pipeline
Realize what this saves. Without the summary, a
401 Unauthorized in the middle of a log of hundreds of lines
It requires you to know where to look and what it means. With the
summary, the answer arrives written: the cause is the credential, the action is
reconnect and reprocess. The log remains there, in one piece, for when
you want the detail. But the starting point is no longer the
investigation and becomes reading.
The script, when a run fails, becomes three steps:
- Open the run and read the analyst summary and log. Start with “What happened”. He almost always already points out the cause. O log is just below if you need to confirm the detail.
- Follow the recommended actions. Nekt lists, in order, what to do: update credentials, reconnect the source, adjust the transformation, check permissions. Run in the suggested order.
- Reprocess. Once the cause has been corrected, run again and confirm by history: green status and volume as expected.
Before touching anything, reprocess it once. A real part of failures is transient: the source flickered, the network dropped for a instant. If the second run passes clean, you have saved a investigation. If it fails again, then open the run: the summary of the analyst and the log will tell you if the cause is structural and what it is.
Translating what the summary points out
The analyst's summary already provides the cause and action in each case. Still So, it helps to have a map of the most common fault families in your pocket, to recognize the pattern when reading:
| What the summary usually points out | Cause behind | Recommended Action |
|---|---|---|
| 401/unauthorized credential, fails early on | Authentication: source token or credential has expired | Update credentials, reconnect source, reprocess |
| Error quoting a non-existent column or field | Schema change: the source renamed or removed a field | Adjust the transformation to the new schema, reprocess |
| Run green, but 0 rows processed | Empty filter, source with no new data, or wrong date window | Check direct source and filter logic |
| Timeout or duration far above normal | Volume too large, or source that is currently slow/unstable | Reprocess; if repeated, reduce the window or grain |
| Only one pipeline that depends on another fails | The source pipeline failed before (ripple effect) | Fix the origin first, then reprocess the one below |
Alerts: be warned, don't find out by hand
Looking at your run history every morning works until the day you forget it, or you're on vacation, or the workspace grows to 40 pipelines. Monitoring by hand does not scale. The ultimate goal is to reverse the logic: in Instead of going after failure, failure comes to you. A warning when a failed run arrives before the customer notices. And when the warning comes, the script is always the same: open the run, read the analyst's summary and the log, follow recommended actions, reprocess. Start simple, a heads up in the most critical pipelines it already changes the game, and cover the rest.
At one agency, a client's report stopped updating in a Friday. Nobody saw it. On Monday the client asked why the media numbers were the same as the previous week. When opening the run, the analyst's summary said it all: "What happened: the token of Meta Ads has expired and the source responded 401. Recommended actions: reconnect the source and reprocess." Five minutes to resolve, without having to hunt for the cause. After that, the agency began to monitor two things in every customer pipeline: currency (stopped for how long) and volume (brought fewer lines than the normal), and adopted the habit of always opening the run and reading the summary before touching anything. The warning now comes on Saturday morning, not on Monday's complaint.