How it works
How Pipedash loads cached pipelines, polls providers with an adaptive interval, pushes updates to the interface and stores data.
Pipedash keeps a local cache of the pipelines, shows it right away and refreshes it in the background. This page follows one refresh from startup to screen.
Startup
When setup is complete, Pipedash loads every saved provider and starts the refresh loop.
The dashboard does not wait for the network. It shows the last stored pipelines from the database first. The first refresh arrives a moment later and replaces them.
Adaptive polling
One loop wakes every 5 seconds. Each time it fetches any provider that needs an immediate fetch, such as a new one. In active mode it also fetches all providers when the current interval has passed.
| State | Value |
|---|---|
| Starting interval | 10 seconds |
| Check every | 5 seconds |
| Growth | Doubles after 3 unchanged cycles, then on every further unchanged cycle |
| Maximum interval | 300 seconds |
| Reset | Back to 10 seconds when a pipeline changes |
| First cycle | Runs on the first check, because no refresh has happened yet |
A cycle counts as changed when the pipeline count differs, a pipeline is new, or a pipeline has a different status or last run. With no change the interval grows to 20, 40, 80, 160 and then 300 seconds.
Active and idle modes
The mode follows the window. The desktop app is active while its window has focus and idle when it loses focus. In a browser, a visible tab is active and a hidden tab is idle.
In idle mode the loop skips the scheduled fetch. Newly added providers still load, and old metrics still get cleaned up every 6 hours. A fetch from one provider times out after 30 seconds, and a fetch from all providers times out after 60 seconds.
Live updates
The backend pushes an event to the interface when something changes, such as new pipeline data, a triggered run or a provider change. The interface refetches the affected data when an event arrives. The web server sends these events as JSON over a WebSocket. The REST API page lists them.
Provider calls
Each provider is a plugin that fetches pipelines and run history, triggers workflows and cancels runs. Identical concurrent fetches for the same provider merge into one request. When a provider fails, the interface shows its error. The plugin development page describes the plugin interface.
Storage
The pipeline cache, run history cache, workflow parameter cache, provider list and metrics live in one database. SQLite is the default and PostgreSQL is the alternative. Tokens live in a separate token store chosen during setup. The architecture page lists the tables, and the refresh and cache pages describe the user-facing behavior.