A dashboard is only as trustworthy as the pipeline behind it. We build the systems that pull data from the sources an organisation actually has, internal databases, the APIs of the systems already in use, the files teams pass between themselves, and pages where the data exists nowhere else, then clean it, reshape it and load it into the database or warehouse your dashboards and models read from. The whole line runs to a schedule on its own, and it has a clear answer for the run that fails: retries, alerting, and never leaving half a dataset behind.
Who it is for
Organisations whose data sits across several systems and where someone still merges the files by hand every week or every month, and teams about to put a dashboard or a model into production that need the data underneath it to arrive on time and complete.
Collecting from the sources you actually have
We start from the sources you have now, not from a request to replace the source systems first. Real organisational data is usually spread across the databases behind the main systems, the APIs of services you subscribe to, spreadsheets somebody maintains by hand, files exchanged by email or SFTP, and, in some cases, web pages where that data exists nowhere else and scraping is the only way to get at it. Every source has its own limits: how much can be pulled at a time, what access has been granted, how often the source itself really updates. So the extraction is designed separately for each one, then brought together into a single line.
- Internal databases such as PostgreSQL, MySQL and SQL Server
- APIs of the systems in use, from sales to accounting to marketing tools
- Excel files and Google Sheets that a person still maintains by hand
- Files arriving by email, SFTP or a shared folder
- Web scraping, where the data only exists on a page
Cleaned, reshaped, then loaded into the one place your dashboards read
Data from several systems almost never speaks the same language: dates in different formats, customer codes that do not line up, product names spelled inconsistently, duplicate rows carried over from an earlier delivery. This layer is where all of it is made comparable before anyone uses it. Rows that fail their checks are held in a separate table where they can be inspected, rather than quietly dropped so that a total comes out short and nobody knows why. The destination is the database or warehouse your analysts and BI tools read from directly, whether that is PostgreSQL on your own server or a warehouse on GCP, Azure or AWS, depending on what your organisation already runs.
- Dates, numbers and units standardised to one format
- Customer and product codes reconciled across systems
- Duplicates removed, and back-dated records handled
- Rows that fail validation kept in a separate table, never dropped in silence
- Load only what changed, or reload the whole set, whichever suits
Scheduling and orchestration
Which tool we reach for depends on who has to look after the job afterwards. Where a flow is mostly a matter of connecting services to one another, we build it in n8n or Make.com, because your own team can open it, see what each step does, and change a small condition without waiting for a developer. Where the transformation is heavy, the business rules are involved, or the volume is large enough that performance has to be managed, we write the job in Python, kept under version control and testable. Both kinds sit in the same system quite happily. What cannot be skipped is order: a step must never run on the output of an earlier step that has not actually finished.
- n8n and Make.com for flows your team should be able to open and adjust
- Written Python jobs for heavy or complicated transformations
- Hourly or daily schedules, or a trigger on an arriving file or webhook
- Enforced order, so a step starts only once the one before it has finished
The failed run, which is the part most pipelines get wrong
The failure everyone prepares for is the one that stops outright, and that one is easy to spot. The dangerous failure is the run that half succeeds and leaves an incomplete dataset in a table that looks complete. So writes land as a single unit: if the whole batch does not succeed, none of it is committed and the previous data stays exactly where it was. Transient problems, a dropped connection or an API that will not answer, are retried a set number of times, and only then does an alert go out, naming the source, the step and the run. Silence is not a sign that all is well, so a run that has not started by the time it should raises an alert too. Every run is recorded: start, finish, rows in and out, and status, so that whether today's number is late or simply wrong is a question with an answer. This is how we run our own estate, roughly twenty applications on one host, deployed by script rather than by hand, and health-checked at the port rather than assumed to be up.
- Automatic retries for transient faults such as a network or API failure
- All-or-nothing writes: if the batch does not complete, nothing is written
- Alerts both when a run fails and when a run has not started on time
- Every run recorded: start, finish, row counts and status
- Traceable answers to which data arrived when, and from which run
What it does
- Extraction from databases, APIs, files and web pages in one line
- Validation and reshaping to a standard before anything is loaded
- Loaded into the database or warehouse your dashboards read directly
- Hourly, daily or event-triggered schedules
- n8n and Make.com alongside written Python jobs
- Automatic retries, and alerts when a run fails
- All-or-nothing writes, so no half-loaded dataset
- A record of every run: start, finish, row counts and status
Want something like this?
Tell us your situation and we will work out what needs adapting to fit it.