Pivoting an Integration to Source SFTP Two Weeks Before Contract End
How discovering a labelled proof of concept in the production data path forced a contained rewrite without slipping the ship date
The challenge
The original integration plan for OneFile, agreed up front during discovery, was that the e-portfolio's overnight CSV exports would arrive on a secure FTP server, be lifted into Azure Blob storage by an existing pipeline that another part of the organisation already maintained, and the progress engine would pick them up from there. On paper this was the path of least resistance: someone else owned the SFTP-to-Blob hop, the dashboard team owned the Blob-to-database hop, and the line between us was clean.
About two weeks before the contract end the picture changed. As I dug into the actual behaviour of the upstream pipeline (something nobody on either side had felt the need to revisit during the build) it became clear that it was not a production data path. It was a labelled proof of concept built in a low-code drag-and-drop tool, living in a different Azure subscription with its own boundary concerns, and there was no one inside the wider organisation prepared to take responsibility for promoting it to production by our deadline.
The choice was uncomfortable. Inheriting the upstream pipeline would have meant absorbing infrastructure I did not own, in a subscription I did not have proper access to, in a tool none of the team were comfortable maintaining, two weeks before ship date. Waiting for someone else to productionise it was not a real option against the Ofsted-driven deadline that was driving the wider programme.
The results
Key results
- Discovered with two weeks to go that the planned ingest pipeline was a labelled PoC sitting in a different Azure boundary, with no owner for production hardening
- Rewired the OneFile integration to read directly from the source SFTP, removing reliance on a non-production data path
- Change contained inside the progress engine boundary: the BFF contract and the front end did not move
- MVP shipped at the end of the engagement, roughly two weeks after the Ofsted inspection itself, with the build state at inspection enough for the client to evidence the gap was being actively addressed
- Avoided inheriting an untested low-code data pipeline as a production dependency
- Flagged the original infrastructure assumption gap in writing for the in-house team to act on after the engagement
The MVP shipped at the end of the engagement, roughly two weeks after the Ofsted inspection itself. The build state at the inspection date, including the rewritten OneFile ingestion path, was enough for the client to evidence to inspectors that the apprenticeship visibility gap was being actively addressed. The OneFile data flow ran end-to-end on the rewritten ingestion function. The dashboard, the BFF and the progress engine's external contract all stood unchanged through the pivot. There was no scramble in the front-end code, no migration in the database, and no last-minute surprise for the rest of the team.
The other outcome was that the dashboard programme stopped having a non-production dependency on a labelled proof of concept that nobody owned. That on its own is the kind of result that is hard to put in a metric, but it was the right thing to leave behind. The in-house team inherited a data path they could maintain without depending on a low-code pipeline in a different Azure boundary.
The lesson I took away (and have applied since) is to validate every external infrastructure assumption with a smoke-test in the first two weeks of an engagement, not the last two. The original assumption, "someone else owns the SFTP-to-Blob hop", was reasonable on the face of it. It was also wrong in a way that nobody noticed until the build was nearly finished. A 30-minute conversation about who actually owned what, in week one, would have surfaced the same problem at a point when the response could have been a normal architectural choice rather than a deadline pivot.
The solution
I pivoted the OneFile integration to read directly from the source SFTP feed. The existing OneFile ingestion Azure Function, which had been written to pick up files from Azure Blob, was repointed at the SFTP server, given the parsing responsibility for the CSV exports, and made responsible for landing the parsed data into the progress engine's own store. The intermediate Blob hop was kept (because it gave us a useful artefact for replay and debugging) but it now belonged to us, not to the upstream PoC pipeline.
The pivot was deliberately a contained rewrite rather than a rebuild. The progress engine's contract upstream of the ingestion function did not change. The BFF's contract did not change. The front end did not change. The blast radius of the late pivot was confined to the OneFile ingestion function itself and the function's deployment configuration. That containment was only possible because of the BFF and progress engine separation that had been in place from the start of the build (covered in a separate case study); the layered architecture earned its keep at exactly the moment a layered architecture is supposed to.
In parallel I documented two things in writing. First, the technical rollback plan, in case the SFTP reroute hit a problem we had not foreseen during the remaining time. Second, a clear note for the in-house team about the upstream pipeline's status, what it actually was, and what they would need to do if they ever wanted to revisit the original integration shape after the engagement.
Technical deep dive
What the original pipeline actually was
The original SFTP-to-Blob pipeline was implemented in a low-code drag-and-drop Azure tool, living in a separate subscription from the dashboard programme. It had been built as a proof of concept for an unrelated piece of work, was tagged as such in the resource group, and had no production runbook, monitoring, or named owner for incident response. From the dashboard programme's point of view it had been treated as a black box that already existed, which is exactly the assumption that did not survive contact with reality.
The reroute
The OneFile ingestion Azure Function had three responsibilities after the pivot:
- Connect to the source SFTP server on a schedule, on credentials that the dashboard programme owned.
- Pull the available CSV exports, parse them, and validate the records against the schema the progress engine expected.
- Land the parsed records into the progress engine's store, with the raw CSV preserved in our own Blob container as a replay and audit artefact.
The function's contract with the rest of the system (what it produces, where it produces it, and on what cadence) did not change. That is what made the rewrite contained rather than systemic.
Why I did not try to fix the original pipeline
A reasonable reader may ask why I did not try to take ownership of the upstream PoC pipeline and harden it for production rather than rerouting around it. The honest answer is boundary cost. Promoting a low-code drag-and-drop pipeline in a different Azure subscription, owned organisationally by a different team, with no current runbook, into a production dependency the dashboard programme could rely on, in two weeks, was not a credible commitment. Building a small, owned, testable .NET path to the same data was. Boundary cost is one of the variables I weigh hardest under time pressure, and it pointed clearly in one direction.
The rollback plan
The rollback plan was straightforward: if the SFTP reroute hit a blocker we had not foreseen, the original ingestion code path was preserved on a feature branch and could be redeployed against the upstream PoC pipeline within a deployment cycle. We did not need to use it. It existed because in a deadline pivot the cost of preparing the rollback is small and the cost of needing one and not having one is enormous.
What I would do differently
The change I would make to my own process is structural, not technical. In the first two weeks of any engagement that includes external data feeds, I now run a deliberate "trace the bytes" exercise: pick one record, follow it from the source system through every infrastructure hop until it appears in the consuming system, and write down who owns each hop and what the production status of each hop is. That exercise would have surfaced the upstream PoC pipeline in week two of this engagement, not week thirteen. The pivot itself was the right call once the problem had been found. The cost is in finding the problem late.
Ready to achieve similar results?
Let's discuss how we can help your organisation achieve these results.
Book a strategy call