Uchit Vyas

Work

One archival platform for 1,000+ applications.

One place to keep a bank’s records, so its old systems could be switched off.

Client
Tier-1 bank in Australia and New Zealand
Year
2024
My role
Solution architect
Platform
AWS, built in-house

1,000+ applications. One archive.

$50M+

Saved each year

1,000+

Applications onto one platform

15TB

Of data, one set of retention rules

The problem

Old systems stayed on only to hold their data.

The move

One AWS platform for every data type, run by the bank.

The result

Teams switch old systems off and keep the records.

  1. 3 monthsDiscovery of all 1,000+ apps
  2. StructuredPhase 1
  3. Semi-structuredPhase 2
  4. UnstructuredPhase 3
Rolled out by type of data, easiest first, so the team learned on the simpler phase.
How data moves from source systems into the archive and back out to the teams who need it. Everything is built with Terraform and deployed through CI/CD.

Built on AWS.

Processing
AWS Lambda
Orchestration
AWS Step Functions
Data movement
AWS DataSync
Storage
Amazon S3
Metadata and search
MongoDB Atlas
Data lineage
Alation
Infrastructure
Terraform
Classification
Automated metadata analysis
The full story

Old systems stayed on just to hold their data.

The bank had no shared way to archive data. Each application team had built its own, with its own tools, its own process and its own bill.

Regulation says some records must be kept for years. Because there was nowhere central to keep them, old systems stayed switched on long after anyone used them, just to hold their data.

1,000+ applications and every kind of data.

  • Scale. More than 1,000 applications and 15 TB of data, spread across on-premises systems, other clouds and other regions.
  • Mixed data. Database tables, files, documents and logs all needed the same retention, privacy and audit controls.
  • Adoption. Application teams had to choose to move. If getting data back out was slow or awkward, they wouldn’t.

One AWS platform, rolled out by data type.

We spent three months assessing all 1,000+ applications before designing anything. We compared off-the-shelf products against a custom build on data patterns, compliance needs and cost. A custom AWS platform won.

We rolled it out by type of data rather than by application: structured data first, then documents and files. Retention, expiry and deletion run automatically. Teams get their data back through self-service, with access controlled by their existing Active Directory groups.

The bank’s own team runs it. We handed over full design documents, runbooks and review processes.

Four decisions that made it work.

  • Three months of discovery before any design. We looked at data volumes, formats, retention rules and regulatory limits for every application, then built decision trees to choose between cloud services, an enterprise product and a custom build.
  • One platform for every data type. A single application often holds tables, files and documents. Handling all of them in one place meant an application could be switched off in one go.
  • Roll out by data type. Structured data first, then semi-structured and unstructured. Each phase was harder than the last, and the team learned on the easier one.
  • Ship the operating model, not just the platform. Onboarding guides, runbooks, security documentation and review processes, so new applications could join without us.