The Appsbroker Data Migration Strategy Checklist

Over the years, Appsbroker’s team has helped many organisations to manage successful data migration projects and to avoid the sorts of pitfalls that can easily derail efforts. In this post we’ll have a look at typical checklist questions that we use with clients to ensure the right results, and then cover off the three main challenges that can be faced during data migrations.

Data migration projects are journeys and so as a starting point we categorise checklists into Scoping, Discovery and Delivery.

1. Identify Value (Scoping)

The first question to answer is around objectives and scoping - how does a cloud data migration provide data transformation?

Does it deliver:

Additional business capabilities (e.g. advanced analytics, real-time insight, faster analysis over large dataset)?
Reduced cost of ownership?
The joining of multiple data sets together that otherwise exist in silos, and hence providing additional insight?
Democratised analytics and insight across an organisation?

For example, this line of questioning could have the output that “our fraud detection model would deliver an additional £3m per annum of benefit if we could run over 24 months of data rather than 6 months of data”.

2. Define and Prioritise Value (Discovery)

Following stage one, we can look a bit more in depth by asking:

What KPIs or success criteria can be used to measure our progress?
What will the business see first? Which pressing challenge can we solve early in our journey?
Which workloads can be ported to the new platform to demonstrate the value?

For example, “solving for use case A will deliver requirements for 5 other use cases and allows us to track sales on our eCommerce store within 3 seconds, allowing us to manage inventory more effectively and saving £5m in working capital”.

3. Deliver Value (Delivery)

Finally, we should make sure to:

Build agile - focus on thin slices of end-to -end functionality. By focussing on use cases we ensure we architect for the full journey, not just the platform.
Build product teams - reduce handovers between architecture, development, and end-users. The product team succeeds when the use case is delivered, not when their part is completed.

Focus on building the end to end use case and only build the capabilities required to support, for example if the first use case doesn’t contain PII then some of the data governance and security capabilities may not be initially required, and you can introduce them only when they are needed for support.

Watchpoints - Is This You?

By following the above strategy, you should be able to achieve the results you need from your data migration. However, it is still worth bearing in mind the three main challenges that you can face - and seeing whether these apply to you or your migration strategy:

Technology-led migration
Reducing benefits by reducing lock-in
Data migration by migrating data

We can take a closer look at these three areas to address how businesses can adopt a more successful mindset when looking to migrate data.

a. Technology-led Migration

Too often, data migration projects are technology-led activities, in which teams want to experiment with the latest and greatest tools.

Given the rapidly changing technological landscape over the past 10 years, it has been easy for organisations to have fallen into the trap of migrating multiple times to the latest technologies. For example, they may have first moved from a Data Warehouse to Hadoop on-premises (in itself a time-consuming activity), then from Hadoop to a cloud provider, and then repeatedly between different offerings from that same provider - such as DB on IaaS, then moving to DB on PaaS, and then onto serverless DB options.

Whilst it is perfectly sensible to keep abreast of technologies, it is far more important to keep a finger on the pulse of how the business currently uses technology, what the pain points that originate from that are, and how technologies may solve these going forwards.

b. Reducing benefits by reducing lock-in

We’ve seen several organisations operate Hadoop-on-IaaS as their target state for a data migration activity. The two most commonly cited reasons for this is that either the Hadoop-as-a-Service offerings from Cloud providers lack the enterprise-grade security required, or that the organisation would want to be able to port its data platform with ease.

This can be problematic as it effectively decreases the value of the cloud provider to simply an outsourced infrastructure provider, and ignores the large number of innovations in the public cloud in this very space - such as serverless data pipelines with Dataflow, PB-scale Data Warehousing with BigQuery, and scalable data storage with no limits with Google Cloud Storage.

c. Data migration by migrating data

This might sound a little odd, but many organisations plan out their data migration activities by looking at the breadth of data, or at least range of data platforms, and iteratively working their way through this data backlog. This poses a number of challenges:

Data migration activities aren’t being accomplished with a view to specific business benefit gained - and we might not load all the datasets required to deliver a benefit
This delays validation with users, who may not have everything they need in the platform, delaying learning which in turn can substantially delay building a viable platform
Incomplete requirements - by not migrating use cases, many requirements may be missed out. For example, it’s not unusual to build a Hadoop data platform where users across the organisation can source data, but one of the biggest innovations in the data and analytics space has been Spark - the ability to push computing workloads to the cluster.

Remaining aware of these challenges while following the template of the checklist can help ensure your success in data migration.