In this special guest feature, Mike Waas, CEO of Datometry, takes a look at mistakes companies should avoid when they are moving away from Teradata. Datometry is a SaaS database virtualization platform enabling existing applications to run natively on modern cloud data management systems without being rewritten. Mike has held senior engineering positions at Microsoft, Amazon, EMC, and Pivotal and is the architect of Greenplum’s ORCA query optimizer. He has authored 40+ peer-reviewed scientific publications in various areas of database research and holds over 50 patents.
There is a rapidly growing movement of companies moving away from Teradata. They’re tired of being told how IT only runs the business but doesn’t help grow it, and they cannot afford to miss out on an opportunity to generate new revenue by leveraging the vast array of data processing facilities in the public cloud.
Yet, replatforming instills fear in even the most hardened IT leader. For a perspective, consider this quote a major enterprise recently received for a rewrite-based migration: $26m for a project that was supposed to take 36 months. Take into account how practically every migration runs late and goes way over budget. Now, you’re looking easily at $40m to $50m, probably over 5 years.
IT leaders who assume they can rewrite their way out of Teradata in a couple of months will face a rude awakening, along with a potentially career-ending misstep. What are some other common mistakes companies need to avoid when replatforming from Teradata?
Assuming that existing 3rd party applications can simply be repointed
Most BI/ETL vendors support cloud data warehouses (CDWs) in the latest versions of their software (emphasis is on the latest version). Companies that have a well-established application ecosystem are most likely running older versions of these systems – so they cannot just repoint them.
To make them work with new destination data warehouses, companies must first upgrade. This effort depends on the size and structure of the existing system, but upgrading is an operation that typically spans multiple quarters and quickly costs millions of dollars.
However, even an expensive upgrade doesn’t do the job just yet. All commercial BI/ETL tools support the injecting of customized SQL into reports and data pipelines. As a result, there is Teradata SQL embedded practically in all daily processes. In a conventional rewrite, all that logic needs to be extracted, rewritten, re-inserted, re-tested, and re-deployed.
While it is tempting to believe the ecosystem is portable, in practice it requires significant effort which is highly labor-intensive.
Rewriting ETL when going to the cloud
This may seem absurd considering how much effort the reimplementation of ETL requires. However, it comes often on the heels of the above (see “Assuming that existing 3rd party applications can simply be repointed”). If repointing an existing system is not an option because of all the rewrites needed, one might just bite the bullet and rewrite the data pipelines entirely. Or so the logic goes. But that is grossly flawed.
This is a great example of why rewrite-based migrations are failing regularly. Companies that go from adjusting ETL to a complete redesign will undo years of investment and most likely end up redesigning and reinventing the exact same business logic of the existing system anyway.
While rewriting and evolving ETL can be an important design project for a company, there’s a time and place for everything. When an enterprise needs to replatform to the cloud as fast as possible, it is not the time nor the place for such experimental designs.
Expecting to migrate from Teradata to a modern CDW in a couple of months
The truth is that companies can replatform from Teradata to a CDW only if they do not rewrite. Provisioning, rerouting applications, and testing alone will take several months. Consider this for comparison: Just upgrading Teradata itself takes months to plan and execute. It does not require the rewrite of a single application, adjusting any embedded SQL or replacing loaders and utilities.
If migrating from Teradata to a modern CDW requires rewriting, all bets are off. If companies must rewrite, “a couple of months” can turn into years. Testing alone is a big pain: companies must rewrite all tests, revalidate and re-deploy them. Validation is complex because of operating two distinctly different test environments now. Then come the user acceptance tests which are even more burdensome because of the differences of the systems.
In short, a couple of months is the bare minimum for any operation. If companies must rewrite applications, they’re quickly looking at a large multiple of that baseline.
Assuming that rewrites lead to better performance
All modern CDWs pride themselves on being general purpose data warehouses. Powerful query optimizers built into these systems ensure the best possible execution of any query, no matter the query. They are capable of processing just about anything thrown at them.
Yet, there’s still that persistent misconception that Teradata queries need to be rewritten in certain, unspecified ways. Supposedly that makes them perform better on CDWs. This was true a few years ago when the cloud was still in its infancy, but luckily, those days are long gone.
Not only do companies not need these customizations, but they should really stay away from them. Any such optimization, no matter how well intended, adds complexity and quickly turns into technical debt.
It is in the cloud vendors’ core interest to eliminate the need for customization or special formulation of queries. What might seem like a clever optimization today is nothing but an obstacle tomorrow. In short, “optimizations” require significant effort, yet are almost always a waste of time.
Solving only 80% of the problem
Most migrations fail because IT leaders underestimate the effort. Database migrations are an 80-20 problem – the first 80% of the journey requires only 20% of the effort. It’s easy to make good progress with a rewrite and almost all the migration appears surprisingly easy. However, it’s the remaining 20% that bring out all the problems, take years and years and kill migration projects.
A slew of migration tools has emerged recently. They attempt to convert Teradata SQL code to equivalent SQL on the CDW. Now consider that most of these automatic code conversions claim a 50-70% success rate and we can quickly see why they are not a solution to this problem. They basically solve most of the easy issues that weren’t the problem to begin with. In the best case, automatic code converters reduce the overall effort by 10-15%. But in the greater scheme of things that’s just noise.
Replatforming from Teradata is not as simple as many enterprises are led to believe. Contrary to what IT leaders are told – replatforming requires much more than a rewrite. It’s important for enterprises to be aware of these common misconceptions around replatforming, to avoid making costly mistakes and to ensure success when migrating from Teradata.
Sign up for the free insideAI News newsletter.
Join us on Twitter: @InsideBigData1 – https://twitter.com/InsideBigData1
Mistake #1 – Believing you need to replatform from Teradata “to generate new revenue by leveraging the vast array of data processing facilities in the public cloud”. I would refer to Gartner on this topic and read up on the momentum behind Teradata’s multi-cloud strategy
There’s simply no benefit to spending three years migrating an application to get you to the same point where you started.
None of the CDWs have proven themselves to work with anything more than simple single domain, departmental workloads. Very few (if any) large enterprises have been successful in these migrations.
The CDW vendors are not even making any money, Snowflake for example can’t turn a profit until all those 3 year contracts it signed can be resigned at higher rates – at the point the expensive 3 year migration is done – and the business has been left behind because they’ve focused all their resources rebuilding something they already had!!!