As enterprises fiercely compete for data engineers, a new global poll out today by Wakefield Research and Fivetran, a leading provider of automated data integration, shows that, on average, 44 percent of their time is wasted building and rebuilding data pipelines, which connect data lakes and warehouses with databases and applications. This significant outlay of human capital isn’t adding up: 71 percent of respondents say end users are making business decisions with old or error-prone data — with 66 percent saying their C-suite doesn’t know this is even happening. As a result, 85 percent of enterprises have made bad decisions that have cost them money.
Companies are paying large sums to reach these bad outcomes. Data and analytics leaders report a median of 12 data engineers earning an average of $98,400 a year each. Spending 44 percent of their time away from working on advanced models and analytics adds up to $520,000 a year.
“What we’re seeing from this study is that data and analytics leaders are really struggling to keep up,” said George Fraser, CEO of Fivetran. “It would be one thing if the processes companies used for manually building and managing pipelines were optimized, but 80 percent of those surveyed admit they have to rebuild data pipelines after deployment — from changing APIs, for example. For 39 percent, they say this happens often or all of the time.”
In addition to being error-prone, the process for deriving perceived value from the data also moves at a snail’s pace. Only 13 percent report being able to derive value from newly-collected data in minutes or hours. Instead for 76 percent of companies, it takes days or up to a week to prepare the data for revenue-impacting decisions, including 74 percent of companies with $500 million-plus in revenue.
In other findings:
- 69 percent of data and analytics leaders say that business outcomes at their company would be somewhat or significantly improved if their data team were able to contribute more to business decisions rather than manual pipeline management.
- 90 percent of those polled in the U.S. say hiring staff would be a major component if they could effectively scale up data management capacity, compared with 74 percent outside the U.S.
- Nearly all (97 percent) say business outcomes would be improved if their data team could spend more time devoted to the analytics behind data-driven business decisions.
Survey Methodology
The Fivetran Survey was conducted by Wakefield Research among 300 data and analytics leaders, VP and above in the U.S., UK, Germany and France, at companies with $100 million+ in annual revenue, a minimum of 100 employees, and who are familiar with the data strategy/data use at their organization, between September 27 and October 12, 2021, using an email invitation and an online survey.
Sign up for the free insideAI News newsletter.
Join us on Twitter: @InsideBigData1 – https://twitter.com/InsideBigData1
Speak Your Mind