Years ago, prior to the advent of Agile Development, a friend of mine worked as a release engineer. His job was to ensure a seamless build and release process for the software development team. He designed and developed builds, scripts, installation procedures and managed the version control and issue tracking systems. He played a mean mandolin at company parties too.
The role of release engineer was (and still is) critical to completing a successful software release and deployment, but as these things go, my friend was valued less than the software developers who worked beside him. The thinking went something like this — developers could make or break schedules and that directly contributed to the bottom line. Release engineers, on the other hand, were never noticed, unless something went wrong. As you might guess, in those days the job of release engineer was compensated less generously than development engineer. Often, the best people vied for positions in development where compensation was better.
Rising Fortunes
Today, the fortunes of release engineers have risen sharply. In companies that are implementing DevOps there is no more important person than the release engineer. The job title has been renamed DevOps engineer and it is one of the most highly compensated positions in the field of software engineering. According to salary surveys, experienced DevOps engineers make six figure salaries. DevOps specialists are so hard to find that firms are hiring people without college degrees, if they have the right experience.
Whereas a release engineer used to work off in a corner tying up loose ends, the DevOps engineer is a high-visibility role coordinating the development, test, IT and operations functions. If a DevOps engineer is successful, the wall between development and operations melts away and the dev team becomes more agile, efficient and responsive to the market. This has a huge impact on the organization’s culture and ability to innovate. With so much at stake, it makes sense to get the best person possible to fulfill the DevOps engineer role, and compensate them accordingly. When DevOps came along, the release engineer went from fulfilling a secondary supporting role to occupying the most sought after position in the department. Many release engineers have successfully rebranded themselves as DevOps engineers and significantly upgraded their careers.
DataOps for Data Analytics
A similar change, called DataOps, is transforming the roles on the data analytics team. DataOps is a better way to develop and deliver analytics. It applies Agile development, DevOps and lean manufacturing principles to data analytics producing a transformation in data-driven decision making.
Data engineers, data analysts, data scientists – these are all important roles, but they will be valued even more under DataOps. Too often, data analytics professionals are trapped into relying upon non-scalable methods: heroism, hope or caution. DataOps offers a way out of this no-win situation.
The capabilities unlocked by DataOps impacts everyone that uses data analytics — all the way to the top levels of the organization. DataOps breaks down the barriers between data analytics and operations. It makes data more easily accessible to users by redesigning the data analytics pipeline to be more flexible and responsive. It will completely change what people think of as possible in data analytics.
In many organizations, the DataOps engineer will be a separate role. In others, it will be a shared function. In any case, the opportunity to have a high-visibility impact on the organization will make DataOps engineering one of the most desirable and highly compensated functions. Like the release engineer whose career was transformed by DevOps, DataOps will boost the fortunes of data analytics professionals. DataOps will offer select members of the analytics team a chance to reposition their roles in a way that significantly advances their career. If you are looking for an opportunity for growth as a DBA, ETL Engineer, BI Analyst, or another role look into DataOps as the next step.
And watch out Data Scientist, the real sexiest job of the 21st century is DataOps Engineer.
Here’s a job posting for a DataOps Implementation Engineer:
DataOps Engineer
The DataOps Engineer will plan and execute the implementation of DataOps projects. This position requires top technical skills, business communication skills, excellent attention to detail, follow-up, and the ability to self-manage.
Responsibilities
- Plan and implement the use of DataOps software with Proof of Concept projects through ongoing production operation.
- Some projects will be SQL focused. You will gather requirements, work with raw data, design a schema, do data transformation, write automated tests, and manage deployment and operations.
- Other projects will be more integration focused. You will orchestrate the customer’s existing tools and analytic assets via Docker, APIs, or CLIs. You will use cloud (e.g. AWS, GCP, Azure) facilities to spin up environments.
- In both cases, you will become a master at using DataOps software to orchestrate, test and deploy Recipes.
- Engage in consistent, proactive communication to positively impact customer/user loyalty. Partner effectively with internal teams to drive growth and address customer/user concerns efficiently and decisively.
Qualifications and Skills: Technical
- Experience on implementation projects
- Experience with SQL and Python (or equivalent)
- Continuous integration frameworks and unit testing
- Cloud technologies like AWS and GCP and others
- Experience delivering products in data management, analytics, data pipelines or data science is required
- Experience with Docker is a plus
About the Author
Christopher Bergh is a Founder and Head Chef at DataKitchen where, among other activities, he is leading DataKitchen’s Agile Data initiative. Chris has more than 25 years of research, engineering, analytics, and executive management experience. Chris has an M.S. from Columbia University and a B.S. from the University of Wisconsin-Madison. He is an avid cyclist, hiker, reader, and father of two teenagers.
Sign up for the free insideAI News newsletter.
Speak Your Mind