Cover image by Michael S Adler - Own work, CC BY-SA 4.0

Data Engineering is a new career field that will create thousands of jobs in the near future. But how did we get here? The actual work of dealing with data has been around for a long time (think Database Administrators or DBAs). Though as a unique discipline, data engineering is relatively new. Google Trends shows the term starting to pick up steam in 2016, and recently it has hit near peak popularity.

Today, data is available in surplus due to massive digitalization. Hence, there is a rapid growth in the need for data engineering and the related data infrastructure. As a result, people and tooling are required to perform the job.

Isn't this advanced AI Stuff?

When you see projects like GPL-3 and other massive algorithmic bodies of work to create self-intelligent agents, it can feel like only people with advanced degrees should even be touching "data."

The reality for many companies and business applications is that they only need basic machine learning, not things like advanced neural networks. Basic machine learning skills can be picked up by developers and engineers, and we are already seeing this change begin to take place with the rise of “ML Engineering” roles — where people are conversant in designing ML algorithms, as well as training them on real data and deploying them in production.

Data is coming into its own

The most advanced companies are proactively creating dedicated resources for individual teams. This takes the term “data-driven” to another level. Instead of using data to perform existing tasks with more velocity and impact, teams at companies like Mattermost are designing and rebuilding initiatives, tactics, systems and processes in partnership with data engineering.

Instead of asking “how can we use data to make this better,” teams are partnering with data engineering to ask, “how can our data and data systems shape the way we think about solving this problem.”

How to get started

There are very limited barriers to entry for Data Engineering once you have a basic understanding of software tools. Most resources are available free with some training material freely available. See my earlier article on resources for getting started as a data engineer.

Going Further

If you want to dive deeper and get some hands-on exprience, our upcoming webinar compares two dominant data modeling tools and how you can make them work for your team.