Photo by Mick Haupt on Unsplash
Being a data engineer can be both challenging and rewarding. But it's not always easy to break into this part of the tech field.
Data engineering in itself is such a broad term filled with tools, buzzwords and ambiguous roles.
This can make it very difficult for developers and prospective graduate to get these roles as well as understand how they can create a career path towards said role.
Let's talk about the different challenges you will face trying to become a data engineer. While finishing this article off with some solutions and paths you can take in order to break into the data engineering field.
Challenges With Learning Data Engineering
One of the biggest challenges with this career trajectory is related to a lack of general opportunity and gaps in the process of making your way from one level of experience and mastery to another. There may not be an easy path for you to follow as you work your way up to the higher levels of this type of expertise.
It starts with a relative lack of relevant college and university programs that really prepare you for the ins and outs of data engineering, such as dealing with wild or unstructured data or higher-level database concepts.
Much of the available higher education in this realm has more to do with other aspects of data science or with software development as a specific practice. That's all well and good, but what about specialization in data engineering? Data engineering as a concept is a more elusive part of a given college program.
In fact, experts who understand the pathways to becoming a data engineer often see what they would call a "gap" between the education that's available and the skills and experience that are required on the job. That's something you'll have to deal with as you put together your career strategy for this type of role.
A Lack Of Junior Data Engineer Positions
Another related problem is that there are a finite number of junior data engineering positions available at today's companies, as well as challenges with building the skill sets and experience that take you from a junior to a senior position as you go.
For example, getting the requisite experience with ETL processes and data warehousing can be difficult.
Without access to large complex data sets as well as complex infrastructure, it can be difficult to really replicate the challenges you will face as a data engineer.
So without a wide range of junior positions, many employees who may want to become a data engineer, might not have an easy route. Now I do think I am seeing some companies trying to fix this problem but its a slow process.
Developing A Good Enough Data Engineering Project
So what about doing projects on your own and building a portfolio that any professional would be proud of?
There are some challenges with that too. If you're not in the field, in a practical position, it's likely that you're not seeing a lot of the specific challenges you would encounter with unruly data or workarounds to particular enterprise problems. Trying to re-create these in a vacuum is also tough, to put it mildly.
It's not to say that it's impossible. In fact I did put together a list of data engineering projects I thought looked really great. However, it can be challenging because data engineering also doesn't lend itself as much to applications compared to software engineering.
One More Challenge: Diversity in Skill Sets
In talking about why it's so hard to break into data engineering, we should also talk about the breadth of skills that are needed.
A good data engineer will have a basis in programming, not just with Python or Scala but in a diverse environment of technologies. Employers and headhunters talk about the "full-stack data scientist or engineer" as a holy grail in the industry, but they don't often talk about how that personal achievement gets unlocked.
Or if it ever truly gets unlocked.
In addition to programming, supposedly the successful data engineer should also have a good understanding of around DevOps. He or she should understand the setup of cloud services and APIs, know how to work with SQL as well as interact with NoSQL database, and how open source and vendor tools can work together. He or she should have some hands-on experience with data pipelines and data warehousing.
All of that is a tall order!
So how do you get it done?
Ways to Become a Data Engineer
Now we can talk about some of the solutions that may put you closer to your goals as a professional data engineer.
Becoming a Data Engineer: Analysts and Software Engineers
One of the best ways to work toward a career in data engineering is to start in a related position that's a bit more common or maybe more geared toward the entry-level end of the spectrum.
To put it another way, the analyst is a fairly standard role in IT. Like the associate in retail, being an analyst in IT can mean various things. You may have a role as a data analyst, business analyst or financial analyst. There really are a ton of roles with the term analyst that can require some similar skills. For example, you will likely need some SQL and you will also need to understand data warehouses. These are the beginning skills of a data engineer.
In this way you will likely be working close to a data engineering position and maybe even have the chance to occasionally build a data pipeline or two. Setting up a clear path for you to go from analyst to data engineer.
The basic idea is that, as you do these things, you build skills that are directly parallel to what you would do as a data engineer. As you do that, you also are hopefully networking toward having more prominence in your particular company. You're getting some of that proximity to the types of problem-solving we talked about above that will come in handy in the trenches.
There is a similar thing to be said for people wanting to go from software engineer to data engineer.
Working With Startups
Another route to a successful data engineering career is to get yourself into a team situation where you are one of a few people taking on a new and exciting project from scratch.
Startups are a great place to start a tech career.
The idea of the startup is that small, scrappy teams are competing with those established blue-chip companies and their armies of developers and engineers to come up with their own data solutions.
The reason I sometimes suggest working with startups is that you have a lot more general skill-building opportunities. You have more skin in the game, and you're closer to the heart of the machine. You tend to wear a lot of hats, but as you go, you develop more skills accordingly.
It's not always as lucrative as working for a FAANG, but it will likely be a great learning opportunity.
The Internship Route
There is a very different way to approach data engineering that you might think about.
If you don't want the uphill challenge of trying to design work from a startup perspective, you can try for an internship at one of those more established companies, where others before you have likely fine-tuned DevOps and cloud approaches and other elements of data engineering.
The pros and cons are evident. On one hand, you won't be blazing a trail. On the other hand, you have a defined structure to work with as you try to climb the ladder.
In addition, if you perform well, then you are likely to get a return offer and you won't have to stress about getting a job once you graduate.
Next Steps
The truth is for many people reading this article, you may not even know what role you want. I often have people asking me how they would know which role they would enjoy. The truth is, this is so variable depending not only on the person but also the company and team that you end up working on.
In some cases you may really enjoy working as a data engineer at a start-up and dislike at a different company. Some companies use lots of code to develop their pipelines, others use drag and drop code.
All in all, your next steps is to start your journey. Try building a data engineering project or get an internship at a company.
Good luck!
Read/Watch These Next
- Modernizing Your Data Architecture With The "Modern Data Stack"
- How I Went From Analyst To Data Engineer\ How to become a data engineer --- and know if it's right for you
- How To Start A Consulting Business As A Consultant\ Getting Your First Client
✉️ Subscribe to my Mailing List For Community Updates And Freebies
Connect with Me on Social Network
✅ YouTube:https://www.youtube.com/channel/SeattleDataGuy\
✅ Website: https://www.theseattledataguy.com/\
✅ LinkedIn: https://www.linkedin.com/company/18129251\
✅ Personal Linkedin: https://www.linkedin.com/in/benjaminrogojan/\
✅ FaceBook: https://www.facebook.com/SeattleDataGuy
Top comments (0)