How many technical tasks are solved by senior (Big) Data engineers?

Question

big data

Good afternoon.

Colleagues who have been working on (Big) Data for a long time and have grown to the level of Senior (or higher), tell me how many engineering/programming/architecture tasks do you do?

I came to (Big) Data from Software Engineers a year ago because I started to enjoy data science tasks over API design (no offense to anyone). For reference: there is a good experience in Python/DevOps.

But now I'm starting to see that I need less and less technical topics to grow up the "ladder", and more SQL / Cloud services / BI and so on. For example, at one of the courses, a speaker who worked at Microsoft and Amazon for several years, as a Data engineer, recently threw off an article where they tell on their fingers how concurrency and parallel processing works, with the words “I never understood this.” I thought this was the base for the Data engineer.

Next, I finished reading the book “High Performance Spark: Best Practices for Scaling and Optimizing Apache Spark” and I really liked how Spark works “inside”, but writing typical tasks on it is not particularly enthusiastic.

In general, I noticed that in almost every tool that I used, I climb inside and see how it works. And this is sometimes much more interesting to me than using it.

On the one hand, I am interested in the technical side of processing (big) data, but I see more and more that most processing tasks can be solved knowing basic Python and using different SaaS Cloud solutions.

Maybe I'm not working on "those" projects.

Share your thoughts with a colleague.