10 Lessons Learned In 10 Years Of Data [1/2]
It’s the end of 2022, and a common tradition in the data community is to predict trends in 2023. But what do you need for predictions? Data. And looking solely at 2022 will not help us too much to give accurate predictions. So let’s go back to 2012.
I’ll highlight my lessons learned, and you draw your own prediction for 2023. Don’t worry; some will be obvious. And, of course, there will be memes.
2012 ⏱️ Meet Bob, the Big data engineer
Bob is happy. His company just invested in an on-premise Hadoop cluster. No more proprietary BI tools. They will be dead in a few years, anyway (right!?). Bob is happy to care about distributed systems rather than business value.
A few months, system engineers, and thousand of $$$ later, the cluster is finally ready.
Bob is thinking: “Oh, it would be nice to have a service that does that for us, but what will we do then? It will steal our jobs!”
✔️ Lesson #1: Cloud didn’t take our job
Technology doesn’t replace people; it rather changes the way we work. So if you are scared about all these ChatGPT highlights, look at the past and think twice.
You definitely will need to adapt as many companies did for the cloud, but you will still get a job to do.
2013 ⏱️ Another day in Bob’s Big Data Engineer life
Today Bob has a big batch job to run that will probably take all resources from the cluster for a while. He kindly warns his teammates. They are ready for a long coffee break.
✔️ Lesson #2: Unlimited cloud resources can be painful
