Let’s evaluate AWS step functions, Google workflows, Prefect next to Airflow. — While Airflow has dominated the market in terms of usage and community size as a data orchestrator pipeline, it’s pretty old and wasn’t designed initially to meet some of the needs we have today. Airflow is still a great product, but the article's goal is to raise awareness on the…

Story of datacreators.club, a hub to discover 100+ data content creators — Technology is moving blazing fast. The open-source era increased the speed of new tools adoption, and when we seek knowledge, it mostly comes from tech individuals online. YouTube has seen tremendous growth, claiming 500+ hours of content uploaded every minute. Medium reported an increase of 106% in writers in 2020…

Let’s cover an API use case with Terraform HCL & Python — Today, most of the infrastructure code is done through Terraform. It’s been there for quite a while, has a strong community, and it’s multicloud. However, things start to be tricky when it comes to testing Terraform code. While Terraform uses its own language (HCL), its backend is written in Golang. …

Why Job Hopping Now Is Intentional, Not Impatient (And What You Need To Know) — “We are in the most heated tech hiring market of all time.” commented the Pragmatic Engineer based on incontestable fresh data points. We, Software Engineers, are in an exceptional situation that may be comparable to the dot com bubble. As things are moving fast, so are our career and job…

Understanding the modern table formats and their current state — Data Lakehouse is the next-gen architecture presented by Databricks paper in December 2020. Data Lake can be run with open formats like Parquet or ORC and leverage Cloud object storage but lacks rich management features from data warehouses, such as ACID transactions, data versioning, and schema enforcement. Today, we have…

Let's discuss the next contender for 2022 — Picking the right programming language for the right job can be challenging as the technology moves fast, and so many frameworks are popping up. Let's break down these past years and understand data engineers' current programming language ecosystem and the ideal candidate for 2022. Could Scala, Golang, or Rust be…

Learn how to manage dockerfiles and version through a working Python API project — In today's development world, containers are everywhere. It's a practical way to test things locally and be able to deploy it on any cloud service, as long as it supports a container image. It's easy however to fall into the hell of dockerfile(s) and versions. I'll provide in this guide…

Strategies to help you to land your first Data Engineer job — There is a ton of knowledge out there on how to be a 10x data engineer, but what's worth taking for you to land your first data engineer job? There are so many tooling and concepts you need to learn: it's scary 😱. In this article, I will give you…

Here it goes again, one of my favorite free data conferences! And the name has been again rebranded for the better. It was initially the "Spark Summit," then the “Spark+AI Summit,” and now the "Data+AI" Summit. Databricks now offers a couple of products, and it’s not anymore considered only as…