Let’s evaluate AWS step functions, Google workflows, Prefect next to Airflow. — While Airflow has dominated the market in terms of usage and community size as a data orchestrator pipeline, it’s pretty old and wasn’t designed initially to meet some of the needs we have today. Airflow is still a great product, but the article's goal is to raise awareness on the…

Storage is not always cheap — We are in the data gold era. The global data sphere produced around ~2 zettabytes in 2010 and ~64 in 2020 (x32!). The growth of data volume is exponential. As individuals, we produce and consume an insane amount of data. As a company, we want to leverage these to make…

A pragmatic approach to data contracts — Recently, there has been a lot of noise around data contracts on social media. Some data practitioners shared opinions about pros and cons but mostly about what it is and its definition. While I think data contracts are a wild topic, I wanted to share my experience with pragmatic tips…

And you don't need to code to get started. — Today's data world uses many open source tools, from data engineering to data science and deep learning. This is a unique opportunity to grow your career in multiple ways. In a recent podcast of DataTalks.Club, Merve Noyan, shared how she went from baby steps on GitHub to Developer Advocate Engineer…

Story of datacreators.club, a hub to discover 100+ data content creators — Technology is moving blazing fast. The open-source era increased the speed of new tools adoption, and when we seek knowledge, it mostly comes from tech individuals online. YouTube has seen tremendous growth, claiming 500+ hours of content uploaded every minute. Medium reported an increase of 106% in writers in 2020…

Let’s cover an API use case with Terraform HCL & Python — Today, most of the infrastructure code is done through Terraform. It’s been there for quite a while, has a strong community, and it’s multicloud. However, things start to be tricky when it comes to testing Terraform code. While Terraform uses its own language (HCL), its backend is written in Golang. …

Why Job Hopping Now Is Intentional, Not Impatient (And What You Need To Know) — “We are in the most heated tech hiring market of all time.” commented the Pragmatic Engineer based on incontestable fresh data points. We, Software Engineers, are in an exceptional situation that may be comparable to the dot com bubble. As things are moving fast, so are our career and job…

Understanding the modern table formats and their current state — Data Lakehouse is the next-gen architecture presented by Databricks paper in December 2020. Data Lake can be run with open formats like Parquet or ORC and leverage Cloud object storage but lacks rich management features from data warehouses, such as ACID transactions, data versioning, and schema enforcement. Today, we have…

Let's discuss the next contender for 2022 — Picking the right programming language for the right job can be challenging as the technology moves fast, and so many frameworks are popping up. Let's break down these past years and understand data engineers' current programming language ecosystem and the ideal candidate for 2022. Could Scala, Golang, or Rust be…