If you completed part 1 on this guide and created the Managed Airflow Environment on AWS, you are ready for the next level - Part 2 - Enabling EKS scheduling In Airflow we often use KubernetesPodOperator DAG class. Airflow contacts k8s API-server and asks it to spawn a pod to...

Apache Airflow by Airbnb has been around for a while, and though newer scheduling engines are available nowadays, it is still popular among data engineers and being used in many companies such as AT&T, Robinhood and Slack. Airflow and its stateful components are not cloud native out of the box,...

Ever wondered whether its possible to mount S3 as storage for pods? Well it is, thanks to datashim.io CSI driver - A Kubernetes Framework to provide easy access to S3 and NFS Datasets within pods. Who needs this feature and who doesn’t Since S3 doesn’t allow file editing and the...

I was really proud to take part in HashiTalks Israel this year The EKS + GitOps module is on Github and ready for use: terraform-eks-with-gitops Link to the demo resources: terraform-eks-demo Visit my presentation - Moving from EKSCTL to Terraform Feel free to contact on Twitter or LinkedIn

In this post I will walk you through the steps required to define a manual RDS Mysql 5.8 replication across 2 AWS regions, in my case - Ireland (eu-west-1) and Frankfurt (eu-central-1) Pre-steps VPC peering: make sure the source region and destination region VPCs are connected via AWS VPC Peering...

So you are running your PHP app locally on a docker container, and you also got your code mounted to your working directory, so you can edit your code live. But how can you debug it? In this post I will cover the steps for installing & configuring xdebug inside...

A Container Network Interface, or CNI, is an essential part of a functioning kubernetes cluster. A popular one is Weave. Weave exports metrics which you can stream right to your monitoring system, so in this post I will cover the steps I took to monitor my weave CNI DaemonSet using...