Added Traefik and Airflow to the family of services behind my personal websites.
Traefik – an amazing modern reverse proxy that integrates extremely well with docker containers, saving me a lot of troubles in manual configurations (looking at you nginx).
Traefik makes it trivially simple to redirect internet traffic to multiple Dash docker containers, by just adding tags to docker compose services.
Despite the flexibility offered by Traefik, I feel more comfortable using nginx as the first layer reverse proxy as it has worked very well for me for a long time.
Airflow – an industry standard tool to schedule workflows. I finally have a proper tool to run and schedule long-running tasks, without having to resort to manual executions.
Setting up DAGs to run on Airflow is relatively easy. But what I did not expect is the complexity in setting up Airflow infrastructure. In essence Airflow consists of not just a single service, but multiple services that cross talk with one another.
Configuring them took some time, but official Airflow docker image has greatly simplify this process. That being said, standing up Airflow almost doubled my shoestring cloud budget.
Now, time to write and get some DAGs running!