To finish off the system we needed to orchestrate the process of extracting, transforming loading and visualising the data with the added function of emailing the reports as well.
There’s quite a lot to airflow and I highly recommend you take a look at the following udemy course https://www.udemy.com/course/apache-airflow-course/
I’ll post a link to my code here https://github.com/AlexanderA28/sample_report_system/blob/main/big_query_data_load.py
It’s a very basic example of using airflow however I found it to be a great experience with the amount of customisation you can do with it.
I enjoyed doing this project because I gave me the opportunity to use new technologies and deliver a relatively small project end to end.