To finish off the system we needed to orchestrate the process of extracting, transforming loading and visualising the data with the added function of emailing the reports as well.

There’s quite a lot to airflow and I highly recommend you take a look at the following udemy course https://www.udemy.com/course/apache-airflow-course/

I’ll post a link to my code here https://github.com/AlexanderA28/sample_report_system/blob/main/big_query_data_load.py

It’s a very basic example of using airflow however I found it to be a great experience with the amount of customisation you can do with it.

I enjoyed doing this project because I gave me the opportunity to use new technologies and deliver a relatively small project end to end.

This article follows on from my previous post https://alexanderamlani24.medium.com/transforming-a-pre-formatted-excel-report-into-a-tabular-format-without-macros-reporting-system-a78e1a0751a8

So once the star schema is loaded into our database(I’m using google big query — in the following post I will go over how I loaded that in the airflow section)

We will generate the assets /charts for the reports —…

This article follows on from my previous article https://alexanderamlani24.medium.com/extracting-and-writing-email-attachments-to-xlsx-using-the-ms-graph-api-reporting-system-part-1-82819256b630

In the past when dealing with pre-formatted reports I’ve either had to ask for direct access to the reporting database and when thats not been possible i’ve had to rely on vba/macros or manual intervention to cleanse the report into a…

Alexander

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store