To finish off the system we needed to orchestrate the process of extracting, transforming loading and visualising the data with the added function of emailing the reports as well.
There’s quite a lot to airflow and I highly recommend you take a look at the following udemy course https://www.udemy.com/course/apache-airflow-course/
I’ll post a link to my code here https://github.com/AlexanderA28/sample_report_system/blob/main/big_query_data_load.py
It’s a very basic example of using airflow however I found it to be a great experience with the amount of customisation you can do with it.
I enjoyed doing this project because I gave me the opportunity to use new technologies and deliver a relatively small project end to end.
This article follows on from my previous post https://alexanderamlani24.medium.com/transforming-a-pre-formatted-excel-report-into-a-tabular-format-without-macros-reporting-system-a78e1a0751a8
So once the star schema is loaded into our database(I’m using google big query — in the following post I will go over how I loaded that in the airflow section)
We will generate the assets /charts for the reports — I’ll post a link to the following github where the full code can be seen.
Whilst generating the charts I learnt about the library shutil and also that I could store plotly plots in a dictionary which helped me generate the plots in a loop and write and name them…
This article follows on from my previous article https://alexanderamlani24.medium.com/extracting-and-writing-email-attachments-to-xlsx-using-the-ms-graph-api-reporting-system-part-1-82819256b630
In the past when dealing with pre-formatted reports I’ve either had to ask for direct access to the reporting database and when thats not been possible i’ve had to rely on vba/macros or manual intervention to cleanse the report into a format that i could then work with. So I thought this would be a good challenge for my new python skills.
This is the structure of the data that I extracted in the last blog post. …
I was receiving data from a data supplier on a weekly basis, there was no history and it wasn’t in a format that you could easily bring into a BI tool.
Because I wanted to experiment with open source technologies I thought I’d build a mini data mart from this data to capture the history and automate the data cleansing and create some reports from it.
I’m new to open source after a few years as a Power BI Consultant, so if anyone has any improvements for my methodology/code let me know, equally if my blog has helped you in…
