Sign in

To finish off the system we needed to orchestrate the process of extracting, transforming loading and visualising the data with the added function of emailing the reports as well.

There’s quite a lot to airflow and I highly recommend you take a look at the following udemy course https://www.udemy.com/course/apache-airflow-course/

I’ll post a link to my code here https://github.com/AlexanderA28/sample_report_system/blob/main/big_query_data_load.py

It’s a very basic example of using airflow however I found it to be a great experience with the amount of customisation you can do with it.

I enjoyed doing this project because I gave me the opportunity to use new technologies and deliver a relatively small project end to end.

This article follows on from my previous post https://alexanderamlani24.medium.com/transforming-a-pre-formatted-excel-report-into-a-tabular-format-without-macros-reporting-system-a78e1a0751a8

So once the star schema is loaded into our database(I’m using google big query — in the following post I will go over how I loaded that in the airflow section)

We will generate the assets /charts for the reports — I’ll post a link to the following github where the full code can be seen.

Whilst generating the charts I learnt about the library shutil and also that I could store plotly plots in a dictionary which helped me generate the plots in a loop and write and name them…

This article follows on from my previous article https://alexanderamlani24.medium.com/extracting-and-writing-email-attachments-to-xlsx-using-the-ms-graph-api-reporting-system-part-1-82819256b630

In the past when dealing with pre-formatted reports I’ve either had to ask for direct access to the reporting database and when thats not been possible i’ve had to rely on vba/macros or manual intervention to cleanse the report into a format that i could then work with. So I thought this would be a good challenge for my new python skills.

This is the structure of the data that I extracted in the last blog post. …

I was receiving data from a data supplier on a weekly basis, there was no history and it wasn’t in a format that you could easily bring into a BI tool.

Because I wanted to experiment with open source technologies I thought I’d build a mini data mart from this data to capture the history and automate the data cleansing and create some reports from it.

I’m new to open source after a few years as a Power BI Consultant, so if anyone has any improvements for my methodology/code let me know, equally if my blog has helped you in…

Alexander

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store