how do you automate multiple reports requirements that are complex?

Question

I was tasked to automate reports used by multiple teams. Each team have their own procedure on how to generate the report that they need to deliver.

I can automate the reports but it takes so much time since I need to study each of their procedures and then test them.

The issue occurs when a team needs to update a procedure. If I will perform that adjustment, then it might take a week to deploy that to the live server which is not acceptable to the customer.

What I am thinking was to create interfaces for the users so they can configure their own reports, but the amount of complexity that they need to consider is too much. It would be nice if a team's requirements will just be to filter field1 with this value and field2 with this but no. They do all sorts of stuff like merging values, applying conditions, etc. It's multi step every team and not all teams have the same steps.

Closest tool that I can use is django's report builder. But it can only satisfy the requirements up to a certain point.

Answer 1

Programmatically transforming Excel sheets is not the way to go. I would review the whole procedure completely and use a database as data source for the reports.

Either let the users enter the source data directly into the database through an appropriate application or import the Excel data into the database (in an automated process).

Once that data is in the database, you can put the data into the right shape using database queries. This is much easier than manipulating cells in an Excel sheet programmatically. Based on these queries you can create reports using a (commercial) report generator. Modern report generators can create reports in various formats including a screen preview, PDF, HTML, RTF and even Excel.

Go away from this Excel bricolage.

A major disadvantage of Excel is that the data has no predefined structure. Users can enter the data in any way they want from highly organized to completely chaotic. This makes it extremely difficult to process it programmatically. A database on the other hand has predefined tables with predefines columns, predefined column types and predefined relations between the tables, which makes it much easier to implement automatisms, because you have a solid base to work on. While Excel sheets can be spread among different file systems and folders and can have random names, a database is usually located at a central and unique place and accessible through a predefined connection string.

Excel sheets may be very sophisticated, but are not scalable. I am developing software professionally. Many of my database projects (including applications as front-end) started with Excel sheets I got from customers. They were very skilled in creating complex sheets. But they reached a point where the sheet became too complex and could hardly evolve any more. The major problem is that complex data sets almost always contain 1-to-n relations and these can only be reproduced very badly in a spreadsheet. A spreadsheet is basically suitable for flat data only. Also, presentation and structure are the same in Excel. But often you need different evaluations and presentations of the same data. Here is where databases (back-end) with an appropriate application (front-end) and report generators come into play.

Answer 2

Depending on how diverse your teams are and what the guts of your objective is, an approach to consider is:

1. Providing one or two common reporting solutions to the projects and let projects that have more specific or divergent requirements create their own solutions.
2. Implementing the one or two common reporting solutions as microservices, so that they stand independently from a lot of project specific elements - possibly even independent from the project data sources. For example an API that expects as parameters the report data and instructions (output format, template, destination etc) means each project that calls the API can obtain the data they require however they need to and then transform it for the API call.

Hopefully that gives you another way of thinking about the problem.