Database - GROUP BY? GROUP_CONCAT? Custom Application?

Question

Database type: Multiple ... I have it in CSV, MDB, ACCDB, and loaded into MySql. I can get to it anyway I need to.

I work for a Medical Facility and have been handed a database that has millions of 'line items' in it. Each line item contains the patients Chart Number, the Date of Service, the Charge Code, Payment Amount, and Total Charges for that charge code. I have a snippet of the data below:

+-----+-------------+---------+--------------+----------+
|Chart| ServiceDate |   Code  | Payment      |  Charges |
+-----+-------------+---------+--------------+----------+
| 2241| 1/27/2014   | 12002   | 411.00       |  51.24   |
| 2241| 1/27/2014   | 17311   | 1491.00      | 616.10   |
| 2241| 1/27/2014   | 17312   | 2108.00      | 722.98   |
| 2241| 1/7/2014    | 17311   | (1491.00)    |   0.00   |
| 2241| 1/7/2014    | 17312   | (1054.00)    |   0.00   |
| 2241| 2/17/2014   | 17311   | 1491.00      | 616.10   |
| 2241| 2/17/2014   | 17312   | 1054.00      | 361.49   |
| 2241| 3/20/2014   | 17311   | (1491.00)    |   0.00   |
+-----+-------------+---------+--------------+----------+

These are all for the same Patient, which I can tell due to the 'Chart' column being the same. In this example, the patient has 3 dates of service (1/27/2014, 1/7/2014, 2/17/2014, and 3/20/2014).

Now this is where things get complicated:

I need to run a query/script/program/anything that basically combines those line items and turns them into appointments.

This is how they would be grouped together (shown just for reference):

+-----+-------------+---------+--------------+----------+
|Chart| ServiceDate |   Code  | Payment      |  Charges |
+-----+-------------+---------+--------------+----------+
| 2241| 1/27/2014   |   12002 |  411.00      |  51.24   |
|     |             |   17311 | 1491.00      | 616.10   |
|     |             |   17312 | 2108.00      | 722.98   |
+-----+-------------+---------+--------------+----------+


+-----+-------------+---------+--------------+----------+
| 2241| 1/7/2014    |   17311 | (1,491.00)   | 0.00     |
|     |             |   17312 | (1054.00)    | 0.00     |
+-----+-------------+---------+--------------+----------+


+-----+-------------+---------+--------------+----------+
| 2241| 2/17/2014   |   17311 | 1491.00      |  616.10  |
|     |             |   17312 | 1054.00      |  361.49  |
+-----+-------------+---------+--------------+----------+


+-----+-------------+---------+--------------+----------+
| 2241| 3/20/2014   |   17311 | (1491.00)    |  0.00    |
+-----+-------------+---------+--------------+----------+

Now having blank cells in a database is just bad practice (IMHO), so I'd love to be able to go a step further and do it like this:

The final result would be more of a line-by-line setup, each line containing everything for the appointment.

Any column where the Chart and ServiceDate are the same, combine the Code,Payment, and Charges column (delim with a comma) as follows:

+-----+-------------+-------------------+------------------------+---------------------+
|Chart| ServiceDate |   Codes           |       Charges          |        Payments     |
+-----+-------------+-------------------+------------------------+---------------------+
| 2241| 1/27/2014   | 12002,17311,17312 | 411.00,1491.00,2108.00 | 51.24,616.10,722.98 |
+-----+-------------+-------------------+------------------------+---------------------+
| 2241| 1/7/2014    | 17311,17312       | (1491.00),(1054.00)    | 0.00,0.00           |
+-----+-------------+-------------------+------------------------+---------------------+
| 2241| 2/17/2014   | 17311,17312       | 1491.00,1054.00        | 616.10,361.49       |
+-----+-------------+-------------------+------------------------+---------------------+
| 2241| 3/20/2014   | 17311             | 1491.00                | 0.00                |
+-----+-------------+-------------------+------------------------+---------------------+

However, I can not think of any way to go about that in SQL or Excel, etc. The only thing that I can think of being able to do is to open up VS.Net and whip up an application that would loop through the entire DB and look for the columns where the Chart and ServiceDate are the same, and try to go from there. However, the logic behind how I would need to go about that is what is throwing me for a loop.

Spoken out in worded form ("Any column where the Chart and ServiceDate are the same, combine the Code,Payment, and Charges column") it seems so easy...but in practice I am at a loss as to where to start.

Does anyone have any ideas as to what direction I should take? Am I making this harder than it actually is?

I have an odd feeling this is going to be one of those: "OMG, I should have realized that" type moments when a response is posted, and I am going to apologize for that ahead of time :)

Answer 1

I will answer with a MySQL-based solution.

First of all, assumptions. Your Chart identifies the patient. Your ServiceDate defines the encounter, or appointment. This assumes a patient has at most one encounter per day. That's probably OK, but may not be in certain clinical situations. You haven't shown us how to disambiguate that, so we'll just go with it.

Millions of rows seem like a lot, but don't worry. That's not a cripplingly large table.

Your ServiceDate items should be in DATE format ('2014-03-20') rather than the text string format ('3/20/2013')you are using. This allows efficient arithmetic, ordering, and grouping. You need to convert them. You can do it in the query, or you can build a better-formatted table.

All that being said, here's the detail report (http://sqlfiddle.com/#!2/4a018e/2/0) you mentioned, with the column values duplicated.

SELECT Chart, 
       ServiceDate, 
       Code, 
       SUM(Payment) AS Payments,
       SUM(Charges) AS Charges
  FROM charge
 GROUP BY Chart, ServiceDate, Code
 ORDER BY Chart, ServiceDate, Code

Your comma-separated report (http://sqlfiddle.com/#!2/4a018e/6/0) can be done like this.

SELECT Chart, 
       ServiceDate, 
       GROUP_CONCAT(Code ORDER BY Code SEPARATOR ', ') AS Codes, 
       GROUP_CONCAT(Payment ORDER BY Code SEPARATOR ', ') As Payments,
       GROUP_CONCAT(Charges ORDER BY Code SEPARATOR ', ') As Charges
  FROM charge
 GROUP BY Chart, ServiceDate
 ORDER BY Chart, ServiceDate

But, to many eyes, this isn't a very easy-to-read report. You might try this report (http://sqlfiddle.com/#!2/4a018e/7/0) instead, it sums the payments and charges, and details the codes.

SELECT Chart, 
       ServiceDate, 
       GROUP_CONCAT(Code ORDER BY Code SEPARATOR ', ') AS Codes, 
       SUM(Payment) As Payments,
       SUM(Charges) As Charges
  FROM charge
 GROUP BY Chart, ServiceDate
 ORDER BY Chart, ServiceDate

With SUM(), GROUP_CONCAT(), and GROUP BY in your toolbox you should be able to work out the appropriate formats.

Answer 2

You could try using the GROUP_CONCAT operator which allows to group columns values with a GROUP BY clause.

The corresponding (untested) query could look like that

SELECT Chart, ServiceDate, GROUP_CONCAT(codes), 
       GROUP_CONCAT(payment), GROUP_CONCAT(charges)
FROM TABLE
GROUP BY Chart, ServiceDate

It will join together codes, payment, and charges values (using a comma separator) for each pair of Chart and ServiceDate

For more infos you could refer to the mysql manual : http://dev.mysql.com/doc/refman/5.0/fr/group-by-functions.html