Per minute JSON files parse and amalgamation

Question

I am building a system which uses JSON data to render server statistics into HTML tables. The back-end system produces one JSON file every minute and that JSON files contains information for around 10 different areas of the servers performance. So each JSON snapshot will contain the data for one row of each table.

The tables I want to produce will show the past 15 minutes worth of data, the tables will refresh every minute to include the newest minute data and discard the oldest minute data so we can always see the last 15 minutes worth of statistics.

I am new to JSON and the way it works and performs and the overall best practices in general so I would be very grateful to any suggestions of how I might do this. In terms of loading the data from these files I believe their are three possible methods I can use to do this.

1. Creating a central Javascript Object, lets call it currentDataSet, which will load all the data from the last 15 minutes worth of JSON files. i.e. A object that will hold data from 15 JSON files each containing 10 tables worth of data. Every minute currentDataSet would be updated to forget the oldest minute and include the data from the newest minute. From currentDataSet each table renderer function will look to currentDataSet and product the table.

2. Each table for themselves!, I also thought that instead of having a central object which contains all data that I could also write it so each table renderer function could fetch it's own data. That way each table could parse the 15 JSON files and maintain their own data. So 10 Javascript Object such as performanceByInstance, htmlStatus, databaseStatus, dynacacheStatistics etc...

3. Loading the JSON data into a database of some kind instead of relying on JSON parsing and Javascript Objects for storage.

4. Some method I had not even considered due to lack of experience. :-)

I am most concerned about the performance of loading 15 JSON files in to be rendered at once, in future I would like the system to be able to load in archived data from a past date/time and so will need to quickly render all this data into the tables when a date/time is selected.

I will also be aiming to use Dojo to achieve this as this is what I am using to build the UI dashboard for this tool.

Thanks in advance.

Answer 1

I would choose #1 as you really don't gain anything from #2, and #3 is less scalable, which may or may not matter to your situation.

However, for the initial loading, I would lump all the data-sets "together" (see below). IMO, This decreases usage of and reliance on the most likely point of failure.

Store each data-set as part of an array, because sorting of non-indexed data types like Object is messy and unintuitive. Array is a huge help when managing the "timeliness" of data-sets.

Answer 2

you could make a dojo "store" for each table, as they make managin and ordering data relatively easy as long as you have something in each object of the json array that can be used as an ID http://dojotoolkit.org/reference-guide/1.9/dojo/store.html

[{
    idLikeProp:1,
    //........
},
{
    idLikeProp:2,
    //........
},
//..........
]

using them like this could make it easy for other dojo components to plug to it like

dGrid

http://dojofoundation.org/packages/dgrid/

or

gridX

http://oria.github.io/gridx/

which could ease your life in the displaying of the table data(dGrid is more lightweight than gridX)