Storing large amounts of text (strings) for later usage

Question

I am writing a windows forms application in VB.NET that makes http requests and gets JSON responses.

I would like to store all these JSON responses in a file to read them at some point in the future (line by line) as if I am making the requests from the file instead of the website/API.

I would like the read operations to be as fast as possible, whereas the writing can be relatively slow. So the frequency of the http request can be as small as 1/second to 1/minute, whereas i would like to read the data very fast (I guess 500ms or less?)

Right now, I am simply writing each JSON string as a new line of text in txt file. However the strings are long and the files are getting very big very quickly.

Is there a better/alternative solution to storing these JSONs/strings in a compact form for fast reading later on?

Answer 1

.NET Framework has a built-in support that would allow to do this.

Basically, you will have to take you string and convert it to a byte array. Then you will have to deflate (zip) your byte array and convert it to a base64 string.

Public Function ZipString(stringToZip As String) As String
    Dim returnValue As String = ""

    Using output As New MemoryStream()
        Using gzip As New DeflateStream(output, CompressionMode.Compress)
            Using writer As New StreamWriter(gzip, Encoding.[Default])
                writer.Write(stringToZip)
            End Using
        End Using

        returnValue = Convert.ToBase64String(output.ToArray())

    End Using



    Return returnValue

End Function

To get your original string, you do the opposite:

Public Function UnZipString(stringToUnZip As String) As String
    Dim returnValue As String = ""


    Using inputStream As New MemoryStream(Convert.FromBase64String(stringToUnZip))
        Using gzip As New DeflateStream(inputStream, CompressionMode.Decompress)
            Using reader As New StreamReader(gzip, System.Text.Encoding.UTF8)
                returnValue = reader.ReadToEnd()
            End Using
        End Using
    End Using


    Return returnValue

End Function

You will then be able to store the JSON as a zipped string and unzip it when you'll need it.

Answer 2

In general, I would suggest giving a look at NoSQL databases. These are databases that use a non-relational (no SQL) model to improve performance and some of them (like MongoDB are oriented in storing/retrieving json objects directly.

Another option would be to store json documents in a search server like Solr which could store JSON objects directly. This scenario would be more suitable if you want to search a large amount of JSON objects as if they were documents. Using Solr gives you great flexibility in the way the data are stored, and gives you great searching functionality.

You could take a look at the advanced approaches (NoSQL, Solr) and think if the learning and transition curve of using a new technology would return the time investment regarding performance benefits. I would first try to use a relational database and only if my performance needs are so big then I would test a NoSQL approach.

Hope I helped!