Storing random access data file containing information such as images, sounds and other data

Question

Sorry for the broad question, but I hope someone can give me some advice on doing something like this and give me some ideas because this is seriously a huge roadblock for me right now.

So let me give you some context. I am making a game engine with a game editor. The game editor let's users build their game using the tools and map editor included in the software. Their project file is a folder containing different sub folders for certain elements in the game. For example, Maps are placed in a "maps" sub folder and each map is represented by a Json file. The tiles point to the sprite indexes and which tileset, and the tilesets are also Json files in their own "tilesets" sub folder that contain the image data in Base64 plus some more information related to the tileset.

The project file is then built into the single game file that should be distributed to the other users wanting to play the game. The game file is not a standalone executable, it's just a file that is given to the game engine executable which then interprets it.

My issue is building the single game file and how to do it properly. Suppose that there are many resources to load such as images, sound effects and music for the game, I certainly cannot parse the game file which could be, again, Json or raw binary data from top to bottom and load all of the resources in memory because that would take way too much RAM for a full game.

So my question is how should I approach this problem? How should I store the resources and load them in memory when they are needed. Remember that the whole game project has to be compiled into this one single file that can be easily shared with other users.

I am using Java and Gson for Json parsing just in case you were wondering. If the question or problem is not clear, please let me know. Thanks!

Answer 1

I would distribute your content as a SQLite file. I've done something similar in a mobile application with the same exact use case (people creating game files and distributing them to the players) and it works pretty well.

First of all BASE64 is ASCII encoded after all, so you can use the VARCHAR type to store up to 2GB, which I think in your scenario is definitely enough. You can also use an indexed column to access data in an efficient way.

The best pick according the info you gave us IMHO is to create one table for each type, using a varchar primary key to access data using the key value through a simple select returning just that row. SQLite is incredibly efficient in these kind of scenarios.

Another point that you can take into account is that, being binary, SQLite gives you a decent size reduction storing data in a binary file instead of a text file. This SO question explains a bit more the topic.

Last but not least SQLite is very lightweight and, either natively or via a wrapper, it's avalaible for a huge number of platforms (embedded stuff included).