How to use virtual memory (swap in linux file system ) in java web programming

Question

I work in java web application . In some section i use very huge tree variable that save and persist in memory (RAM) . Can i migrate this to virtual memory (swap) . note : huge tree consist name and email for all user that use in suggestion Ajax text-box .

Answer 1

There isn't a standard way in Linux of forcing a block of memory to swap, thus the JVM will not have a way of asking the OS to perform such a task.

The best you can do if you want this functionality, is to serialize the tree and write the raw data to a disk file then bring it back when you're ready for it.

But you probably don't want this, because writing to a disk is extremely slow when compared to physical memory i/o.

Case in point, let the OS worry about this. It's safe to assume it knows a better way of managing memory than you do.

Answer 2

Let the OS your user is using take care of this.

Answer 3

A running java image that partially pages out to swap is a dead java image. As soon as an eager enough GC kicks in, you get to page everything back in. The page in is bad enough. If you don't actually have enough RAM for the whole thing, you end up with a thrashing, unresponsive wreck of a server. Paged Java is Bad(tm).

If you have enough RAM for the whole thing, you don't need swap at all.

Stuff your list in to a database table, on disk, index it, limit your result sets, and make the proper queries against it. It'll be a net win, and the DB can cache the pages it likes best, so you don't have to think about it.

Or get more RAM.

Answer 4

Your OS automatically manages its own memory and pushes things out to the swapfile as needed.

If you have a lot of data, you might want to consider storing your data in a database instead of a huge in-memory tree. This would probably allow your application scale better, and it may also improve performance--it would certainly give you better performance than reading and writing the entire structure to disk whenever you need to look up or modify a record.

Edit: You don't necessarily have to set up a dedicated database machine. Given that you're currently trying to store all your data in memory right now, you can probably use an embeddable database like HSQLDB or SQLite, which have size limits of 16 GB and 2Tb, respectively.

Answer 5

I find it interesting that everyone is telling him that storing items on disk is terribly terribly ineficient, and at the same time recomending he use a database, probably a remote one that is going to store data on disk.. on another machine..

You are assuming that the system will be more efficient when blindly handling the swap file than it would be to have the swap file influenced by code that knows what the future holds. it is vastly more efficient to swap out memory that you know is not going to be used for a while than it is for the system to look at all of the items in memory and attempt to efficiently put some of it in that file.

Of course while being wrong you are all somewhat right.. a LOCAL database would be the most efficient way of storing data to a FILE (where it will be written and read). If you have no access to a local database then code one. A hashmap is designed to be stored in memory and an ordered indexed linked list is designed to be stored on disk. Trying to push directly from memory to disk without some consideration for the efficiency of both mediums is not effective.

Answer 6

How about this as a different take on the same problem: I'm creating a lot of PDFs server-side, I've got 10's of 1000's of clients who generally want to run reports at the same time of the month. Average PDF size might be 7-10Mb. With a finite heap available, 'swapping' the data out to a temporary file is a valid way to create the PDFs since I need to be able to set the content-length on the response prior to streaming the PDF data to the client.

Perhaps instead of just questioning the design some useful options might be handy. Personally I'm looking at using either one temp file per process or concurrent access of a single 'swap' file.

What would you suggest?