How to extend the allocated memory of a bytebuffer

Question

I have a bytebuffer and I put Ints,Chars etc.. Because I do not know how much space I need I would like to dynamically growth the bytebuffer. How can this be done ?

Example : - I have a Bytebuffer of 2 bytes - I add a character to the bytebuffer (bytebuffer is full now ) - I like to add a Integer to the bytebuffer by extending the bytebuffer with 4 bytes. I can't allocate my bytebuffer with 6 bytes at start.

    ByteBuffer byteBuffer = ByteBuffer.allocate(2);

    byteBuffer.putChar('a');
    byteBuffer.putInt(1);

---

I am very impressed, on how many people are working on my question that is some minutes old. Thank you all very much and thanks also to Stackoverflow, which is a great Plattform !

All of you asked, what I am doing. So I try to explain it here. My usecase: I have structured data represented as javaobjects (javaclasses) which I would like to store and read to/from a db. Reading should be very fast. What I have done so far:

• java serialized and deserialized and stored it in a blob -> works well but too slow.

• tried several 3rd party serializer like kryo (which is very good), but not useable in my case (android).

my new strategy :-) : I do my own externalizing of my class. for that I would like to construct my entire data of my class sequential as an array of bytes. This can be slow. Then I store the bytearray to a blob in my db. When reading I would like to read the bytearray at once (bytearray is about 10k). (I will have a lot of them). Then parsing the bytearray to extract the structured data.

I thought using a bytebuffer is ideal for doing this, because of methods like putX and readX ? (X for chars, floats, int)

Answer 1

You can't. This is by design. You can allocate a new byte buffer and write the overflowing data to that. You can keep the ByteBuffers in a LinkedList (which will grow as needed), and you can even spool off the old ones to disk, if you're running out of memory to allocate new ones. If each ByteBuffer is the same size, trivial equations will let you access it as if it where just the one buffer, but you lose the ability to use slicing or compacting, or any of the cool things you can do with one of those. :)

But like people have said over and over again, it depends on what you need it for.

Answer 2

IMHO the best answer is to make sure you have more than enough space from the start. Having your ByteBuffer dynamically re-size is very expensive and much slower.

ByteBuffer byteBuffer = ByteBuffer.allocate(6 /* or more */);

byteBuffer.putChar('a');
byteBuffer.putInt(1);

---

The simplest buffer to use for characters is StringBuilder.

StringBuilder sb = new StringBuilder();
sb.append('a');
sb.append('b');

How would you add a new character after your last statement ?

sb.append('n');

Only when you are finished.

// if you need a ByteBuffer
ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes(StandardCharsets.UTF_8));
// bb will have two bytes for 'a' and 'b'

However, if you are appending to a ByteBuffer (and not using chars) I would suggest making the buffer larger than you should ever need and thus you never need to resize it. If you are concerned about using heap, you can use off heap instead.

// uses about 48 bytes of heap.
ByteBuffer bb = ByteBuffer.allocateDirect(1024*1024);

bb.putInt(1234);
bb.putDouble(1.111);
bb.putLong(12345678987654321L);

Answer 3

The problem with using byte buffers is that you are starting with a fixed size. This is great if you know how much data you want to hold. For example, if you are reading input and you only want to read 12 bytes, then only create a 12-byte array.

If you don't know how much data you are going to have in a byte buffer, you can do 2 things:

1. If you need fast speed, then just allocate a large array based on a prediction of how much data you are going to need. For example, if you are reading in a large file and you want to read at fast speeds, then you should use a very large byte buffer (perhaps a couple of MB depending on the size of the file).

2. Use a structure that uses dynamic allocation. This is the much better solution, but it is slower. If you are allocating very large arrays, then this will be very slow, but it will not waste any memory. If you take a byte buffer and allocate 512 KB but you only use 1KB, that is a ton of wasted space! With dynamically allocated structures (LinkedList, Stack, Queue, Trees), you can add and remove elements as you need

The final solution, which is not recommended at all because it not only wastes a lot of memory, it is also slow. You can allocate as much space as you need in a byte buffer, then when you need more memory, create a new buffer and copy the data over. This is what you're trying to do and it will be very inefficient.

In my opionion, Peter Lawrey's answer is an awesome solution to this problem just because you can easily go from a StringBuilder to a byte array. This has all of the efficiency and speed that you will need.

Answer 4

I found a solution for my usecase, which I would like to describe in short here. I accepted the answer of Mikkel, because regarding the question I guess it is the right answer, which also John mentoined. Also Thanks to Peter for the great explanations I could learn from.

I use DataInputStream, which is convenient for my situation. I create a stream of data based on my Javaobject with DataInputStream and the convenient methods like putInt,PutBoolean, PutString . Then I get the binaryarray of the DataInputStream amd store it as a blob to my db.

To reading is exactly the opposit way with DataInputStream.

With this I get a 500% better performance in reading the objects compared to javadeserialization of my javaobjects and a 25% less storage usage.

Answer 5

You can, if You keep buffer instance in variable. For example like this (kotlin):

fun ShortBuffer?.clone(addLimit: Int = 0): ShortBuffer? {
    if (this == null) {
        return ShortBuffer.allocate(addLimit)
    } else {
        val copy = ShortBuffer.allocate(this.limit() + addLimit)
        copy.put(this.array())
        return copy
    }
}

fun ShortBuffer?.safePut(shortArray: ShortArray?): ShortBuffer? {
    if (this == null) {
        if(shortArray == null) {
            return null
        } else {
            val ret = ShortBuffer.allocate(shortArray.size)
            ret.put(shortArray)
            return ret
        }
    } else if (shortArray == null) {
        return this
    } else if (this.remaining() < shortArray.size) {
        val ret = clone(shortArray.size-this.remaining())
        ret?.put(shortArray)
        return ret
    } else {
        this.put(shortArray)
        return this
    }
}

usage:

var buff = ShortBuffer.allocate(0)
buff = buff.safePut(shortArrayOf(1,2,3))