Requested array size exceeds VM limit

Question

Your embedding is wrong. Embedding a byte in RGBA pixels is easy because you can conveniently fit half a byte per pixel. But for RGB you can fit 3/8 of a byte, which is not an integer number. Here is a demonstration of how complicated it is for RGB (assume that we start at pixel (0, 0)):

// Byte 1
(0, 0, R) (0, 0, G) (0, 0, B)
(0, 1, R) (0, 1, G) (0, 1, B)
(0, 2, R) (0, 2, G)

// Byte 2
                    (0, 2, B)
(0, 3, R) (0, 3, G) (0, 3, B)
(0, 4, R) (0, 4, G) (0, 4, B)
(0, 5, R)

// Byte 3
          (0, 5, G) (0, 5, B)
(0, 6, R) (0, 6, G) (0, 6, B)
(0, 7, R) (0, 7, G) (0, 7, B)

As you can see, some times you need to embed to 3 or 4 different pixels and you don't always start/end from the R component. The code you need to achieve this is the following.

EncodeMessage

private void embedMessage(BufferedImage img, String mess) {
   int messageLength = mess.length();

   int imageWidth = img.getWidth(), imageHeight = img.getHeight(),
      imageSize = imageWidth * imageHeight;
   if((messageLength * 8 + 32)/3 > imageSize) {
      JOptionPane.showMessageDialog(this, "Message is too long for the chosen image",
         "Message too long!", JOptionPane.ERROR_MESSAGE);
      return;
      }
   embedInteger(img, messageLength, 0);

   byte b[] = mess.getBytes();
   for(int i=0; i<b.length; i++)
      embedByte(img, b[i], i*8+32);
   }

private void embedInteger(BufferedImage img, int n, int start) {
   int mod = start%3;
   start = start/3;
   int maxX = img.getWidth(), maxY = img.getHeight(), 
      startX = start/maxY, startY = start - startX*maxY, count=0;
   for(int i=startX; i<maxX && count<32; i++) {
      for(int j=startY; j<maxY && count<32; j++) {
         int rgb = img.getRGB(i, j), bit = 0, pp = 0;
         if(count <= 29) {
            for(pp=mod; pp<3; pp++) {
               bit = getBitValue(n, count); rgb = setBitValue(rgb, 8*pp, bit);
               count += 1;
               }
            mod = 0;
            }
         else {
            for(pp=0; pp<(33-count); pp++) {
               bit = getBitValue(n, count); rgb = setBitValue(rgb, 8*pp, bit);
               count += 1;
               }
            }
         img.setRGB(i, j, rgb);
         }
      }
   }

private void embedByte(BufferedImage img, byte b, int start) {
   int mod = start%3;
   start = start/3;
   int maxX = img.getWidth(), maxY = img.getHeight(), 
      startX = start/maxY, startY = start - startX*maxY, count=0;
   for(int i=startX; i<maxX && count<8; i++) {
      for(int j=startY; j<maxY && count<8; j++) {
         if(j==maxY-1){
            startY = 0;
            }
         int rgb = img.getRGB(i, j), bit = 0, pp = 0;
         if(count <= 5) {
            for(pp=mod; pp<3; pp++) {
               bit = getBitValue(b, count); rgb = setBitValue(rgb, 8*pp, bit);
               count += 1;
                  }
            mod = 0;
            }
         else {
            for(pp=0; pp<(9-count); pp++) {
               bit = getBitValue(b, count); rgb = setBitValue(rgb, 8*pp, bit);
               count += 1;
               }
            }        
         img.setRGB(i, j, rgb);
         }
      }
   }

DecodeMessage

private void decodeMessage() {
   int len = extractInteger(image, 0);
   byte b[] = new byte[len];
   for(int i=0; i<len; i++)
      b[i] = extractByte(image, i*8+32);
   message.setText(new String(b));
   }

private int extractInteger(BufferedImage img, int start) {
   int mod = start%3;
   start = start/3;
   int maxX = img.getWidth(), maxY = img.getHeight(), 
      startX = start/maxY, startY = start - startX*maxY, count=0;
   int length = 0;
   for(int i=startX; i<maxX && count<32; i++) {
      for(int j=startY; j<maxY && count<32; j++) {
         int rgb = img.getRGB(i, j), bit = 0, pp = 0;
         if(count <= 29) {
            for(pp=mod; pp<3; pp++) {
               bit = getBitValue(rgb, 8*pp); length = setBitValue(length, count, bit);
               count += 1;
               }
            mod = 0;
            }
         else {
            for(pp=0; pp<(33-count); pp++) {
               bit = getBitValue(rgb, 8*pp); length = setBitValue(length, count, bit);
               count += 1;
               }
            }
         }
      }
   return length;
   }

private byte extractByte(BufferedImage img, int start) {
   int mod = start%3;
   start = start/3;
   int maxX = img.getWidth(), maxY = img.getHeight(), 
      startX = start/maxY, startY = start - startX*maxY, count=0;
   byte b = 0;
   for(int i=startX; i<maxX && count<8; i++) {
      for(int j=startY; j<maxY && count<8; j++) {
         if(j==maxY-1){
            startY = 0;
            }
         int rgb = img.getRGB(i, j), bit = 0, pp = 0;
         if(count <= 5) {
            for(pp=mod; pp<3; pp++) {
               bit = getBitValue(rgb, 8*pp); b = (byte)setBitValue(b, count, bit);
               count += 1;
               }
            mod = 0;
            }
         else {
            for(pp=0; pp<(9-count); pp++) {
               bit = getBitValue(rgb, 8*pp); b = (byte)setBitValue(b, count, bit);
               count += 1;
               }
            }
         }
      }
   return b;
   }

I will briefly explain the logic behind this. Since it's similar for both encoding and decoding, I will only describe the former. And since embedInteger and embedByte are similar, I will only describe embedByte.

In embedMessage, we need to pass i*8+32 in embedByte because we need the number of bits we have embedded so far. This is necessary so to know where the embedding from the previous byte stopped (as shown above, after byte 1 we have to start from B, while after byte 2 from G). This is achieved with the modulo operation (int mod = start%3), which gives you the remainder of division by 3. For example, 8%3 = 2. For mod = 0, we start at R, for mod = 1 at G and for mod = 2 at B.

start = start/3 tells you at which pixel we have to start. An integer division gives you a rounded down integer result, so for example 8/3 = round down 2.666 = 2. As you can see, start and mod give us all the information for where we have to start. For example, after one byte we start at pixel 2, B component. Now we can start embedding the byte inside the i and j loops.

Inside the loops, we fetch our new rgb pixel. Depending on how many bits we have embedded so far, we may embed in the whole RGB, or just part of it. The object count tells us how many bits we have embedded so far and overall we embed 8 bits. This is where the if-else statement comes in. Effectively, we ask the question "do we have more than 3 bits left to embed?" This is translated as 8-count >= 3 and when this is algebraically rearranged, you get count <= 5. To sum up:

if: we have enough bits left to embed in all 3 components

else: we don't have enough bits left to embed in all 3 components

Now, pp decides in which colour component we embed our bit and it can take the values 0, 1 and 2. The java syntax for that is for(pp=0; p<3; pp++). That's just how it is. Then 8*pp can be 0, 8 or 16, which is the LSB for R, G or B.

In the if block, we have for(pp=mod; ...), because we may not start from 0. Look at the above example, where byte 2 has mod=2, because we start from the blue component. But once that loop is over, mod will be reset to 0, so for the next pixel we do start from 0.

To understand the else block (for(pp=0; pp<(9-count); p++)), look at the example from above for byte 1. We have already embedded in the first 2 pixels, so we have two more bits left. This means we have count=6.

pp=0; the condition to exit is pp<9-6 --> pp<3, so we continue. We embed the 7th bit in R --> count=7.

pp=1; the condition to exit is pp<9-7 --> pp<2, so we continue. We embed the 8th bit in G --> count=8.

pp=2; the condition to exit is pp<9-8 --> pp<1 --> WE HAVE TO EXIT.

A simpler equivalent to that logic is the following and I don't know why I don't go with this.

for(pp=0; pp<(8-count); pp++) {
   getBit...; setBit...;
   }
count = 8;