Truncating a stream of unknown length x bytes from the end? (.NET)

Question

I need to read a stream of unknown length, excluding the last 20 bytes (hash data). The setup is roughly:

Source Stream (with SHA1 hash last 20 bytes) -> SHA1 Hasher Stream (compute on-the-fly and compare with embedded stream hash when stream ends) -> AES Decryption Stream -> Do stuff with the data...

I cannot buffer the whole source stream before processing as it may be many gigabytes, it all needs to happen on-the-fly. The source stream is not seekable. At present the SHA1 stream is reading the last 20 bytes into its buffers which breaks everything, and I'm not aware of any way to control this behaviour.

I was thinking of inserting a wrapper stream in between the Source and SHA1 streams, implementing a rolling buffer(?) that presents the source stream to the AES wrapper in 4096 byte chunks, and then 'fakes' the end of the stream 20 bytes earlier on the last read. The 20 byte hash would then be exposed via a property.

Would this be the best solution, and how would I implement it?

The rough code flow is below (from memory, probably won't compile):

SourceStream = TcpClient.Stream
HashedStream = New CryptoStream(SourceStream, Sha1Hasher, CryptoStreamMode.Read)
AesDecryptedStream = New CryptoStream(HashedStream, AesDecryptor, CryptoStreamMode.Read)

' Read out and deserialize data
AesDecryptedStream.Read(etc...)

' Check if signatures match, throw data away if not
If Not Sha1Hash.SequenceEqual(ExpectedHash)

' Do stuff with the data here

Edit: The stream format is as follows:

[  StreamFormat  |  String  |  Required  ]
[  WrapperFlags  |  8 Bit BitArray  |  Required  ]
[  Sha1 Hashed Data Wrapper  |  Optional  ]
   [  AesIV  |  16 Bytes  |  Required if Aes Encrypted  ]
   [  Aes Encrypted Data Wrapper  |  Optional  ]
      [  Gzip Compressed Data Wrapper  |  Optional  ]
         [  Payload Data  |  Binary  |  Required  ]
      [  End Gzip Compressed Data  ]
   [  End Aes Encrypted Data  ]
[  End Sha1 Hashed Data  ]
[  Sha1HashValue  |  20 Bytes  |  Required if Sha1 Hashed  ]

Answer 1

I've written you a quick little stream which buffers ahead 20 bytes. The only real implementation I've overriden properly is the Read() member, you may have to examine the other Stream members appropriately to your case. Comes with a free test class, too! Bonus! I tested it more thoroughly but you can adapt these test cases to your will. Oh and by the way, I didn't test it for streams fewer than 20 bytes in length.

Test Case

[TestClass]
    public class TruncateStreamTests
    {
        [TestMethod]
        public void TestTruncateLast20Bytes()
        {
            string testInput = "This is a string.-- final 20 bytes --";
            string expectedOutput = "This is a string.";
            string testOutput;
            using (var testStream = new StreamWhichEndsBeforeFinal20Bytes(new MemoryStream(Encoding.ASCII.GetBytes(testInput))))
            using (var streamReader = new StreamReader(testStream, Encoding.ASCII))
            {
                testOutput = streamReader.ReadLine();
            }

            Assert.AreEqual(expectedOutput, testOutput);
        }

        [TestMethod]
        public void TestTruncateLast20BytesRead3BytesAtATime()
        {
            string testInput = "This is a really really really really really long string, longer than all the others\n\rit even has some carriage returns in it, etc.-- final 20 bytes --";
            string expectedOutput = "This is a really really really really really long string, longer than all the others\n\rit even has some carriage returns in it, etc.";
            StringBuilder testOutputBuilder = new StringBuilder();
            using (var testStream = new StreamWhichEndsBeforeFinal20Bytes(new MemoryStream(Encoding.ASCII.GetBytes(testInput))))
            {
                int bytesRead = 0;
                do
                {
                    byte[] buffer = new byte[3];
                    bytesRead = testStream.Read(buffer, 0, 3);
                    testOutputBuilder.Append(Encoding.ASCII.GetString(buffer, 0, bytesRead));
                } while (bytesRead > 0);
            }
            Assert.AreEqual(expectedOutput, testOutputBuilder.ToString());
        }
    }

Stream Class

 public class StreamWhichEndsBeforeFinal20Bytes : Stream
    {
        private readonly Stream sourceStream;

        private static int TailBytesCount = 20;

        public StreamWhichEndsBeforeFinal20Bytes(Stream sourceStream)
        {
            this.sourceStream = sourceStream; 
        }

        public byte[] TailBytes { get { return previousTailBuffer; } }

        public override void Flush()
        {
            sourceStream.Flush();
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            return sourceStream.Seek(offset, origin);
        }

        public override void SetLength(long value)
        {
            sourceStream.SetLength(value);
        }

        private byte[] previousTailBuffer;

        public override int Read(byte[] buffer, int offset, int count)
        {
            byte[] tailBuffer = new byte[TailBytesCount];
            int expectedBytesRead;

            if (previousTailBuffer == null)
                expectedBytesRead = count + TailBytesCount;
            else
                expectedBytesRead = count;

            try
            {
                byte[] readBuffer = new byte[expectedBytesRead];
                int actualBytesRead = sourceStream.Read(readBuffer, offset, expectedBytesRead);

                if (actualBytesRead == 0) return 0;

                if (actualBytesRead < TailBytesCount)
                {
                    int pickPreviousByteCount = TailBytesCount - actualBytesRead;

                    if (previousTailBuffer != null)
                    {
                        int pickFromIndex = previousTailBuffer.Length - pickPreviousByteCount;
                        Array.Copy(previousTailBuffer, 0, buffer, offset, count);
                        Array.Copy(previousTailBuffer, pickFromIndex, tailBuffer, 0, pickPreviousByteCount);
                    }

                    Array.Copy(readBuffer, 0, tailBuffer, pickPreviousByteCount, actualBytesRead);
                    return actualBytesRead;
                }

                Array.Copy(readBuffer, actualBytesRead - TailBytesCount, tailBuffer, 0, TailBytesCount);
                Array.Copy(readBuffer, 0, buffer, offset, actualBytesRead - TailBytesCount);

                if (actualBytesRead < expectedBytesRead)
                {
                    return actualBytesRead - TailBytesCount;
                }
                return count;
            }
            finally
            {
                previousTailBuffer = tailBuffer;
            }
        }


        public override void Write(byte[] buffer, int offset, int count)
        {
            sourceStream.Write(buffer, offset, count);
        }

        public override bool CanRead
        {
            get { return sourceStream.CanRead; }
        }

        public override bool CanSeek
        {
            get { return sourceStream.CanSeek; }
        }

        public override bool CanWrite
        {
            get { return sourceStream.CanWrite; }
        }

        public override long Length
        {
            get
            {
                if (sourceStream.Length < TailBytesCount) return sourceStream.Length;
                return sourceStream.Length - TailBytesCount;
            }
        }

        public override long Position
        {
            get { return sourceStream.Position; }
            set { sourceStream.Position = value; }
        }
    }