왜 BCL GZipStream(와 StreamReader)지 않은 안정적으로 검출하는 데이터 오류로 CRC32?
https://stackoverflow.com/questions/9471826
-
13-11-2019 - |
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
Russian문제
다른 날에 내가 달렸으로 질문 GZipStream 를 감지하지 않 손상 데이터(도 CRC32 전달)? (는 이 될 것"duplicate",나는 감정을 혼합하게 확인할 수 있게 되었습니다.나 또한 하나 추가 CRC32 타이틀,그러나 돌이켜 보면서 느끼는 장소의 나머지의 게시물).를 탐험 한 후에 문제가에 조금 내 자신,내가 생각하는 문제입 크 다른 것보다 질문은 처음에 그렸다.
나는 확장에 따라 다른 질문들을 테스트 코드는 실행 가능한에서 LINQPad 고 시도하는 더 나은 쇼케이스 CRC32(Cyclic Redundancy Check) 문제 않을 경우,실제로 존재한다.(이후 코드만 약간 수정 에 따라 원래는 것이 가능 테스트 설정/방법론의 결함 또는 또 다른 이상한 특질/PEBCAK 니다.)
결과는 홀므 손상된 데이터는 항상 일(어떤!) 예외 을 발생합니다.Note 단지 때로는 가 CRC32 확인하는 것 같이 실제로"일하기".손상된 바이트는 원인의 범위 밖의/나쁜 헤더의/나쁜 footer 무시할 수 있기 때문에 우리가 추측할 수 있는 이들을 죽이고 압축해제 이전 을 CRC32 검사(는 완벽하게 이해할, 는 경우에도,IndexOutOfRangeException 해야 합성에 의해 래핑 InvalidDataException)그래서,
왜 CRC32 확인 훨씬 적은 신뢰할 수 있는 것보다 해야할까? (왜 그것이 있는 경우"잘못된 데이터를(예외 없음)"아래?)
이후 GZip 으로 바닥이 포함 모두 CRC32 및 길이의 압축하지 않은 데이터 은 것으로 보인 오류 발견율이어야 한다"상당히 높은" -즉,나 지 않을 것입 예상 단 하나 실패하는 경우 아래,많고 적은 숫자의 발견되지 않은 손상된 스트림이 있습니다.(물론 그것은 좋은 감지된 증기있는 지금:하지만 최종전 감시 checksum 될 것으로 보인 솔직 무시됩 는 경우에.)
형식 CorruptByteIndex+FailedDetections: Message:
0+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream. 1+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream. 2+0: System.IO.InvalidDataException:The compression mode specified in GZip header is unknown. 3+0: Good data (No Exception) 4+0: Good data (No Exception) 5+0: Good data (No Exception) 6+0: Good data (No Exception) 7+0: Good data (No Exception) 8+0: Good data (No Exception) 9+0: Good data (No Exception) 10+0: System.IO.InvalidDataException:Unknown block type. Stream might be corrupted. 11+1: Invalid data (No Exception) 12+1: System.IO.InvalidDataException:Found invalid data while decoding. 13+1: System.IO.InvalidDataException:Found invalid data while decoding. 14+1: System.IO.InvalidDataException:Found invalid data while decoding. 15+1: System.IO.InvalidDataException:Found invalid data while decoding. 16+1: System.IO.InvalidDataException:Found invalid data while decoding. 17+2: Invalid data (No Exception) 18+2: System.IO.InvalidDataException:Found invalid data while decoding. 19+2: System.IndexOutOfRangeException:Index was outside the bounds of the array. 20+2: System.IndexOutOfRangeException:Index was outside the bounds of the array. 21+3: Invalid data (No Exception) 22+3: System.IndexOutOfRangeException:Index was outside the bounds of the array. 23+3: System.IndexOutOfRangeException:Index was outside the bounds of the array. 24+4: Invalid data (No Exception) 25+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 26+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 27+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 28+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 29+5: Invalid data (No Exception) 30+5: System.IndexOutOfRangeException:Index was outside the bounds of the array. 31+6: Invalid data (No Exception) 32+7: Invalid data (No Exception) 33+7: System.IndexOutOfRangeException:Index was outside the bounds of the array. 34+7: System.IndexOutOfRangeException:Index was outside the bounds of the array. 35+7: System.IndexOutOfRangeException:Index was outside the bounds of the array. 36+8: Invalid data (No Exception) 37+8: System.IndexOutOfRangeException:Index was outside the bounds of the array. 38+8: System.IndexOutOfRangeException:Index was outside the bounds of the array. 39+9: Invalid data (No Exception) 40+9: System.IndexOutOfRangeException:Index was outside the bounds of the array. 41+9: System.IndexOutOfRangeException:Index was outside the bounds of the array. 42+10: Invalid data (No Exception) 43+10: System.IO.InvalidDataException:Found invalid data while decoding. 44+10: System.IndexOutOfRangeException:Index was outside the bounds of the array. 45+10: System.IO.InvalidDataException:Found invalid data while decoding. 46+11: Invalid data (No Exception) 47+11: System.IndexOutOfRangeException:Index was outside the bounds of the array. 48+11: System.IndexOutOfRangeException:Index was outside the bounds of the array. 49+11: System.IndexOutOfRangeException:Index was outside the bounds of the array. 50+12: Invalid data (No Exception) 51+12: System.IndexOutOfRangeException:Index was outside the bounds of the array. 52+12: System.IndexOutOfRangeException:Index was outside the bounds of the array. 53+13: Invalid data (No Exception) 54+13: System.IndexOutOfRangeException:Index was outside the bounds of the array. 55+14: Invalid data (No Exception) 56+14: System.IndexOutOfRangeException:Index was outside the bounds of the array. 57+15: Invalid data (No Exception) 58+15: System.IndexOutOfRangeException:Index was outside the bounds of the array. 59+15: System.IndexOutOfRangeException:Index was outside the bounds of the array. 60+16: Invalid data (No Exception) 61+17: Invalid data (No Exception) 62+18: Invalid data (No Exception) 63+19: Invalid data (No Exception) 64+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 65+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 66+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 67+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 68+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 69+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 70+19: System.IO.InvalidDataException:Found invalid data while decoding. 71+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 72+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 73+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 74+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 75+19: System.IO.InvalidDataException:Found invalid data while decoding. 76+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 77+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 78+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 79+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 80+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 81+19: System.IO.InvalidDataException:Found invalid data while decoding. 82+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 83+20: Invalid data (No Exception) 84+21: Invalid data (No Exception) 85+22: Invalid data (No Exception) 86+22: System.IndexOutOfRangeException:Index was outside the bounds of the array. 87+23: Invalid data (No Exception) 88+24: Invalid data (No Exception) 89+25: Invalid data (No Exception) 90+25: System.IndexOutOfRangeException:Index was outside the bounds of the array. 91+26: Invalid data (No Exception) 92+26: System.IO.InvalidDataException:Found invalid data while decoding. 93+26: System.IndexOutOfRangeException:Index was outside the bounds of the array. 94+27: Invalid data (No Exception) 95+27: System.IndexOutOfRangeException:Index was outside the bounds of the array. 96+27: System.IndexOutOfRangeException:Index was outside the bounds of the array. 97+28: Invalid data (No Exception) 98+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 99+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 100+29: Invalid data (No Exception) 101+30: Invalid data (No Exception) 102+31: Invalid data (No Exception) 103+32: Invalid data (No Exception) 104+32: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 105+33: Invalid data (No Exception) 106+34: Invalid data (No Exception) 107+35: Invalid data (No Exception) 108+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 109+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 110+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 111+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 112+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 113+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 114+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 115+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 116+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 117+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 118+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 119+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 120+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 121+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 122+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 123+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 124+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 125+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 126+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 127+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 128+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 129+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 130+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 131+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 132+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 133+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 134+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 135+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 136+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 137+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 138+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 139+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 140+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 141+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 142+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 143+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 144+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 145+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 146+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 147+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 148+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 149+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 150+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 151+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 152+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 153+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 154+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 155+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 156+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 157+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 158+36: Invalid data (No Exception) 159+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 160+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 161+37: Invalid data (No Exception) 162+38: Invalid data (No Exception) 163+39: Invalid data (No Exception) 164+40: Invalid data (No Exception) 165+41: Invalid data (No Exception) 166+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 167+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 168+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 169+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 170+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size. 171+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size. 172+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size. 173+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.
여기에는 이 테스트는 복사본을'n 하여 붙여넣기 실행 가능한에서 LINQPad(습니다.NET3.5 과 4 사용"으로 C#문에서"모드):
string sample = "This is a compression test of microsoft .net gzip compression method and decompression methods";
var encoding = new ASCIIEncoding();
var data = encoding.GetBytes(sample);
string sampleOut = null;
byte[] cmpData;
// Compress
using (var cmpStream = new MemoryStream())
{
using (var hgs = new System.IO.Compression.GZipStream(cmpStream, System.IO.Compression.CompressionMode.Compress))
{
hgs.Write(data, 0, data.Length);
}
cmpData = cmpStream.ToArray();
}
int corruptBytesNotDetected = 0;
// corrupt data byte by byte
for (var byteToCorrupt = 0; byteToCorrupt < cmpData.Length; byteToCorrupt++)
{
var corruptData = new List<byte>(cmpData).ToArray();
// corrupt the data
corruptData[byteToCorrupt]++;
using (var decomStream = new MemoryStream(corruptData))
{
using (var hgs = new System.IO.Compression.GZipStream(decomStream, System.IO.Compression.CompressionMode.Decompress))
{
using (var reader = new StreamReader(hgs))
{
string message;
try
{
sampleOut = reader.ReadToEnd();
// if we get here, the corrupt data was not detected by GZipStream
// ... okay so long as the correct data is extracted
if (!sample.SequenceEqual(sampleOut)) {
corruptBytesNotDetected++;
message = "Invalid data (No Exception)";
} else {
message = "Good data (No Exception)";
}
}
catch(Exception ex)
{
message = (ex.GetType() + ":" + ex.Message);
}
string.Format("{0}+{1}: {2}",
byteToCorrupt, corruptBytesNotDetected, message).Dump();
}
}
}
}
여기에 압축된 데이터 .NET3.5 (GZipStream 악명에서"압축하는"작은 페이로드 하지만 그것은"해결되지 않습니다"문제이기 때문에 스트림은 기술적으로 유효):
1F 8B 08 00 00 00 00 00 04 00 ED BD 07 60 1C 49 96 25 26 2F 6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10 EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D 66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B 9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80 AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A AC EA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D 3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CE F2 FE 17 CD FF 03 5C 51 5E 27 5E 00 00 00
(고,그냥 웃음을 위해,습니다.순 4 성미/다른 압축된다.)
1F 8B 08 00 00 00 00 00 04 00 EC BD 07 60 1C 49 96 25 26 2F 6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10 EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D 66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B 9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80 AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A AC EA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D 3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CE F2 FE 17 CD FF 13 00 00 FF FF 5C 51 5E 27 5E 00 00 00
추가 사항:
테스트 수 이 미묘하게 결함이 이 경우.때 GZipStream"를 감지하지 못 손상"(예외)다음에서 읽은 데이터 StreamReader 은""(an empty string):이 경우에는 이유는 무엇 ReadToEnd() 지 예외를 발생시킬(IOException 또는 그렇지 않으면)?
따라서 그것은 지 GZipStream 지만 오히려 StreamReader 는"quirky"여기에서 또는 이전에 문제가 GZipStream(를 던지지 않는 예외)?이 올바르게 처리하는 방법이 사용할 경우 안정적으로?(고려할 때 입력 스트림에서 현재 위치 말 비어 있습니다.)
해결책
서문
.NET[4 과 이]사용자가 사용하여 Microsoft 에서 제공하는 GZipStream 또는 사용 하 여 deflatestream 클래스가 어떤 상황에서도하지 않는 한,Microsoft 체와 함께 그들을 완전히 무언가는 작품이다.사용 DotNetZip 라이브러리 대신 합니다.
업데이트를 서문
니다.NET Framework4.5 그리고 나중에 고정된 압축 문제가 및 GZipStream 및 사용 하 여 deflatestream 사용 zlib 에서 이러 버전입니다.내가 알지 못하는 경우 CRC 문제가 아래 참조하는 수정되었습니다.
또 다른 업데이트
CRC 버그만 고쳐지지 않지만,Microsoft 기로 결정했다는 그 해결되지 않습니다 니다!
나의 응답에 왜 C#gzip 생성보다 큰 파일 또는 Fiddler PHP? 이 동작하지 않을 반영한 적절한 구현 gzip 손상을 탐지합니다.모든 경우의 테스트,에 오류가 감지에 의해 올바로 구현하시기 바랍니다.(는 응답도 왜 일곱의 경우 결과에 좋은 데이터입니다.)
다른 질문입니다:이것은 어떻게"압축"방법을 생산하 174 의 바이트 출력에서 94 바이트 문자열?특히 방법으로 보고서 문자열 압축식--gzip 으로 감소하 84-bytes,도의 오버헤드와 18 바이트 헤더 및 트레일러입니다.를 제공할 수 있습니 hex 의 덤프하는 174 바이트?
제휴하지 않습니다 StackOverflow
