Pourquoi BCL GZIPStream (avec StreamReader) ne détecte-t-il pas de manière fiable les erreurs de données avec CRC32?

Question

L'autre jour où j'ai rencontré la question GZIPStream ne détecte pas les données corrompues (même les passes CRC32)? (Dont cela pourrait très bien être un "duplicata", j'ai des sentiments mitigés sur le sujet. Je suis aussi celui qui a ajouté le CRC32 au titre, mais rétrospectivement qui se sent à sa place avec le reste du poste). Après avoir exploré un peu le problème par moi-même, je pense que le problème est bien meilleur que l'autre question dépeint initialement.

J'ai étendu l'autre question et j'ai rendu le code de test passant sous Linqpad et j'ai tenté de mieux présenter le CRC32 (chèque de redondance cyclique) problème, s'il existe effectivement. (Puisque le code n'est qu'un légère modification Sur la base de l'original, il est possible que la configuration / méthodologie du test soit défectueuse ou qu'il y ait une autre étrange bizarrerie / pebcak les deux.)

Les résultats sont étranges car Les données corrompues sont pas toujours provoquant une exception (n'importe quoi!) être élevé. Notez que Seulement parfois Le chèque CRC32 semble-t-il réellement "fonctionner". Les octets corrompus qui provoquent l'index en dehors de la gamme / le mauvais en-tête / le mauvais pied de page peuvent être ignorés car nous pouvons supposer qu'ils tuent la décompression préalable à la vérification CRC32 (qui est parfaitement compréhensible, même si l'indexoutofrangeException devrait probablement être enveloppé par une invaliddataexception) donc,

Pourquoi le contrôle CRC32 est-il beaucoup moins fiable qu'il ne devrait l'être? (Pourquoi est-il le cas qu'il y a "des données non valides (pas d'exception)" ci-dessous?)

Depuis le Le pied de page GZIP contient tous les deux le CRC32 et la longueur du données non compressées est semble que le Le taux de détection des erreurs doit être "significativement plus élevé" - c'est-à-dire je ne serait pas Attendez-vous à un seul cas défaillant ci-dessous, encore moins un certain nombre de flux corrompus non détectés. (Bien sûr, c'est bien de détecter une vapeur corrompue dès que possible: mais la somme de contrôle de sécurité finale semble être carrément ignoré dans les cas.)

Le format est CorruptByteIndex+FailedDetections: Message:

0+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
1+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
2+0: System.IO.InvalidDataException:The compression mode specified in GZip header is unknown.
3+0: Good data (No Exception)
4+0: Good data (No Exception)
5+0: Good data (No Exception)
6+0: Good data (No Exception)
7+0: Good data (No Exception)
8+0: Good data (No Exception)
9+0: Good data (No Exception)
10+0: System.IO.InvalidDataException:Unknown block type. Stream might be corrupted.
11+1: Invalid data (No Exception)
12+1: System.IO.InvalidDataException:Found invalid data while decoding.
13+1: System.IO.InvalidDataException:Found invalid data while decoding.
14+1: System.IO.InvalidDataException:Found invalid data while decoding.
15+1: System.IO.InvalidDataException:Found invalid data while decoding.
16+1: System.IO.InvalidDataException:Found invalid data while decoding.
17+2: Invalid data (No Exception)
18+2: System.IO.InvalidDataException:Found invalid data while decoding.
19+2: System.IndexOutOfRangeException:Index was outside the bounds of the array.
20+2: System.IndexOutOfRangeException:Index was outside the bounds of the array.
21+3: Invalid data (No Exception)
22+3: System.IndexOutOfRangeException:Index was outside the bounds of the array.
23+3: System.IndexOutOfRangeException:Index was outside the bounds of the array.
24+4: Invalid data (No Exception)
25+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
26+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
27+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
28+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.
29+5: Invalid data (No Exception)
30+5: System.IndexOutOfRangeException:Index was outside the bounds of the array.
31+6: Invalid data (No Exception)
32+7: Invalid data (No Exception)
33+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.
34+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.
35+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.
36+8: Invalid data (No Exception)
37+8: System.IndexOutOfRangeException:Index was outside the bounds of the array.
38+8: System.IndexOutOfRangeException:Index was outside the bounds of the array.
39+9: Invalid data (No Exception)
40+9: System.IndexOutOfRangeException:Index was outside the bounds of the array.
41+9: System.IndexOutOfRangeException:Index was outside the bounds of the array.
42+10: Invalid data (No Exception)
43+10: System.IO.InvalidDataException:Found invalid data while decoding.
44+10: System.IndexOutOfRangeException:Index was outside the bounds of the array.
45+10: System.IO.InvalidDataException:Found invalid data while decoding.
46+11: Invalid data (No Exception)
47+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.
48+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.
49+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.
50+12: Invalid data (No Exception)
51+12: System.IndexOutOfRangeException:Index was outside the bounds of the array.
52+12: System.IndexOutOfRangeException:Index was outside the bounds of the array.
53+13: Invalid data (No Exception)
54+13: System.IndexOutOfRangeException:Index was outside the bounds of the array.
55+14: Invalid data (No Exception)
56+14: System.IndexOutOfRangeException:Index was outside the bounds of the array.
57+15: Invalid data (No Exception)
58+15: System.IndexOutOfRangeException:Index was outside the bounds of the array.
59+15: System.IndexOutOfRangeException:Index was outside the bounds of the array.
60+16: Invalid data (No Exception)
61+17: Invalid data (No Exception)
62+18: Invalid data (No Exception)
63+19: Invalid data (No Exception)
64+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
65+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
66+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
67+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
68+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
69+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
70+19: System.IO.InvalidDataException:Found invalid data while decoding.
71+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
72+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
73+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
74+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
75+19: System.IO.InvalidDataException:Found invalid data while decoding.
76+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
77+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
78+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
79+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
80+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
81+19: System.IO.InvalidDataException:Found invalid data while decoding.
82+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.
83+20: Invalid data (No Exception)
84+21: Invalid data (No Exception)
85+22: Invalid data (No Exception)
86+22: System.IndexOutOfRangeException:Index was outside the bounds of the array.
87+23: Invalid data (No Exception)
88+24: Invalid data (No Exception)
89+25: Invalid data (No Exception)
90+25: System.IndexOutOfRangeException:Index was outside the bounds of the array.
91+26: Invalid data (No Exception)
92+26: System.IO.InvalidDataException:Found invalid data while decoding.
93+26: System.IndexOutOfRangeException:Index was outside the bounds of the array.
94+27: Invalid data (No Exception)
95+27: System.IndexOutOfRangeException:Index was outside the bounds of the array.
96+27: System.IndexOutOfRangeException:Index was outside the bounds of the array.
97+28: Invalid data (No Exception)
98+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
99+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
100+29: Invalid data (No Exception)
101+30: Invalid data (No Exception)
102+31: Invalid data (No Exception)
103+32: Invalid data (No Exception)
104+32: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
105+33: Invalid data (No Exception)
106+34: Invalid data (No Exception)
107+35: Invalid data (No Exception)
108+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
109+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
110+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
111+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
112+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
113+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
114+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
115+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
116+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
117+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
118+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
119+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
120+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
121+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
122+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
123+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
124+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
125+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
126+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
127+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
128+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
129+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
130+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
131+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
132+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
133+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
134+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
135+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
136+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
137+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
138+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
139+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
140+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
141+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
142+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
143+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
144+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
145+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
146+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
147+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
148+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
149+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
150+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
151+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
152+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
153+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
154+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
155+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
156+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
157+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
158+36: Invalid data (No Exception)
159+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
160+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
161+37: Invalid data (No Exception)
162+38: Invalid data (No Exception)
163+39: Invalid data (No Exception)
164+40: Invalid data (No Exception)
165+41: Invalid data (No Exception)
166+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
167+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
168+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
169+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.
170+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.
171+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.
172+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.
173+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.

Voici le test qui est Copy'n'Paste Runnable dans Linqpad (pour .NET 3.5 et 4, utilisez "AS C # instructions" Mode):

   string sample = "This is a compression test of microsoft .net gzip compression method and decompression methods";
   var encoding = new ASCIIEncoding();
   var data = encoding.GetBytes(sample);
   string sampleOut = null;
   byte[] cmpData;

   // Compress 
   using (var cmpStream = new MemoryStream())
   {
      using (var hgs = new System.IO.Compression.GZipStream(cmpStream, System.IO.Compression.CompressionMode.Compress))
      {
         hgs.Write(data, 0, data.Length);
      }
      cmpData = cmpStream.ToArray();
   }

   int corruptBytesNotDetected = 0;

   // corrupt data byte by byte
   for (var byteToCorrupt = 0; byteToCorrupt < cmpData.Length; byteToCorrupt++)
   {
      var corruptData = new List<byte>(cmpData).ToArray();
      // corrupt the data
      corruptData[byteToCorrupt]++;

      using (var decomStream = new MemoryStream(corruptData))
      {
         using (var hgs = new System.IO.Compression.GZipStream(decomStream, System.IO.Compression.CompressionMode.Decompress))
         {
            using (var reader = new StreamReader(hgs))
            {
               string message;
               try
               {
                  sampleOut = reader.ReadToEnd();

                  // if we get here, the corrupt data was not detected by GZipStream
                  // ... okay so long as the correct data is extracted

                  if (!sample.SequenceEqual(sampleOut)) {
                    corruptBytesNotDetected++;
                    message = "Invalid data (No Exception)";
                  } else {
                    message = "Good data (No Exception)";
                  }
               }
               catch(Exception ex)
               {
                    message = (ex.GetType() + ":" + ex.Message);
               }
               string.Format("{0}+{1}: {2}",
                    byteToCorrupt, corruptBytesNotDetected, message).Dump();
            }
         }
      }

   }

Voici les données compressées dans .NET 3.5 (Gzipstream est notoirement mauvais pour "compresser" de petites charges utiles, mais c'est un problème "ne résoudre pas" car le flux est toujours techniquement valide):

1F 8B 08 00 00 00 00 00 04 00 ED BD 07 60 1C 49 96 25 26 2F
6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10
EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D
66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B
9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80
AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A AC
EA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D
3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CE
F2 FE 17 CD FF 03 5C 51 5E 27 5E 00 00 00

(Et, juste pour les rires, dans .NET 4, il génère un flux comprimé légèrement plus grand / différent.)

1F 8B 08 00 00 00 00 00 04 00 EC BD 07 60 1C 49 96 25 26 2F
6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10
EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D
66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B
9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80
AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A AC
EA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D
3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CE
F2 FE 17 CD FF 13 00 00 FF FF 5C 51 5E 27 5E 00 00 00

Notes complémentaires:

Le test peut être subtilement défectueux dans ce cas. Lorsque le gzipstream "ne parvient pas à détecter la corruption" (pas d'exception), les données lues par le leader est "" (une chaîne vide): Dans ce cas, pourquoi ReadToEnd() ne pas élever une exception (ioException ou autre)?

Est-ce ainsi ne pas Gzipstream mais plutôt le réalisateur qui est "original" ici ou est-ce toujours un problème avec le gzipstream (pour ne pas avoir exprimé)? Existe-t-il une façon correcte de gérer ce cas d'utilisation de manière fiable? (Considérez lorsque le flux d'entrée de la position actuelle c'est vrai vide.)

Answer 1

Préface

Les utilisateurs .NET [4 et précédents] ne doivent pas utiliser les classes GZIPStream ou Delatestream fournies par Microsoft en aucune circonstance, à moins que Microsoft ne les remplace complètement par quelque chose qui fonctionne. Utilisez plutôt la bibliothèque dotnetzip.

Mise à jour de la préface

Le .NET Framework 4.5 et ultérieurement ont résolu le problème de compression, et GZIPStream et Deflatestream utilisent ZLIB dans ces versions. Je ne sais pas si le problème CRC référencé ci-dessous a été résolu.

Une autre mise à jour

Le bogue CRC n'est pas seulement corrigé, mais Microsoft a décidé qu'ils ne réparera pas ce!

 

Ma réponse dans Pourquoi mon C # GZIP produit-il un fichier plus grand que Fiddler ou PHP? montre que ce comportement ne reflète pas une mise en œuvre appropriée de la détection de corruption GZIP. Dans tous les cas testés, l'erreur serait détectée par une implémentation appropriée. (Cette réponse note également pourquoi sept des cas entraînent de bonnes données.)

Une autre question est: comment cette méthode de "compression" produit-elle 174 octets de sortie à partir d'une chaîne de 94 octets? Surtout vu que la chaîne est compressible - GZIP le réduit à 84 octets, même avec les frais généraux d'un en-tête et d'une remorque de 18 octets. Pouvez-vous fournir un vidage hexagonal de ces 174 octets?