HttpWebResponse cutting off early when downloading large file

Question

I am using a C# Client class provided by the USPS EPF (Electronic Product Fulfillment) in order to download USPS files through a console application. I use my console app to login with my USPS credentials, specify the file I want to download, and get the file. All of this works great for the two smaller files we have access to (AMS Developer Kit, 47.7 MB, and DPV Developer Kit, 1.59 MB). However, when I try to download the 2.8 GB AMS Commercial DVD file, which is the only one I really care about, I run into problems. The console app stops downloading the file at 1.75 GB every time. Since it's a .tar file, I can open it and see some of the contents, but naturally a lot is missing. The USPS-provided Client class does not throw an exception or error of any kind; it's supposed to read to the end of the file, but it just stops early.

I have tried everything I can think of: altering the HttpWebRequest properties (changing KeepAlive to true, increasing the Timeout value), modifying the getEpfFile method to use an IsolatedStorageFile instead of a MemoryStream to get the file, even checking with our networking people to make sure there isn't some arbitrary network setting causing a timeout. I've tried downloading from my machine and different network servers with the same result. I looked into using WebClient instead, but that requires a parameter of the entire URL of the file to download, which is not known. I have to use HttpWebRequest, as far as I can tell, to access the USPS EPF files at all.

This is the getEpfFile method from the Client.cs class provided by USPS (I apologize for any formatting issues, this is my first post on the site):

// get actual file

    public bool getEpfFile(String fileid)
    {
        bool downloadSuccess = true;
        string strUrl = this.strBaseUrl + "/download/epf";

        try
        {

            Console.WriteLine("Starting file download ...");

            // add json to URL
            Dictionary<string, string> json_value = new Dictionary<string, string>();
            json_value.Add("logonkey", this.logon_key);
            json_value.Add("tokenkey", this.token_key);
            json_value.Add("fileid", fileid);

            JavaScriptSerializer jsonSerializer = new JavaScriptSerializer();
            string json_string = "obj=" + jsonSerializer.Serialize(json_value);

            System.Text.ASCIIEncoding encoding = new System.Text.ASCIIEncoding();
            Byte[] byteArray = encoding.GetBytes(json_string);

            // set URL              
            Uri address = new Uri(strUrl);

            // web request  
            HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;

            request.UserAgent = "USPS .NET Sample";
            request.KeepAlive = false;
            request.Timeout = 100000;

            request.ProtocolVersion = HttpVersion.Version10;

            request.Method = "POST";
            request.ContentLength = byteArray.Length;
            request.ContentType = "application/x-www-form-urlencoded";

            // add headers
            // request.Headers.Add("Akamai-File-Request", filepath);
            request.Headers.Add("logonkey", this.logon_key);
            request.Headers.Add("tokenkey", this.token_key);
            request.Headers.Add("fileid", fileid);

            // post request
            Stream dataStream = request.GetRequestStream();
            dataStream.Write(byteArray, 0, byteArray.Length);
            dataStream.Close();

            // Get response  
            HttpWebResponse response = request.GetResponse() as HttpWebResponse;


            if (request.HaveResponse == true && response != null)
            {

                Stream remoteStream = response.GetResponseStream();

                Directory.CreateDirectory("c:\\Atemp");
                Stream localStream = File.Create("c:\\Atemp\\dd.tar");

                //byte[] buffer = new byte[2048];
                byte[] buffer = new byte[1000000];

                int bytesRead = 0;

                do
                {
                    // Read data (up to 1k) from the stream
                    bytesRead = remoteStream.Read(buffer, 0, buffer.Length);

                    // Write the data to the local file
                    localStream.Write(buffer, 0, bytesRead);

                } while (bytesRead > 0);


                int i = response.Headers.Count;


                for (int x = 0; x < i; x++)
                {

                    if (response.Headers.Keys[x].ToString() == "User-Tokenkey")
                    {
                        this.token_key = response.Headers[x].ToString();
                    }
                    else if (response.Headers.Keys[x].ToString() == "User-Logonkey")
                    {
                        this.logon_key = response.Headers[x].ToString();
                    }

                    else if (response.Headers.Keys[x].ToString() == "Service-Response")
                    {
                        Console.WriteLine("Web service result: " + response.Headers[x].ToString());
                    }
                    else if (response.Headers.Keys[x].ToString() == "Service-Messages")
                    {
                        Console.WriteLine("Resulting Messages: " + response.Headers[x].ToString());
                    }

                }

                // close resources
                localStream.Close();
                remoteStream.Close();
                response.Close();

                Console.WriteLine("File Download completed.");
            }
        }
        catch (Exception ex)
        {
            downloadSuccess = false;
            string str = ex.Message;
            str += "";
        }

        return downloadSuccess;
    }

Any insight as to why it keeps cutting off early would be massively appreciated.

Answer 1

Turned out that USPS had a Download Manager program the whole time which accomplishes what I'm trying to do 10 times easier. I was emailed the zip file directly by the USPS rep, but the documentation (including a download link) is available here if anyone needs it: https://ribbs.usps.gov/acs/documents/tech_guides/ACS_EPF_DownloadManagerTechnicalGuide.pdf

Answer 2

Some time ago I was ran into same problem turns out the file was actually on the AKAMAI server and not on the EPF server that whats causing the problem. It its more that 2 GB then its most chances are it is on the AKAMAI server so I am giving the code that worked for me as below,

 public static bool DownloadEPFByFileIDUsingNewList(ref USPSUserDTO obj, USPSProductsDTO productList)
    {
        USPSFileDTO userDTO = new USPSFileDTO();
        if (obj != null)
        {
            foreach (var product in productList.fileList)
            {
                product.productcode = "NCAW";
                product.productid = "NCL18H";
                downloadSelectedFile(product, ref obj);
            }
        }
        return true;
    }


     private static void downloadSelectedFile(USPSProductInfo uspsProductDownload, ref USPSUserDTO obj)
    {
        string serviceResponse = string.Empty;
        string serviceMessages = string.Empty;
        USPSProductInfo uspsProductInfo = uspsProductDownload;
        int downloadPercentage = 0;
        bool isAkamaiFile = IsAkamaiFile(uspsProductDownload.productcode); //true;

        try
        {
            string[] statusResponse = SetStatus(USPS_URL, VERSION, obj.logonkey, obj.tokenkey, "S", uspsProductInfo.fileid, obj.login, obj.pword);
            if (statusResponse != null)
            {
                obj.logonkey = statusResponse[0];
                obj.tokenkey = statusResponse[1];
            }
            Uri StatusURL = new Uri(USPS_URL + (isAkamaiFile ? "/download/file" : "/download/epf"));
            byte[] byteLength = new ASCIIEncoding().GetBytes("obj=" + new JavaScriptSerializer().Serialize((object)new Dictionary<string, string>()
            {
                {
                  "logonkey",
                  obj.logonkey
                },
                {
                  "tokenkey",
                  obj.tokenkey
                },
                {
                  "fileid",
                  uspsProductInfo.fileid
                }
            }));

            HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(StatusURL);
            httpWebRequest.UserAgent = "EPF Download Manager - " + VERSION;
            httpWebRequest.KeepAlive = false;
            httpWebRequest.Timeout = 100000;
            httpWebRequest.Method = METHOD_TYPE;
            httpWebRequest.ContentLength = (long)byteLength.Length;
            httpWebRequest.ContentType = CONTENT_TYPE;
            (httpWebRequest.Headers).Add("Akamai-File-Request", uspsProductInfo.filepath + uspsProductInfo.filename);
            (httpWebRequest.Headers).Add("logonkey", obj.logonkey);
            (httpWebRequest.Headers).Add("tokenkey", obj.tokenkey);
            (httpWebRequest.Headers).Add("fileid", uspsProductInfo.fileid);

            //TODO chage path 
            string path = @"C:\try\newZipFiles" + "\\" + uspsProductInfo.fulfilled;
            if (!Directory.Exists(path))
                Directory.CreateDirectory(path);

            string outputFilePath = path + "\\" + uspsProductInfo.filename;

            Stream stream = ((WebRequest)httpWebRequest).GetRequestStream();
            stream.Write(byteLength, 0, byteLength.Length);
            stream.Close();
            HttpWebResponse httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
            int headercount = httpWebResponse.Headers.Count;
            for (int headCount = 0; headCount < headercount; ++headCount)
            {
                if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "User-Tokenkey")
                {
                    string responseToken = (httpWebResponse.Headers)[headCount].ToString();
                    if (responseToken.Contains(","))
                    {
                        responseToken = responseToken.Replace(",", "").Trim();
                    }
                    obj.tokenkey = responseToken.Trim();
                }
                else if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "User-Logonkey")
                {
                    string responseLogonkey = ((httpWebResponse.Headers)[headCount]).ToString();
                    if (responseLogonkey.Contains(","))
                        responseLogonkey = responseLogonkey.Replace(",", "");
                    obj.logonkey = responseLogonkey.Trim();
                }
                else if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "Service-Response")
                    serviceResponse = ((httpWebResponse.Headers)[headCount]).ToString().Replace(",", "").Trim();
                else if ((httpWebResponse.Headers.Keys[headCount]).ToString() == "Service-Messages")
                    serviceMessages = ((httpWebResponse.Headers)[headCount]).ToString();
            }

            if (serviceResponse == "success")
            {
                long countr = 0;
                Stream streamResponse = httpWebResponse.GetResponseStream();
                Stream fileStream = (Stream)System.IO.File.Create(outputFilePath);
                long downloadedFileSize = 0L;
                long originalFileSize = long.Parse(uspsProductInfo.filesize);
                byte[] bufferSize = isAkamaiFile ? new byte[1024] : new byte[512];
                int bytesRead = streamResponse.Read(bufferSize, 0, bufferSize.Length);
                try
                {
                    while (bytesRead > 0)
                    {
                        countr++;
                        fileStream.Write(bufferSize, 0, bytesRead);
                        downloadedFileSize += (long)bytesRead;
                        downloadPercentage = (int)(downloadedFileSize * 100L / originalFileSize);
                        bytesRead = streamResponse.Read(bufferSize, 0, bufferSize.Length);
                        Console.Clear();
                        Console.WriteLine("Downloaded " + downloadedFileSize + " of " + originalFileSize + " bytes." + " " + downloadPercentage + "%" + "counter" + countr);
                    }
                }
                catch (Exception)
                {
                }

                fileStream.Close();
                streamResponse.Close();
                httpWebResponse.Close();

                if (downloadedFileSize == long.Parse(uspsProductInfo.filesize))
                {
                    string[] statusCompleted = SetStatus(USPS_URL, VERSION, obj.logonkey, obj.tokenkey, "C", uspsProductInfo.fileid, obj.login, obj.pword);
                    if (statusCompleted != null)
                    {
                        obj.logonkey = statusCompleted[0];
                        obj.tokenkey = statusCompleted[1];
                    }
                }

            }

            ExtractRarFile(outputFilePath, uspsProductInfo.fulfilled, path);


        }
        catch (Exception exception_0)
        {

        }
        finally
        {

        }

    }


   public static bool IsAkamaiFile(string productCodeIn)
    {
        return productCodeIn == "AISVR" || productCodeIn == "AMS" || (productCodeIn == "NCAW" || productCodeIn == "NCAWM") || productCodeIn == "NCAM";
    }

Check with your product code and try this it will work with some adjustment with your product code.Best Luck.

Answer 3

Your timeout is only set to a little over one minute...can you download that fast? Increase your timeout and try again. (beware it's in milliseconds)