Concurrently fill a dictionary with HttpClient PostAsJsonAsync extension

Question

I have a .net site in which I'm trying to get the results of several web service calls, to be shared among several dropdownlist elements, in a parallel fashion. My problem is all of the dropdowns end up having the same values, or some have the same values with one having different values (probably not the correct ones). How can I fix this to get these things in parallel?

Code Updated:

using (HttpClient hc = new HttpClient())
{
    hc.BaseAddress = new Uri(CatalogUri);
    hc.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/jsonp"));
    // request for standard options
    HttpResponseMessage stdResponse = hc.PostAsJsonAsync(CatalogSearchOptionPath, searchmeta).Result;
    List<string> keynames = {"Key3", "Key2","Key1"};
    ConcurrentDictionary<string, List<string>> customOptions = new ConcurrentDictionary<string, List<string>>();

    IEnumerable<Task<KeyValuePair<string, List<string>>>> tasks = from key in keynames select GetCustomOptionList(key, hc, searchmeta);

    customOptions = new ConcurrentDictionary<string, List<string>>(await Task.WhenAll(tasks));

    if (stdResponse.IsSuccessStatusCode)
    {
        string g = stdResponse.Content.ReadAsStringAsync().Result;
        stdOptions = Newtonsoft.Json.JsonConvert.DeserializeObject<List<SearchOption>>(g);
        //options = response.Content.ReadAsAsync<SearchOption[]>().Result.ToList();
    }
}

Async method for doing the requests:

private async Task<KeyValuePair<string, List<string>>> GetCustomOptionList(string key, HttpClient client, SearchMetadata sm)
{
    sm.OptionFieldName = key;
    var response = await client.PostAsJsonAsync(CatalogSpecificOptionPath, sm);
    var result = await response.Content.ReadAsStringAsync();
    return new KeyValuePair<string, List<string>>(key, Newtonsoft.Json.JsonConvert.DeserializeObject<List<string>>(result));
}// end task

Answer 1

This is the way I have gotten this to work so all my dropdowns have the correct values. It FEELS like I'm doing something wrong with it, but it works and I'm on a deadline so it will have to do for now. Maybe later I'll be able to get a better solution. Maybe someone on StackOverflow will provide a better solution - we'll see!

/// Do Stuff in Async method
using (HttpClient hc = new HttpClient())
{
    hc.BaseAddress = CatalogUri;
    hc.DefaultRequestHeaders.Accept.Add(HttpJsonHeader);

    if (Session["StandardSearchOptions"] == null || rebindStandardOptions)
    {
        // request for standard options
        HttpResponseMessage stdResponse = hc.PostAsJsonAsync(CatalogSearchOptionPath,
                                                 searchmeta).Result;

        IEnumerable<Task<KeyValuePair<string, List<string>>>> tasks = 
                               from key in keynames 
                               select GetCustomOptionList(key, hc, searchmeta);

        customOptions = new ConcurrentDictionary<string, List<string>>(await
                                                    Task.WhenAll(tasks));

        if (stdResponse.IsSuccessStatusCode)
        {
            string g = stdResponse.Content.ReadAsStringAsync().Result;
            stdOptions = Newtonsoft.Json
                          .JsonConvert.DeserializeObject<List<SearchOption>>(g);
        }
        Session["StandardSearchOptions"] = stdOptions;
    }
    else // only rebinding the custom options
    {

        IEnumerable<Task<KeyValuePair<string, List<string>>>> tasks = 
                                                  from key in keynames 
                                                  select GetCustomOptionList(key, hc,
                                                                         searchmeta);

        customOptions = new ConcurrentDictionary<string, List<string>>(await 
                                                              Task.WhenAll(tasks));
    }
}
/// Do other stuff in Async Method

Async method for doing the individual custom searches:

private async Task<KeyValuePair<string, List<string>>> GetCustomOptionList(string key,
                                    HttpClient client, SearchMetadata sm)
{
    sm.OptionFieldName = key;
    var response = await HttpClientExtensions
                      .PostAsJsonAsync(client, CatalogSpecificOptionPath, sm)
                      .Result.Content.ReadAsStringAsync();
    return new KeyValuePair<string, List<string>>(key,
                Newtonsoft.Json.JsonConvert
                            .DeserializeObject<List<string>>(response));
}// end task

Answer 2

One reason why this could be happening is the fact that there is one HttpClient instantiated on the outside of the loop. Since the loop then uses HttpClientExtensions.PostAsJsonAsync which takes in the hc variable, then it's possible that the repeated values for the keys in the ConcurrentDictionary is the result of response being overwritten as the first call to return from the HttpClient within all of the loop iterations. This would depend on the implementation of PostAsJsonAsync, but one easy way to test would be to instantiate a new HttpClient in the loop and see if that fixes it.

Edit: Beware of using .Result, though it may not be the cause of your issue.

“Async all the way” means that you shouldn’t mix synchronous and asynchronous code without carefully considering the consequences. In particular, it’s usually a bad idea to block on async code by calling Task.Wait or Task.Result. This is an especially common problem for programmers who are “dipping their toes” into asynchronous programming, converting just a small part of their application and wrapping it in a synchronous API so the rest of the application is isolated from the changes.

source

Lastly, if you end up achieving only a 2x speedup when this is all working, consider tweaking this web.config tag:

Sytem.NET MaxConnection property