502 bad gateway azure app service + provider hosted add-in

Question

I have a provider hosted add-in deployed to azure web application (app service).

I have a long running task, iterate through all site collections (including sub webs) in tenant and perform some task in them.

Firstly I created an aspx page and written all the logic in button click in code behind, it works for 30-40 site collections with 5-10 sub-webs but throws 502 bad gateway when number of site collections increases.

I came to know about ARR request timeout which is set for 3 minutes and 50 seconds, and for long running tasks we have to make the process async.

So I tried to convert whole process into ajax call using web method (moved the long running code to web method), still getting bad gateway error after 4 minutes:

502 - Web server received an invalid response while acting as a gateway or proxy server.
There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.

Ajax code:

$.ajax({  
                type: "POST",  
                url: "testpage.aspx/mywebmethod",  
                contentType: "application/json; charset=utf-8",
                dataType: "json",
                async: true,
                data: JSON.stringify({ 'sitecolllist': $('#' + '<%= txtSiteCollectionList.ClientID%>').val(), 'tenantadminURL': $('#' + '<%= txtTenantAdminURL.ClientID%>').val(), 'tenantAppCatURL': spHostURL }),
                beforeSend: function() {
                    //show loader
                },
                success: function (data) {
                alert('success: ' + data.d);  
                },
                error: function (XMLHttpRequest, textStatus, errorThrown) { 
                    alert("Status: " + textStatus); alert("Error: " + errorThrown); 
                },
                complete: function() {
                    //hide loader
                }
        });

Code behind web method:

        [WebMethod]
    public static string mywebmethod(string sitecolllist,string tenantadminURL,string tenantAppCatURL)
    {
    //long running code here
    //Fetch all site collection object properties using tenant.GetSitePropertiesFromSharePointByFilters
    //create client context using site coll url and perform actions
    //call recursive method to iterate through all sub sites and perform actions
    }

Next idea got from googling is create an c# async task, move the long running code inside async task, continuously calls the async task via multiple ajax calls until the task complete.

Return "202 accepted" if task is in progress, 200 when task completes.

Not sure if this will resolve the issue, Can anyone have code snippet or blog reference achieve the above idea?

How to write async task which returns string?

How to call async task from web method?

how to make multiple ajax calls until tasks complete?

Answer 1

used continuous azure webjobs to perform the long running operation. Used azure queue trigger to trigger the webjob whenever there is a new message in queue.

Creating the queue in web method and passing require parameters there.