Web Application property bag : returned value is wrong - SharePoint 2016

Question

We are currently encountering a strange problem with some values stored in the property bag of our Web Applications that we cannot explain.

Basically, we have two Web Applications, one for the main "Intranet" part and a second that is used for the "My Site" part. In both Web Applications, the different collections have the same main navigation bar with links pointing to some places in the main Web Application. To achieve this, we stored the main URL in the property bag of each Web Application. So it means that we have something like this:

Web Application 1 - Properties
------------------------------
MainHostUrl : https://mainurl.com
MySiteHostUrl : https//mysiteurl.com

Web Application 2 - Properties
------------------------------
MainHostUrl : https://mainurl.com
MySiteHostUrl : https//mysiteurl.com

With this, we access this value in various ways. For example:

string rootUrl = SPContext.Current.Site.RootWeb.Site.WebApplication.Properties["MainHostUrl"].ToString();

// OR

using (var site = new SPSite(url))
{
    SPWebApplication app = site.WebApplication;
    Hashtable properties = app.Properties;
}

// etc

This concept was deployed and worked fine until a few days ago. For a reason we can't figure out, when the property "MainHostUrl" is accessed, it returns the wrong value in the second Web Application (My Site), but only in our production environment.

We checked several times the values stored in the property bag and they are correct. We tried to delete the properties and to recreate them, but it didn't change anything. We then thought it was related to the Blob Cache or the Distributed Cache, so we flushed the Blob Cache on our different Front Servers and completely rebuilt the Distributed Cache Service on each of them. We even restarted the whole infrastructure (Front Servers, Application Servers, Database Servers). We did several "iisreset" and recycled the Application Pools without any success.

When we look at the rendered HTML, we can see that the value stored in the "href" attribute is wrong. It is href="https//mysiteurl.com" while it should be href="https://mainurl.com".

What it is also surprising, is we have some links stored in the Term Store, that is shared by both Web Application, that point to the main Web Application that are correctly referenced but return the wrong value. For example, we have "https://mainurl.com/link-stored-in-term-store" in the Term Store but it returns "https://mysiteurl.com/link-stored-in-term-store".

We also spotted another thing: we have an .ashx file with a .cs as Web Handler. This .ashx is called through JavaScript and we can see in the console that the response is incorrect. It is

{
    "webAppProperties": 
    {
        "MainHostUrl" : "https//mysiteurl.com"
    }
}

...but it should be

{
    "webAppProperties": 
    {
        "MainHostUrl" : "https://mainurl.com"
    }
}

We already tried to change and refresh the values stored in the property bag. We even deleted them and restored them. It works fine in our development and QA environment, but not in our production environment. It has absolutely no effect and the package deployed is the same. If we change the URL in the Term Store, the change is also not reflected in the second Web Application. The problem is likely related to the environment.

When we check the properties with PowerShell, we clearly see that the properties have the right values.

We have the same version of each component in our different environments. We always take care to update the development and the QA servers before the production environment. If it was related to an update, we would have experienced this problem in other environments before. The Web Applications, except the name, have the same configuration. What really differs is we have only one server for the development, two front servers, one application server and one database server for the QA while we have four front servers, two database servers and two applications servers and one Office server for the production. Two of the front servers are for internal access while the others are for the connections from the outside. The "channels" between the different front servers are open and they can see each other. The services started are the same for the different front servers and we use the same accounts. The configuration of the two application servers is also the same. The different web.config files haven't been touched since we went into production. We noticed that we encounter the same behaviour regardless we access the infrastructure from the inside or from the outside. Now, maybe someone has changed something without telling us, but the question is where and what?

When a property with a value of "https://mainurl.com" is called in the second Web Appplication (from the WebApplication property bag or from the Term Store), it returns "https://mysiteurl.com", like if the root of the URL was systematically replaced (regardless if there is something after or not).

We also checked if a rewrite rule was changed in the reverse proxy or somewhere else (server aliases, load balancer), but everything seems to be alright. There is also no rewrite rules in IIS. AAMs are also correctly configured.

We tried many things for hours to understand why it is not working anymore, but without any success. Has anyone encountered such a behaviour? Any suggestion would much be appreciated.

Best regards,

SOLUTION

As suggested in the comments by Anon, the problem was in the Reverse Proxy, but not where we thought. We had to look another time to find the problem.

Answer 1

As posted by the enquirer Lancelot, this comment led to the solution, as actually a configuration in the reverse proxy was the reason for the misbehaviour:

Do you have knowledge of the rest of the infrastructure? I'm currently thinking of something like a reverse proxy between you and the server that (wrongfully) rewrites the URLs in the content that it returns based on rules that someone created.

(original comment from 2019-02-08)