I managed to find out that in the function that calls the WebAPI using AJAX, we first disable the onbeforeunload event. Still, this does not seem to get called when CodedUI clicks the button. I've managed to work around this by using
BrowserWindow.ExecuteScript("window.onbeforeunload = null;");
However, I still can't figure out why I need to do this...