Ok got it ...
Actually, falling on that post and that one, it appears that it is Azure misleading between the connection string names both given the same in Azure Sql Server and the Azure Web Site.
It has nothing to do with a whatsoever bad EF implementation between code first/model first or db first ...
So I gave connection string name a new one in the web App config, Context Class constructor, and in the azure web sites connection string name parameters ... And it worked.
It was a real pain figuring this out ...