The problem was because of a bug in the source code. I communicated with Nuget core team and it was confirmed as a recently introduced bug which was fixed but not pushed at the time. Per grenade suggestion, I could publish my packages through Nuget.exe and I could avoid this problem altogether (because Nuget.exe wasn't built from the problematic code). However, that was not an option for me because I needed to fix the UI so other users could take advantage of the UI.
I fixed the problem myself in the Nuget source code. It was more like a serialization problem. I am sure as of now it has been fixed in the source code by the Nuget core team.
Kudos to Microsoft Nuget team for amazing support and responsiveness.