Drupal Commerce, Can't add to cart without HTTPS enabled

Question

I have a quite weird situation here, I assume this problem must be a server side issue, but so far I couldn't figure it out how to fix this.

I have a Nginx server with Varnish (I even disabled it totally still got the error) running a Drupal Commerce website on it, the quite funny issue here is the "Add to Cart" button doesn't work on a session without HTTPS, if I add HTTPS in front of my URL, I can add the product to my cart!

I'd really appreciate any help!