Solved the problem with avinash help. I ended up changing the following in my CSS. Since I have all the content inside the container div I made container height 100% - footer. I also removed bottom:0px from footer.
footer{
position: relative;
}
html,body {
height: 100%; /* Needed for container's min-height */
}
.container{
min-height:100%;
margin-bottom: -40px; /* Put negative height of the footer here */
padding-bottom: 40px; /* Put height of the footer here. Needed for higher than screen height pages */
}