The reason this happened is that in your original code
<div class="col-xs-12 col-sm-8 col-md-8 contact-area">larger column</div>
<div class="col-xs-12 col-sm-4 col-md-4 social-area">smaller column</div>
you were specifying that each column on extra small screens and UP should be the full size 12. This caused them to stack since both columns can't be full width.
When you remove the "col-xs-12" your code is specifying that on small screens and UP, you would like one column to be 3/4 width and the other to be 1/4, sizes 8 and 4 respectively. And on screens smaller than small you would like the columns to stack.
You actually don't need to specify anything for lg screens since you're specifying the same sizes.