This is a side effect of using the fillViewPort attribute on your ScrollView. If you have set fillViewPort to true, then this will only be taken into account if the measured height of your child is lower than the height of your ScrollView. In that case the ScrollView will stretch the content to fill the viewport and your margins won't be applied correctly. As soon as your child's content is greater than the height of your ScrollView, your margins will be applied correctly.
I suggest to not use margins on your child layout (LinearLayout) that you have placed in the ScrollView, but only use padding.
So your Container style could become
<style name="Container">
<item name="android:paddingTop">32dp</item>
<item name="android:paddingLeft">164dp</item>
<item name="android:paddingRight">164dp</item>
<item name="android:layout_width">match_parent</item>
<item name="android:layout_height">match_parent</item>
<item name="android:orientation">vertical</item>
</style>