I found the answer already on my own.
It was not my issue i had all the time. It's a issue the AndroidStaggeredGrid currently has. There's already a merged Pull-Request with a fix but it was no uploaded to maven yet.
I refer to those two commits:
https://github.com/etsy/AndroidStaggeredGrid/commit/18e9ae8251ff30ae29125613c8fd909e5e8c27a6
https://github.com/etsy/AndroidStaggeredGrid/commit/2fce475c197bb18871d2a1076e259a57d8513104
For now the solution is to build the library on your own and publish it to a private maven repo, or you include the project as library project!