UPDATE:
Using the new RecyclerView and the StaggeredGridViewLayoutManager you can achieve this now without third-party libs.
OLD ANSWER:
I found out that this kind of View was already done by Google under the name of StaggeredGridView. It was once in the AOSP, but was taken out again (I lost the source for this info...but anyway). But you can find a modified version of StaggeredGridView here:
The original Google source is here:
And another very good implementation of it is found in the official Etsy App. The code was made public: