I never could find an exact answer to this specific setup. My guess is that JGoodies dies bit handle nested layouts very well.
I ended up "fixing" this by using only one single panel for the entire locations scroll pane. This made the layout a bit complicated: Multiple cells now had to span columns and I had to manually adjust the column offset. But in the end it works