As far as I can tell there is no proper way. So what I ended up doing was creating a JPanel for each field with a BoderLayout. Then I added the max number of fields I would ever need to the form. Then when I needed to move fields down, I would remove them from the allocated JPanel's and push them to the ones below in a loop.
Not exactly the prettiest or most elegant but I couldn't find another solution at all...