Does Flyway roll back the transaction?
Yes, it will rollback the transaction until the latest well executed script or latest commit done. Then it will log the Exception and notify you (depending how you configure it).
Does it stop doing further migrations?
AFAIK that's the common behavior, it will stop if it encounters an error.