(...) there are actually no drawbacks apart from the resulting bundle possibly depending on bundles that are only required for the test source.
This is the main reason why there are separate eclipse-test-plugin modules for tests in Tycho.
There are no scopes in the OSGi manifest, so the test dependencies would pollute the dependencies of your productive bundle. When you then install your bundle into Eclipse or an RCP, it may pull in test bundles that your productive code doesn't actually need. For most users, this is enough reason to split their code into a productive and a test bundle.
Another potential reason for separating the tests is when you want to explicitly test via the OSGi API of your productive bundle. In this case, you would not use a fragment for the test, which ensures that your tests can only load classes from your productive bundle according to the OSGi package visibility rules.