I have not done what you're suggesting with the floating windows, but I have managed to integrate Caliburn.Micro with AvalonDock. You'll need a couple of examples to help you and documentation.
- Download AvalonDock Source Code
- Download Gemini Source Code (Mash up of Caliburn.Micro and AvalonDock).
In short, use the LayoutInitializer in conjuction with a property on your ToolViewModelBase to indicate that the anchorable should be opened as floating.
<avalonDock:DockingManager.LayoutUpdateStrategy>
<local:LayoutInitializer/>
</avalonDock:DockingManager.LayoutUpdateStrategy>
The AvalonDock source code of interest is below and you'll find equivalents in Gemini.
.\Version2.0\AvalonDock.MVVMTestApp\ToolViewModel.cs
.\Version2.0\AvalonDock.MVVMTestApp\LayoutInitializer.cs