Note that the DITA Open Toolkit does not have the stated limitation and your first code would work fine.
However, you can use the @collection-type attribute outside of the context of a relationship table. This should work even with the DITA Converter tool to the extent that the tool supports the DITA spec.
You do not state whether your deliverable is online help topics or a PDF. If you are generating help topics, you could include this code in your navigation map:
<topicgroup collection-type="family" toc="no">
<topicref href="topics/a.dita" type="concept"/>
<topicref href="topics/b.dita" type="concept"/>
<topicref href="topics/c.dita" type="concept"/>
</topicgroup>
The result is that the links among the topics would be created, but a new instance of the topics would not be listed in the navigation.
Of course, if these three topics already are the only children of a parent node in the navigation, you could simply add the @collection-type attribute to the parent node.