I found out the problem. I need to put a Window inside a tab element and then add the tabBarHidden
property. So my view XML file looks like similar to this:
<TabGroup id="mainTabGroup">
<Tab>
<Window id="mainWindow" title="The Title" tabBarHidden="true">
</Window>
</Tab>
</TabGroup>
And in my controller:
$.mainTabGroup.open();