You need to get a window first then you can open new tab in it.
This example here gets the most recent navigator window and adds a tab to it:
const {classes: Cc, Constructor: CC, interfaces: Ci, utils: Cu, results: Cr, manager: Cm} = Components;
Cu.import('resource://gre/modules/Services.jsm');
var aDOMWindow = Services.wm.getMostRecentWindow('navigator:browser');
aDOMWindow.BrowserOpenTab()
another way to do this is use gBrowser.loadOneTab: so:
aDOMWindow.gBrowser.loadOneTab(aDOMWindow.BROWSER_NEW_TAB_URL, {relatedToCurrent:true});
this opens the tab next to the currently focused tab. more info on it here: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Method/loadOneTab
There are these other useful functions:
OpenBrowserWindow