The parent of your CustomControl will still be TabItem. Try below code:
Clears the content of TabItem:
TabItem tabItem = this.Parent as TabItem;
tabItem.Content = null;
If you want to remove the TabItem from TabControl then:
((TabControl)tabItem.Parent).Items.Remove(tabItem);