There's no issue referencing an element of the same type as part of the type definition, so your first example is fine from that point of view. Trying to reference the parent is a bit odd though, you shouldn't really need to do this... XML is hierarchical after all.
<xs:complexType name="Category">
<xs:sequence>
<xs:element maxOccurs="unbounded" minOccurs="0" name="ChildCategory" type="Category"/>
</xs:sequence>
<xs:attribute name="CategoryName" type="xs:string"/>
</xs:complexType>
The Category type references itself recursively, allowing for 0 or more ChildCategory elements. This should do what you need (there's nothing wrong with recursive type referencing in the XML Schema).
If you need to refer to the parent Category in your document, it's easy enough to chain to the parent node in any DOM implementation or with XPath.