The HTML5 standard specifies two serializations of HTML5: "html" and "xml". "xml" is a valid XML serialization (which in turn is a subset of SGML). "html" is not based on any specific serialization standard anymore, it has its own complete serialization. Herein lies the difference: HTML4 has a "sgml" serialization and "xml" serialization (called XHTML 1.0)
Of course HTML5 is for a large part based on HTML4 (based on SGML) and XHTML (based on HTML4 and XML).
Also see the history section of the HTML5 specification