After ATTLIST declaration in DTD browser renders custom character

Question 1

You should not use an internal subset in a doctype declaration, because browsers do not understand it, or DTDs at all.

If you use a simple added attribute, for some reason, it is often best to just be careful enough with it, or “check it manually”. But to perform DTD-based validation, you would need to construct an external DTD, based on the DTD you wish to use as basis, and with the extra markup added into it. In this case, you would copy the HTML 4.01 Transitional DTD and replace

<!ATTLIST LI
  %attrs;>

by

<!ATTLIST LI
  rel CDATA #IMPLIED
  %attrs;>

(That is, you need to provide the full list of allowed attributes, with your custom attribute added, instead of declarign an attribute list that only allows your attribute [unless that’s what you really want].)

You would then use a doctype declaration that refers to your modified copy by its URL, with

<!DOCTYPE HTML SYSTEM "dtdurl">

where dtdurl is an absolute URL for the DTD. More info: Creating your own DTD for HTML validation.

It is generally not advisable to add attributes of your own, as they may clash with attributes that might be added to HTML in some future version. According to HTML5 drafts, attributes with names starting with data- are meant for site-specific use and will never have any publicly defined meaning, so data-rel would be safer than rel.

Question 2

Browsers don't understand embedded SGML. They simply stop reading the doctype at the first > character. So they see the following ]> as text to be rendered.

Just don't use embedded SGML.

Question 3

Use a pseudo-attribute > delimiter instead of a literal > delimiter to escape the nested > within ]>:

<!DOCTYPE HTML PUBLIC 
"-//W3C//DTD HTML 4.01 Transitional//EN" 
"http://www.w3.org/TR/html4/loose.dtd" 
[<!ATTLIST li rel CDATA #IMPLIED&gt;]>

References

Pseudo-Attributes