You can do this by creating Projections
that call SQL functions. In your case there are nested functions, so there will be multiple building steps.
Here is a draft solution. I haven't tested it, but it gives the idea of the approach:
IProjection nameField = Projections.Property<User>(u => u.Name);
IProjection charIndex = Projections.SqlFunction("CHARINDEX",
NHibernateUtil.String,
nameField,
Projections.Constant(" ", NHibernateUtil.String)
);
IProjection subString = Projections.SqlFunction("SUBSTRING",
nHibernateUtil.String,
nameField,
charIndex,
Projections.Constant(999, NHibernateUtil.UInt32);
var query = session.QueryOver<User>().OrderBy(subString).Asc;
Unfortunately, this solution would not work if there are spaces in the first name. (Jane A. Doe). I'm not aware of an SQL function to find the last space in a string.
See this discussion: Find index of last occurrence of a sub-string using T-SQL