You almost had it, instead of .CustomType
you'll have to define .CustomSqlType
Map(x => x.BirthDate)
.Column("dtBirthDate")
.Not.Nullable()
.CustomSqlType("smalldatetime")
.CustomType("datetime")
Just tested it and it will create a database column with smalldatetime.