Mapping Rate to ProductRate is fairly straight forward with "ForMember"
The one where you have to do a cast to the specific type to see if it is that type is a little trickier but I think the same approach you took is what you might have to do however I don't think you need to do "aftermap". I thought all your destination mappings had to be found OR you need to mark them as ignore of the mapping will fail.
Another thing you could do is just change the OrderEntity.ProductRate to be OrderEntity.Rate. Then it would find it and map it for you except where it was hidden because Product doesn't have a rate (but RatedProducts do).
public class OrderEntity {
public decimal Amount {get;set;}
public int Rate {get;set;} //changed name from ProductRate to just Rate.
}
Mapper.CreateMap<Product, OrderEntity>()
.Include<RatedProduct, OrderEntry>();
Mapper.CreateMap<RatedProduct, OrderEntry>();