The type of square depends on the heights in each vertex of the grid, so what you want is an algorithm to generate the heights at each vertex. The rules in this case (transport tycoon) is that the four corners of one tile has at most a height difference of two steps, and in fact at most two vertices can have the 2 step difference. T
A common way of generating terrain is through simulated erosion, that is, you start with a random terrain and erode it by smoothing and simulated rivers etc. until you get what you need. Given the restrictions in the TT terrain, I doubt this is how it is made. Apart from the constraints of the slopes, the game also requires that the terrain is reasonably flat. The standard terrain generation of OpenTTD appears to be a repeated random heighteing/lowering of vertices.
You can view the terrain generation algorithm at openttd.org
http://svn.openttd.org/trunk/src/landscape.cpp
(Search for "GenerateLandscape").
This open source version supports several generation methods, both erosion based ("Terragenesis") and randomized.