The calculation you are performing is correct if you would like to perform a rotation around the x axis clockwise. If you think of your line as a sheet of paper, a rotation of 0 degrees is you looking directly at the line.
For the example you have given the line is horizontal to the x axis. This will not change on rotation around the x axis (the line and the axis around which it is rotating are parallel to one another). As you rotate between 0 and 90 degrees the y co-ordinates of the line will decrease with P1.y*cos(theta) down to 0 at 90 degrees (think about the piece of paper we have been rotating around it's bottom edge, the x axis, at 90 degrees the paper is flat, and the y axis is perpendicular to the page, thus both edges of the page have the same y co-ordinate, both the side that is the "x-axis" and the opposite parallel side will have y=0).
Thus as you can see for your example this has worked correctly.
EDIT: The reason that multiplying by 90 degrees does not give an exactly zero answer is simply floating point rounding