A rectangle can be defined by two points representing the opposing corners, eg: A(x,y) and B(x,y). If you have a point C(x,y) that you want to test to see if it is inside the rectangle then:
IF( (Cx BETWEEN Ax AND Bx) AND (Cy BETWEEN Ay AND By) ) THEN
point C is in the rectangle defined by points A and B
ELSE
nope
ENDIF
A circle can be defined by a single point C(x,y) and a radius R. If the distance D between the center and the point P(x,y) is less than the radius R, then it is inside the circle:
And of course you remember the Pythagorean Theoreom, right?
C² = A² + B² SO C = SQRT(A² + B²)
So:
D = SQRT( ABS(Cx - Px)² + ABS(Cy - Py)²)
IF( D <= R ) THEN
point P is inside the circle with center C and radius R
ELSE
nope
ENDIF
edit:
The algorithm for checking if a point is within a polygon is a bit more complex than what I'd prefer to write in a SQL query or stored procedure, but it is entirely possible. It's worth noting that it runs in constant-time and is very lightweight. [requires roughly 6 arithmetic ops and maybe 2 or 3 logic ops for each point in the poly]
To pare down the number calculations required you can simply write your select to get points within a rough bounding box before procesing them further:
WHERE
x BETWEEN MIN(x1,x2,x3,x4) AND MAX(x1,x2,x3,x4)
AND
y BETWEEN MIN(y1,y2,y3,y4) AND MAX(y1,y2,y3,y4)
Assuming the columns containing the x and y values are indexed this might use a few less CPU cycles than simply doing the math, but it's debatable and I'm inclined to call it a wash.
As for the circle you can't possibly get more efficient than
WHERE
SQRT( POW(ABS($Cx - x),2) + POW(ABS($Cy - y),2) ) < $radius
You're far too concerned with the perceived cost of these calculations, just write the code and get it working. This is not the stage to be performing such niggling optimizations.