Y here is a fixed-point combinator. Given a function f, it returns a value x for which x = f(x). When, instead of writing x, we write Y(f), we have Y(f) = f(Y(f)). In the traditional lambda calculus notation, this is (F (Y F)) = (Y F), which is what you see in the image.
Some numerical functions have one or more fixed points. E.g., fixed point of the (positive) square root function, as well as the square function are 1 and 0.. Some numerical functions, e.g., f(x) → x+1 doesn't have a fixed point. In some formalisms, including the untyped lambda calculus, every function has a fixed point.
This particular fixed-point operator is the Y-combinator, and is described in more detail in various places, including the Wikipedia article linked above. Fixed point operators are important because, among other things, they allow recursive functions to be defined in formalisms such as the untyped lambda calculus.