Floating-point arithmetic (and integer arithmetic and fixed-point arithmetic) has a certain granularity: Values can only change by a certain step size. For IEEE-754 64-bit binary format, that step size is about 2–52 times the value (about 2.22•10–16). That is very small for physical measurements.
However, when you make h very small, the difference between f(x) and f(x+h) is not very large compared to the step size. The difference can only be an integral multiple of the step size.
When the derivative is d, the change in f(x) is about h•d. Even if you calculate f(x) and f(x+h) as well as possible in the floating-point format, the measured value of their difference must be a multiple of the step size s, so it must be round(h•d/s)•s, where round(y) is y rounded to the nearest integer. Clearly, as you make h smaller, h•d/s is smaller, so the effect of rounding it to an integer is relatively larger.
Another way of looking at this is that, for a given f(x), there is a certain amount of error in calculating values around f(x). As you make h smaller, f(x+h)–f(x) gets smaller, but the error stays the same. Therefore, the error increases relative to h.