There are 4 scenarios. If you:
- return -1 in your callback fixture then the fixture is going to be ignored and the ray will continue
- return 0 the ray will terminate
- return fraction then the ray will stop at this point
- return 1 then the ray will continue until it ends
The first scenario is used when you want to ignore certain fixtures. Second scenario is used when you want to know if your ray hit anything (not necessarily the closest object). Third scenario is used to find the nearest object. And the fourth scenario is used when you wish to know about all objects in the ray's path.
In general you callback logic would have a part that checks for objects that should be ignored and return -1 if they should be ignored which will be followed by a code block that either returns 0, fraction or 1.
The difference between second and third scenario is possibly the hardest to understand. The way I understand it is that if you return 0 none of other potential hits are going to be evaluated. But if you return a fraction (which will clip the ray) other potential hits within the new (clipped) ray length are going to be evaluated - in other words your callback will be executed for all of them and the ray might be clipped even further which will eventually return the nearest object.
See Box2D documentation about Ray Casting here.
One thing that documentation does say is that the order of evaluation is not guaranteed. In your case it might be that the player was being evaluated before the wall blocks.
tl; dr; return fractions as you did in your second code sample.