Testing confirms that indeed the row is chosen entirely at random.
With a simple data source:
value
0
1
2
3
the order of values chosen in a test I just ran was:
3
3
3
1
1
2
3
...etc
For an actual "shuffle" implementation you'd need to write your own WebTestPlugin or WebTestRequestPlugin.