Reference request: Introduction to reinforcement learning with hand calculation examples

Question

For me, the most difficulty when it comes to learning about reinforcement learning is that there is not much to learn in the sense that without running some algorithm, it is very difficult to get a sense of how abstract RL framework is done in practice.

This is in contrast to a lot of the stuff you learn in related courses on optimization, game theory or even pure algorithm. In each of these courses, you have a large amount of examples where you can hand compute to obtain the solution (for algorithms, you can solve for shortest path etc by hand). It is also easy to compute the gradient, and calculate the value at each step using gradient descent algorithm for example in an optimization class, or compute the Nash equilibria in simple games. This allows one to better appreciate how the algorithms work.

Even for supervised machine learning, algorithms such as ANN, KNN, SVM etc. can be analyzed by hand, using step by step calculation. However, this does not seem to be the case for reinforcement learning. Every article I have came across basically runs through the whole "state, action, reward" framework then goes on talking about MDP and Q, TD learning. Then applications that usually follows are not something that are easily programmed on a first try i.e. robotic acrobat, pole cart system etc.

Are there any introduction or tutorial to reinforcement learning where a problem is solved by hand? Meaning, you can actually compute the values at each step following a particular technique and perhaps obtain the optimal solution.

Even if not, are there any mathematical convergence analysis of reinforcement learning algorithms?