You should NOT cache based on row only, you should cache based on both row and column.
Your code as it is now stores one value for row, instead of multiple values per row, and one for each (row,column) combination.
Your 'key' for the cache should be (row,column). It could be solved using a 2D matrix - though it might be more expansive (in terms of weight) than a Dictionary
.