After doing a lot more testing and trying I've found a workaround/solution for the problem.
The row.Cells(4).Text was a boundfield. I've changed that to a TemplateField with a label. By making a local variable (Label) and using that to convert the decimal from it works. I got no idea why but for me it's a solution for now.