So, it turns out I had it set up all wrong. As this article from apple says:
Before you start building custom layouts, consider whether doing so is really necessary.
The UICollectionViewFlowLayout class provides a significant amount of behavior that has already been optimized for efficiency and that can be adapted in several ways to achieve many different types of standard layouts. The only times to consider implementing a custom layout are in the following situations:
The layout you want looks nothing like a grid or a line-based breaking layout (a layout in which items are placed into a row until it’s full, then continue on to the next line until all items are placed) or necessitates scrolling in more than one direction.
You want to change all of the cell positions frequently enough that it would be more work to modify the existing flow layout than to create a custom layout.
And my layout config is really simple. So I was overcomplicating it.