Since my guess in the comments was correct, posting an actual answer for more visibility in case someone has a similar problem:
delete-overlay
may not be doing what you expect. From the manual:
— Function: delete-overlay overlay
This function deletes overlay. The overlay continues to exist as a Lisp object, and its property list is unchanged, but it ceases to be attached to the buffer it belonged to, and ceases to have any effect on display.
A deleted overlay is not permanently disconnected. You can give it a position in a buffer again by calling move-overlay.
Maybe you have a giant pile of disconnected overlays eating up memory or causing a lot of processing.