Then you're going to bite the bullet and stop relying on the GC to do your resource management for you. You're going to have to have your asset manager have an explicit function to delete the objects it allocates, rather than relying on the asset manager's finalizer function. And you're going to have to call that function at a specific place in your code.
Just because you have GC doesn't mean it's the best or only solution.