When you allocate faster than you can garbage collect you will run into OOM. If you do heavy allocations the CLR will insert a Sleep(xx) to throttle allocation but this is not enough in your extreme case.
When you implement a finalizer your object is added to the finalization queue and when it was finalized it is removed from the queue. This does impose additional overhead and you will make your object life longer than necessary. Even if your object could be freed during a cheap Gen 0 GC it is still referenced by the finalization queue. When a full GC is happening the CLR does trigger the finalizaion thread to start cleaning up. This does not help since you do allocate faster than you can finalize (writing to stdout is very slow) and your finalization queue will become bigger and bigger leading to slower and slower finalization times.
I have not measured it but I think even an empty finalizer will cause this issue since the increased object lifetime and two finalization queue handling (finalizer queue and f-reachable queue) do impose enough overhead to make finalization slower than allocation.
You need to remember that finalization is an inherent asynchronous operation with no execution guarantees at a specific point of time. The CLR will never wait to clean all pending finalizers before allowing additional allocations. If you allocate on 10 threads there will still be one finalizer thread cleaning up after you. If you want to rely on deterministic finalization you will need to wait by calling GC.WaitForPendingFinalizers() but this will bring your performance to a grinding halt.
Your OOM is therefore expected.