Is authenticating a TCP connection by source IP safe?

Question

I'm developing an application that accepts connections from clients over the internet. All these clients are on fixed IP addresses and will establish a non-encrypted TCP connection.

The current plan is for the server to check which IP the connections come from and allow only client connections from a list of know IPs?

How safe is that against IP spoofing?

My reasoning is that since this is a TCP connection, an attacker couldn't just fake its sender IP (which is easy), but would have to assure that the packets travel back to him and thus he would have to hack all routers on the path, which seems rather hard.

I know I could use encryption, like SSH, but lets stick with the question of how safe the plain TCP connection would be.