I found out the problem wasn't in the actual reading from the TCPClient. The problem was that my ISP closed the connection after 30 minutes of inactivity in the channel. Since IRC ping-pong is a continuous request-reply from the internet to the (bot) user. My ISP probably sees this kind of request-reply activity as a potential botnet and closes the connection.
To prevent this, I changed my bot to send PINGs to the server every minute. With this change in place, the bot has been stable for a full week (24/7).