A TCP connection requires more packets to provide the reliable transport of the data. TCP is not meant for low latency, it is meant for reliably transmitting a stream of data. For example, if you are sending a file, you need all the packets to be received and to be pieced together in the correct order.
You are seeing the fact that bandwidth and latency are unrelated. Most streaming video systems pre-buffer data to provide the illusion of no interruptions in the transport stream. The underlying behavior is that transport latency can be jittery, but the buffered data keeps the perception of a continuous stream.
For your application, consider using UDP.
TCP is a stream, UDP is for small messages. Your decision will pivot on the question:
What is the effect if a packet is sent but never received?
In the case of a controller input, it may be better to simply ignore the lost data and receive the next transmission. I presuming from your question that you will be repeatedly sending the state of the controller (up down left right?) If so, UDP is you choice. If you choose TCP, the behavior will be to keep retrying to recover this lost data before sending the next data. In your use case, you might as well send the next controller state rather than trying to recover the stream.