TCP connect retries are a function of the OS's socket implementation. Configuring this depends on the platform. See https://security.stackexchange.com/questions/34607/why-is-the-server-returning-3-syn-ack-packets-during-a-syn-scan for a description of what this is and why it is happening.
On Windows, you should be able to modify the retry count in the registry:
Settings related to the RTT are detailed in that documentation as well.
On Linux, the accepted answer in the linked Security post talks about how to configure this parameter:
On a Linux system, see the special files in
/proc/sys/net/ipv4/
calledtcp_syn_retries
andtcp_synack_retries
: they contain the number of times the kernel would emit SYN (respectively SYN+ACK) on a given connection ... this is as simple asecho 3 > tcp_synack_retries
...
Note that this is a system-wide setting.
You can read the current values by reading the registry settings (on Windows) or reading the contents of the special files (on Linux).
Also, MSDN has this to say about the TCP connect RTT on Windows:
TCP/IP adjusts the frequency of retransmissions over time. The delay between the original transmission and the first retransmission for each interface is determined by the value of the
TcpInitialRTT
entry. By default, it is three seconds. This delay doubles after each attempt. After the final attempt, TCP/IP waits for an interval equal to double the last delay, and then it abandons the connection request.
By the way, re: raw sockets - Yes, you would have an extremely difficult time. Also, as of Windows XP SP2, Windows won't actually let you specify TCP protocol numbers for raw sockets under any circumstances (see Limitations).
Also, as an aside: Make sure that the TCP connection is not being blocked by a separate firewall in front of the client, otherwise you only end up measuring round trip time to the firewall.