It looks like if no verify callback is set then OpenSSL library default callback is used. See line 217 in the openssl module verify callback wrapper (the ok
argument is the result of the default callback).
On OS X, Apple has apparently customized the default callback in their OpenSSL library to hook in to the OS X Keychain facility. The root CA of Google's certificate is Equifax Secure Certificate Authority. If you change the trust setting in Keychain (e.g. using the Keychain Access utility) for this CA then your ruby test behaves accordingly.
I believe the answer to your question is that when peer certificate verification is required but not configured, the ruby shipped with OS X verifies using Keychain.