Problem solved. It can't find the route to IP with the wifi on. Simplest way is to disable wifi, do your stuff and then enable wifi.
Here is the code I used:
// Disable wifi if it's active
WifiManager wifiManager = (WifiManager) getApplicationContext().getSystemService(Context.WIFI_SERVICE);
if (wifiManager.isWifiEnabled())
{
mWasWifiActive = true;
wifiManager.setWifiEnabled(false);
Log.e(TAG, "Wifi was enabled, now Off.");
}
// Do stuff here
// Re-enable wifi if it was active before routing
if (mWasWifiActive)
{
WifiManager wifiManager = (WifiManager) getApplicationContext().getSystemService(Context.WIFI_SERVICE);
wifiManager.setWifiEnabled(true);
Log.e(TAG, "Wifi is back online.");
}