CoreOS: fleetctl list-machines show error

Question

I'm really new on CoreOS, I'm trying to use official doc to create one cluster of 3 nodes with Vagrant. But, when I access one of the nodes and execute command: fleetctl list-machines the output show some errors:

E0729 09:29:05.725714 01013 fleetctl.go:141] error attempting to check latest fleet version in Registry: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Error retrieving list of active machines: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

I have Internet access via proxy --> 172.23.1.0:3128

I suspect the error occurs because the nodes can not be connected to https://discovery.etcd.io/2bef6823259cdef6751632f1f7052ff3 But I'm not sure what's the problem, and I do not know how to fix it.

I'm using:

Vagrant -----> 1.6.3

VirtualBox --> 4.3.10_Ubuntu

CoreOS box --> 367.1.0 (Stable)

These are my resource files:

Vagrantfile

# -*- mode: ruby -*-
# # vi: set ft=ruby :

require 'fileutils'

Vagrant.require_version ">= 1.6.0"

CLOUD_CONFIG_PATH = File.join(File.dirname(__FILE__), "user-data")
CONFIG = File.join(File.dirname(__FILE__), "config.rb")

# Defaults for config options defined in CONFIG
$num_instances = 1
$update_channel = "stable"
$enable_serial_logging = false
$vb_gui = false
$vb_memory = 1024
$vb_cpus = 1

# Attempt to apply the deprecated environment variable NUM_INSTANCES to
# $num_instances while allowing config.rb to override it
if ENV["NUM_INSTANCES"].to_i > 0 && ENV["NUM_INSTANCES"]
  $num_instances = ENV["NUM_INSTANCES"].to_i
end

if File.exist?(CONFIG)
  require CONFIG
end

Vagrant.configure("2") do |config|
  config.vm.box = "coreos-%s" % $update_channel
  #config.vm.box_version = ">= 308.0.1"
  config.vm.box_url = "coreos_production_vagrant.box"

  config.vm.provider :vmware_fusion do |vb, override|
      override.vm.box_url = "coreos_production_vagrant.box"
  end

  config.vm.provider :virtualbox do |v|
     v.check_guest_additions = false
     v.functional_vboxsf     = false
     v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
     v.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
  end

  # plugin conflict
  if Vagrant.has_plugin?("vagrant-vbguest") then
     config.vbguest.auto_update = false
  end

  (1..$num_instances).each do |i|
    config.vm.define vm_name = "core-%02d" % i do |config|
    config.vm.hostname = vm_name

    if $enable_serial_logging
      logdir = File.join(File.dirname(__FILE__), "log")
      FileUtils.mkdir_p(logdir)

      serialFile = File.join(logdir, "%s-serial.txt" % vm_name)
      FileUtils.touch(serialFile)

      config.vm.provider :vmware_fusion do |v, override|
        v.vmx["serial0.present"] = "TRUE"
        v.vmx["serial0.fileType"] = "file"
        v.vmx["serial0.fileName"] = serialFile
        v.vmx["serial0.tryNoRxLoss"] = "FALSE"
      end

      config.vm.provider :virtualbox do |vb, override|
        vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
        vb.customize ["modifyvm", :id, "--uartmode1", serialFile]
      end
    end

    if $expose_docker_tcp
      config.vm.network "forwarded_port", guest: 2375, host: ($expose_docker_tcp + i - 1), auto_correct: true
    end

    config.vm.provider :vmware_fusion do |vb|
      vb.gui = $vb_gui
    end

    config.vm.provider :virtualbox do |vb|
      vb.gui = $vb_gui
      vb.memory = $vb_memory
      vb.cpus = $vb_cpus
    end

    ip = "33.33.33.#{i+100}"
    config.vm.network :private_network, ip: "#{ip}"
    config.vm.synced_folder ".", "/home/core/share", id: "core", :nfs => true, :mount_options => ['nolock,vers=3,udp']

    if File.exist?(CLOUD_CONFIG_PATH)
      config.vm.provision :file, :source => "#{CLOUD_CONFIG_PATH}", :destination => "/tmp/vagrantfile-user-data"
      config.vm.provision :shell, :inline => "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", :privileged => true
    end

   end
 end
end

user-data:

#cloud-config

coreos:
  etcd:
      # generate a new token for each unique cluster from https://discovery.etcd.io/new
      # WARNING: replace each time you 'vagrant destroy'
      discovery: https://discovery.etcd.io/2bef6823259cdef6751632f1f7052ff3
      addr: $public_ipv4:4001
      peer-addr: $public_ipv4:7001
  fleet:
      public-ip: $public_ipv4
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

config.rb:

# Size of the CoreOS cluster created by Vagrant
$num_instances=3

# Official CoreOS channel from which updates should be downloaded
$update_channel='stable'

Answer 1

Finally, I was download etcd from github repo and run etcd in background over eternal host with IP address 10.10.10.10. Then, inspired by CoreOS instruction to run your own discovery endpoint, I set discovery address in user-data: as follow:

discovery: http://10.10.10.10:4001/v2/keys/2bef6823259cdef6751632f1f7052ff3

After this, I can execute vagrant up and when I inspect http://10.10.10.10:4001/v2/keys/2bef6823259cdef6751632f1f7052ff3 all CoreOS instance are registered.

Answer 2

I opened an issue for etcd to support the proxy via env vars, similar to the CoreOS update_engine.

To solve this issue in the short term, you can hardcode a list of peer addresses for your machines via cli flag or config file. --peers or --peers-file are probably your best bet.

Answer 3

I had the same problem. In my case, it was caused by a stale discovery URL in the #cloud-config. I fixed it by obtaining a new discovery url from https://discovery.etcd.io/new