ruby/OSX 10.9.2: chef plugin knife-ec2 installs successfully, but is still unavailable

StackOverflow https://stackoverflow.com/questions/22719599

Question

I have been pulling my hair out for about a full 24 hours now. I have searched high and low looking for a solution. I have tried just about everything that I could find.

What i am trying to do: Install the ec2 plugin for the 'knife' program; an invaluable tool for interacting with a Chef server.

What happens everything comes back with rainbows and unicorns. module installed. Module is not available, nor can I find any evidence that it was successfully installed.

My Environment: Details are at the end of this post. high level: OSX 10.9.2 w/ BREW installed as well as Xcode 5.1

It's easiest to just show my problem:

Trying to install the gem

bash-3.2$ sudo -E gem install knife-ec2
Fetching: knife-windows-0.5.15.gem (100%)
Successfully installed knife-windows-0.5.15
Fetching: knife-ec2-0.8.0.gem (100%)
Successfully installed knife-ec2-0.8.0
Parsing documentation for knife-windows-0.5.15
Installing ri documentation for knife-windows-0.5.15
Parsing documentation for knife-ec2-0.8.0
Installing ri documentation for knife-ec2-0.8.0
2 gems installed

Fanstastic! It's been installed. Neat. That was easy! Oh, wait

bash-3.2$ knife ec2
FATAL: Cannot find sub command for: 'ec2'
The ec2 commands were moved to plugins in Chef 0.10
You can install the plugin with `(sudo) gem install knife-ec2

bash-3.2$ sudo knife ec2
FATAL: Cannot find sub command for: 'ec2'
The ec2 commands were moved to plugins in Chef 0.10
You can install the plugin with `(sudo) gem install knife-ec2

It's not available as root or regular user.

After more reading, (see, in part, here How to install knife-ec2 ), I tried a different approach

bash-3.2$ cd /opt/chef/embedded/bin/
bash-3.2$ sudo ./gem install knife-ec2
Building native extensions.  This could take a while...
ERROR:  Error installing knife-ec2:
    ERROR: Failed to build gem native extension.

        /opt/chef/embedded/bin/ruby extconf.rb
Extracting libxml2-2.8.0.tar.gz into tmp/x86_64-apple-darwin13.1.0/ports/libxml2/2.8.0... OK
Running 'configure' for libxml2 2.8.0... OK
Running 'compile' for libxml2 2.8.0... OK
Running 'install' for libxml2 2.8.0... OK
Activating libxml2 2.8.0 (from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.1/ports/x86_64-apple-darwin13.1.0/libxml2/2.8.0)...
Extracting libxslt-1.1.26.tar.gz into tmp/x86_64-apple-darwin13.1.0/ports/libxslt/1.1.26... OK
Running 'configure' for libxslt 1.1.26... OK
Running 'compile' for libxslt 1.1.26... OK
Running 'install' for libxslt 1.1.26... OK
Activating libxslt 1.1.26 (from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.1/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.26)...
checking for libxml/parser.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/opt/chef/embedded/bin/ruby
    --with-zlib-dir
    --without-zlib-dir
    --with-zlib-include
    --without-zlib-include=${zlib-dir}/include
    --with-zlib-lib
    --without-zlib-lib=${zlib-dir}/lib
    --with-iconv-dir
    --without-iconv-dir
    --with-iconv-include
    --without-iconv-include=${iconv-dir}/include
    --with-iconv-lib
    --without-iconv-lib=${iconv-dir}/lib
    --with-xml2-dir
    --without-xml2-dir
    --with-xml2-include
    --without-xml2-include=${xml2-dir}/include
    --with-xml2-lib
    --without-xml2-lib=${xml2-dir}/lib
    --with-xslt-dir
    --without-xslt-dir
    --with-xslt-include
    --without-xslt-include=${xslt-dir}/include
    --with-xslt-lib
    --without-xslt-lib=${xslt-dir}/lib
    --with-libxslt-config
    --without-libxslt-config
    --with-pkg-config
    --without-pkg-config
    --with-libxml-2.0-config
    --without-libxml-2.0-config
    --with-pkg-config
    --without-pkg-config
    --with-libiconv-config
    --without-libiconv-config
    --with-pkg-config
    --without-pkg-config
/opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:381:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:506:in `try_cpp'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:983:in `block in find_header'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:790:in `block in checking_for'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:284:in `block (2 levels) in postpone'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:254:in `open'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:284:in `block in postpone'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:254:in `open'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:280:in `postpone'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:789:in `checking_for'
    from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:982:in `find_header'
    from extconf.rb:166:in `<main>'


Gem files will remain installed in /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.1 for inspection.
Results logged to /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.1/ext/nokogiri/gem_make.out

trying the same approach as above, but with regular user access

bash-3.2$ ./gem install knife-ec2
Fetching: nokogiri-1.6.1.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions into the /opt/chef/embedded/lib/ruby/gems/1.9.1 directory.

more information about my ruby setup

bash-3.2$ rvm -v

rvm 1.25.22 (master) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]


bash-3.2$ ruby -v
ruby 2.0.0p247 (2013-06-27 revision 41674) [universal.x86_64-darwin13]

Below is an env listing. Hopefully this will be helpful to you.

bash-3.2$ env
TERM_PROGRAM=iTerm.app
SHELL=/bin/bash
TERM=xterm
TMPDIR=/var/folders/m7/x_zjzw412rvg7rb_gz8135m5_0892r/T/
Apple_PubSub_Socket_Render=/tmp/launch-bEYnaW/Render
CDC_PREW2KHOST=<xOMITTEDx>
AWS_AVAILABILITY_REGION=us-west-1
OLDPWD=/tmp
COM_GOOGLE_CHROME_FRAMEWORK_SERVICE_PROCESS/USERS/<xOMITTEDx>/LIBRARY/APPLICATION_SUPPORT/GOOGLE/CHROME_SOCKET=/tmp/launch-KSHJfT/ServiceProcessSocket
USER=<xOMITTEDx>
AWS_AVAILABILITY_ZONE=us-west-1a
SSH_AUTH_SOCK=/tmp/launch-lUS8iG/Listeners
__CF_USER_TEXT_ENCODING=0x54042458:0:0
AWS_SSH_KEY_PAIR=<xOMITTEDx>
PATH=/Users/<xOMITTEDx>/.rbenv/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
__CHECKFIX1436934=1
PWD=/tmp/libxslt-1.1.28
EDITOR=/usr/bin/nano
AWS_SECRET_ACCESS_KEY=<xOMITTEDx>
LANG=en_US.UTF-8
ITERM_PROFILE=Default
USER_PRINCIPAL_NAME=<xOMITTEDx>@CORP.ACXIOM.NET
CDC_JOINED_DC=fsccorpdc01.corp.acxiom.net
AWS_ACCESS_KEY_ID=<xOMITTEDx>
HOME=/Users/<xOMITTEDx>
COLORFGBG=7;0
SHLVL=2
CDC_JOINED_SITE=<xOMITTEDx>
ITERM_SESSION_ID=w0t1p0
LOGNAME=<xOMITTEDx>
ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future
CDC_JOINED_ZONE=DC=corp,DC=acxiom,DC=net
CDC_LOCALHOST=<xOMITTEDx>0214mac.corp.acxiom.net
AWS_ACCESS_KEY=<xOMITTEDx>
AWS_SECRET_KEY=<xOMITTEDx>
CDC_JOINED_DOMAIN=corp.acxiom.net
_=/usr/bin/env
bash-3.2$ sudo env
TERM=xterm
SSH_AUTH_SOCK=/tmp/launch-lUS8iG/Listeners
__CF_USER_TEXT_ENCODING=0x54042458:0:0
PATH=/Users/<xOMITTEDx>/.rbenv/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
EDITOR=/usr/bin/nano
LANG=en_US.UTF-8
HOME=/Users/<xOMITTEDx>
COLORFGBG=7;0
SHELL=/bin/bash
MAIL=/var/mail/root
LOGNAME=root
USER=root
USERNAME=root
SUDO_COMMAND=/usr/bin/env
SUDO_USER=<xOMITTEDx>
SUDO_UID=1409557592
SUDO_GID=20
Was it helpful?

Solution 5

The solution was to use virtual box and install a bare-bones ubuntu image.

OTHER TIPS

I had the same problem but with a different plugin.
this got it working for me:

knife wasn't able to find plugins although it says that it'll look for gem install "ed" ones. I think the problem is the rvm part and that the gems are installed in a totally different folder than usual:

~/.rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/

You can check this with the gem env command

$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 1.8.23
  - RUBY VERSION: 1.9.3 (2012-02-16 patchlevel 125) [x86_64-darwin12.5.0]
  - INSTALLATION DIRECTORY: /Users/phil/.rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1
  - RUBY EXECUTABLE: /Users/phil/.rbenv/versions/1.9.3-p125/bin/ruby
  - EXECUTABLE DIRECTORY: /Users/phil/.rbenv/versions/1.9.3-p125/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin-12
  - GEM PATHS:
     - /Users/phil/.rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1
     - /Users/phil/.gem/ruby/1.9.1
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - http://rubygems.org/

So I linked the gems to my ~/.chef/plugins/knife folder like this:

$ ln -s ~/.rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/knife-digital_ocean-0.7.0 ./knife-digital_ocean-0.7.0
$ ln -s ~/.rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/knife-solo-0.4.2 ./knife-solo-0.4.2

and now knife is able to find the plugins alright:

$ knife digital_ocean
FATAL: Cannot find sub command for: 'digital_ocean'
Available digital_ocean subcommands: (for details, knife SUB-COMMAND --help)

** DIGITAL_OCEAN COMMANDS **
knife digital_ocean droplet create (options)
knife digital_ocean droplet destroy (options)
knife digital_ocean droplet list (options)
knife digital_ocean image list (options)
knife digital_ocean region list (options)
knife digital_ocean size list (options)
knife digital_ocean sshkey list

should also work for the ec2 plugin ...

I installed the latest chef, and then ran this:

% sudo /opt/chefdk/embedded/bin/gem install knife-ec2



% knife ec2 server list                              
WARNING: No knife configuration file found
ERROR: You did not provide a valid 'AWS Access Key Id' value.
ERROR: You did not provide a valid 'AWS Secret Access Key' value.

This is expected behaviour. Worked fine, after I setup the right AWS credentials

In the cases you have gem as sudo or run it explicitly from /opt/chef/embedded/bin/, but you are just relying on whatever knife is in your path.

You need to make sure that the gem command and the knife command are from the same place you think they are from.

(follwup for folks passing by)

You probably just needed the libxml headers before installing the gem:

$ yum -y install libxml2-devel # or apt-get install libxml2-dev
$ which knife
/usr/bin/knife
$ ls -l /usr/bin/knife
0 lrwxrwxrwx 1 root root 21 Jun  6 06:39 /usr/bin/knife -> /opt/chefdk/bin/knife
$ head -1 /opt/chefdk/bin/knife 
#!/opt/chefdk/embedded/bin/ruby
$ sudo /opt/chefdk/embedded/bin/gem install knife-ec2 --no-rdoc --no-ri
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top