Joy with Chef 0.8 – and user error!!!

Maybe not really read for prime time, chef 0.8 is a major step forward … but the lack of good docs make it feel like a half a step back.

So install of Chef 0.8.6 on Ubuntu 9.10 karmic was not bad on a clean machine, then I go and do the dumb thing of updating to 0.8.8 now its busted!

root@srv-101-03:~# chef-server
Loading init file from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.8/config/init.rb
Loading /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.8/config/environments/development.rb
/usr/local/lib/site_ruby/1.8/rubygems.rb:230:in `activate’: can’t activate chef (= 0.8.8, runtime) for [“chef-solr-0.8.8”], already activated chef-0.8.6 for [] (Gem::LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems.rb:246:in `activate’

jmiller@srv-101-03:~$ knife node list
/usr/lib/ruby/1.8/net/http.rb:2097:in `error!’: 500 “Internal Server Error” (Net::HTTPFatalError)
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/rest.rb:296:in `run_request’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/rest.rb:106:in `get_rest’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/node.rb:363:in `list’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/knife/node_list.rb:35:in `run’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/application/knife.rb:110:in `run’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/bin/knife:26
from /usr/bin/knife:19:in `load’
from /usr/bin/knife:19
jmiller@srv-101-03:~$

OK here was the dumb and quick fix, the failure was that I run gem upgrade chef … not the command below. Ruby is stupid!

gem install chef -v ‘=0.8.8’

Or maybe not .. that only fixed the error “can’t activate chef”

More progress thank you to the mailing list, webui is back and running but node lists are still messed up:

After looking at your stack trace, you are using Merb 1.1 which is not compatable with Chef .8.8, you should downgrade Merb back to 1.0.15 if you want the webui to work at all.

Damm

root@srv-101-03:~# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
amqp (0.6.7)
bundler (0.9.13)
bunny (0.6.0)
chef (0.8.8)
chef-server (0.8.8)
chef-server-api (0.8.8)
chef-server-webui (0.8.8)
chef-solr (0.8.8)

merb-assets (1.1.0)
merb-core (1.1.0)
merb-haml (1.1.0)
merb-helpers (1.1.0)
merb-param-protection (1.1.0)
merb-slices (1.1.0)

gem uninstall -aIx merb-assets merb-core merb-haml merb-helpers merb-param-protection merb-slices

gem install merb-assets merb-core merb-haml merb-helpers merb-param-protection merb-slices -v ‘~> 1.0.0’

I love the chef mailing list, they pointed out. http://tickets.opscode.com/browse/CHEF-1069

On Tue, Mar 30, 2010 at 5:15 PM, Joshua Miller wrote:
I did a dump of the chef couchdb and am sure this is the problem but do not know enough about couchdb to fix it .. doing research but if anyone just knows the answer.

{“chef_type”: “node”, “name”: null, “_rev”: “1-d40f879d3cbf5d93099b75619d03c8cf”, “defaults”: {}, “run_list”: [], “attributes”: {}, “json_class”: “Chef::Node”, “_id”: “61250eb6-62da-450e-a90a-97856291a2ee”, “overrides”: {}}^M
–==954fdeac87864055bc0716669a22d711==^M
Content-ID: 72b00c98-75c8-4ac8-8aed-723c60686d1c^M
Content-Length: 351^M
Content-MD5: mcsIj4Vf9ssbVNz8jjub2w==^M
Content-Type: application/json;charset=utf-8^M

Joshua,
you probably want to access CouchDB’s webui which is available from a
URL like http://localhost:5984/_utils/

On most installations, CouchDB configured to listen *only* on the
localhost/loopback interface, so you’ll most likely want to set up an
SSH tunnel from port 5984 on your box to localhost:5984. From there,
you can navigate to the “chef” database and then select the nodes >
all_id view. This URL will probably work for that:
http://localhost:5984/_utils/database.html?chef/_design/nodes/_view/all_id

Then find the one with a null/blank id and delete it.

HTH,
Dan DeLeo

Once I deleted the offending node all works again! So happy to have my chef 0.8.8 running again and a big thank you to Dan DeLeo

chef-client 0.7.16 [BUG] Segmentation fault

Here is what it looked like .. it would start then die within 10 seconds:
jmiller@somerandomname:~$ sudo /etc/init.d/chef-client start
* Starting chef-client chef-client
…done.
jmiller@somerandomname:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

So I have been having problems on one of my chef boxes for the last week, it only showed up on this one system and it was driving me nuts. After a bit of time messing around it with it seems to be a known ruby issue and a updated ubuntu package finally came out!

http://tickets.opscode.com/browse/CHEF-530

Here is what it looked like .. it would start then die within 10 seconds:
jmiller@somerandomname:~$ sudo /etc/init.d/chef-client start
* Starting chef-client chef-client
…done.
jmiller@somerandomname:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Running chef client from the commandline would complete just fine.
jmiller@somerandomname:~$sudo chef-client

But the same segfault would occur when adding daemonize flag and here is a excerpt from the log files

jmiller@somerandomname:~$sudo chef-client -d -l debug

[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: —- End uname -m STDERR —-
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Ran (uname -m) returned 0
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Loading plugin virtualization
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Loading plugin linux::virtualization

I had to update the following ruby packages:

libmixlib-cli-ruby libmixlib-cli-ruby1.8 libmixlib-config-ruby libmixlib-config-ruby1.8 libmixlib-log-ruby libmixlib-log-ruby1.8 libohai-ruby libohai-ruby1.8

Now life is all good again … man that sucked.