Chef search and templates … be aware

Search return order is inconsistent, is there a way to deal with it? I really would not care except every time chef runs it restarts perbal which is a issue.

In my recipe:

search(:node, “role:WEBSERVER_ROLE”) do |n|
WEBSERVER_ROLE_host << n['ipaddress'] end search(:node, "role:APACHE_ROLE") do |n| APACHE_ROLE_host << n['ipaddress'] end template "/etc/perlbal/perlbal.conf" do source "perlbal.conf.erb" mode 0440 owner "root" group "root" variables( :WEBSERVER_ROLE_host => WEBSERVER_ROLE_host,
:APACHE_ROLE_host => APACHE_ROLE_host
)
backup 1
notifies :restart, resources(:service => “perlbal”)
end

My Template:

CREATE POOL app_pool
<% @WEBSERVER_ROLE_host.each do |n| -%>
POOL app_pool ADD <%= n %>:80
<% end -%>

CREATE POOL media_pool
<% @APACHE_ROLE_host.each do |n| -%>
POOL media_pool ADD <%= n %>:80
<% end -%>

# run A

CREATE POOL app_pool
POOL app_pool ADD 10.400.441.23:80 <<<< Problem forces restart POOL app_pool ADD 10.400.441.24:80 POOL app_pool ADD 10.400.441.25:80 CREATE POOL media_pool POOL media_pool ADD 10.400.441.27:80 POOL media_pool ADD 10.400.441.28:80 POOL media_pool ADD 10.400.441.27:80 # run B 15 mintues later CREATE POOL app_pool POOL app_pool ADD 10.400.441.25:80 <<<< Problem forces restart POOL app_pool ADD 10.400.441.23:80 POOL app_pool ADD 10.400.441.24:80 CREATE POOL media_pool POOL media_pool ADD 10.400.441.28:80 POOL media_pool ADD 10.400.441.27:80 POOL media_pool ADD 10.400.441.27:80 The fix actually is rather simple, notice the addition of .sort to my vars when passed to the template. I dont need a certain order just a consistent one so this was quick and easy. template "/etc/perlbal/perlbal.conf" do source "perlbal.conf.erb" mode 0440 owner "root" group "root" variables( :WEBSERVER_ROLE_host => WEBSERVER_ROLE_host.sort,
:APACHE_ROLE_host => APACHE_ROLE_host.sort
)
backup 1
notifies :restart, resources(:service => “perlbal”)
end

Xargs

So I have used xargs for years but never real thought much about it as when I wanted to do something more advanced … like reorder output I would simple use awk.

Normally you run xargs as:

jmiller@srv-101-03:~$ echo "sample" | xargs ls -l
-rw-r--r-- 1 jmiller jmiller 0 2010-04-29 15:18 sample
jmiller@srv-101-03:~$

but what if you want to do something more advanced like

echo “sample” | xargs -i echo {} two

work calls will have to finish later

chef, knife, and ssh – loving it!

Opscode added a ssh call to the knife utility which when used with the search syntax can be very nice. A few minor examples below.

Opscode added a ssh call to the knife utility which when used with the search syntax can be very nice. A few minor examples below.

jmiller@srv-101-03: $ knife ssh role:APACHE_ROLE uptime
srv-101-18.example.com  02:07:24 up 140 days, 23:23,  1 user,  load average: 0.00, 0.00, 0.00
srv-101-17.example.com  02:07:24 up 125 days, 10:53,  1 user,  load average: 0.03, 0.06, 0.02

j

miller@srv-101-03:~/operations/chef/roles$ knife ssh “role:BASE_ROLE” ‘ grep paranoia /etc/nscd.conf ‘
srv-101-01.example.com # paranoia
srv-101-01.example.com paranoia no
srv-101-14.example.com # paranoia
srv-101-14.example.com paranoia yes
srv-201-22.example.com # paranoia
srv-201-22.example.com paranoia yes
srv-201-01.example.com # paranoia
srv-201-01.example.com paranoia yes
srv-201-26.example.com # paranoia
srv-201-26.example.com paranoia yes
srv-101-04.example.com # paranoia
….

Backup chef roles

I like to keep my chef roles in git so I do a dump of them and check them when I make changes. Very nice if you remove something and can not recall what it is.

I like to keep my chef roles in git so I do a dump of them and check them when I make changes. Very nice if you remove something and can not recall what it was as you jump around.

#!/bin/bash

####
#
# Must be run from a server that has knife and your key i.e. chef.int.rdio
#
###

# List of all roles:

knife role list | sed s/\”//g | sed s/,// | egrep -v ‘\]|\[‘ > ./rolelist.txt

# Generate a file for each role containing the servers in that role

for i in `cat rolelist.txt`; do echo $i; knife role show $i > $i.json; done

Quick and dirty server list from chef

So I have always used a simple bash look to do quick task on lots of servers:

Example:

for i in `cat server.list`; do ssh $i ‘hostname;uptime’;done

We can use chef to build list of servers by role, and a list all servers in a our farm if managed by chef 🙂

#!/bin/bash

####
#
# Must be run from a server that has knife and your key i.e. chef.server.com
#
###

# I think I am going to make this a recipe
# but for now…

#Generate a list of all chef controlled servers

knife node list | sed s/\”//g | sed s/,// | grep -v \] > /home/operations/servers/all.txt

# List of all roles:

knife role list | sed s/\”//g | sed s/,// | egrep -v ‘\]|\[‘ > /home/operations/servers/roles.txt

# Generate a file for each role containing the servers in that role
# Tetsu likes the files lower case … works for me 🙂

for i in `cat roles.txt`; do echo $i; z=`echo $i | tr ‘[:upper:]’ ‘[:lower:]’`; knife search node role:$i -i > $z.txt; done

Joy with Chef 0.8 – and user error!!!

Maybe not really read for prime time, chef 0.8 is a major step forward … but the lack of good docs make it feel like a half a step back.

So install of Chef 0.8.6 on Ubuntu 9.10 karmic was not bad on a clean machine, then I go and do the dumb thing of updating to 0.8.8 now its busted!

root@srv-101-03:~# chef-server
Loading init file from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.8/config/init.rb
Loading /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.8/config/environments/development.rb
/usr/local/lib/site_ruby/1.8/rubygems.rb:230:in `activate’: can’t activate chef (= 0.8.8, runtime) for [“chef-solr-0.8.8”], already activated chef-0.8.6 for [] (Gem::LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems.rb:246:in `activate’

jmiller@srv-101-03:~$ knife node list
/usr/lib/ruby/1.8/net/http.rb:2097:in `error!’: 500 “Internal Server Error” (Net::HTTPFatalError)
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/rest.rb:296:in `run_request’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/rest.rb:106:in `get_rest’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/node.rb:363:in `list’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/knife/node_list.rb:35:in `run’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/lib/chef/application/knife.rb:110:in `run’
from /usr/lib/ruby/gems/1.8/gems/chef-0.8.8/bin/knife:26
from /usr/bin/knife:19:in `load’
from /usr/bin/knife:19
jmiller@srv-101-03:~$

OK here was the dumb and quick fix, the failure was that I run gem upgrade chef … not the command below. Ruby is stupid!

gem install chef -v ‘=0.8.8’

Or maybe not .. that only fixed the error “can’t activate chef”

More progress thank you to the mailing list, webui is back and running but node lists are still messed up:

After looking at your stack trace, you are using Merb 1.1 which is not compatable with Chef .8.8, you should downgrade Merb back to 1.0.15 if you want the webui to work at all.

Damm

root@srv-101-03:~# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
amqp (0.6.7)
bundler (0.9.13)
bunny (0.6.0)
chef (0.8.8)
chef-server (0.8.8)
chef-server-api (0.8.8)
chef-server-webui (0.8.8)
chef-solr (0.8.8)

merb-assets (1.1.0)
merb-core (1.1.0)
merb-haml (1.1.0)
merb-helpers (1.1.0)
merb-param-protection (1.1.0)
merb-slices (1.1.0)

gem uninstall -aIx merb-assets merb-core merb-haml merb-helpers merb-param-protection merb-slices

gem install merb-assets merb-core merb-haml merb-helpers merb-param-protection merb-slices -v ‘~> 1.0.0’

I love the chef mailing list, they pointed out. http://tickets.opscode.com/browse/CHEF-1069

On Tue, Mar 30, 2010 at 5:15 PM, Joshua Miller wrote:
I did a dump of the chef couchdb and am sure this is the problem but do not know enough about couchdb to fix it .. doing research but if anyone just knows the answer.

{“chef_type”: “node”, “name”: null, “_rev”: “1-d40f879d3cbf5d93099b75619d03c8cf”, “defaults”: {}, “run_list”: [], “attributes”: {}, “json_class”: “Chef::Node”, “_id”: “61250eb6-62da-450e-a90a-97856291a2ee”, “overrides”: {}}^M
–==954fdeac87864055bc0716669a22d711==^M
Content-ID: 72b00c98-75c8-4ac8-8aed-723c60686d1c^M
Content-Length: 351^M
Content-MD5: mcsIj4Vf9ssbVNz8jjub2w==^M
Content-Type: application/json;charset=utf-8^M

Joshua,
you probably want to access CouchDB’s webui which is available from a
URL like http://localhost:5984/_utils/

On most installations, CouchDB configured to listen *only* on the
localhost/loopback interface, so you’ll most likely want to set up an
SSH tunnel from port 5984 on your box to localhost:5984. From there,
you can navigate to the “chef” database and then select the nodes >
all_id view. This URL will probably work for that:
http://localhost:5984/_utils/database.html?chef/_design/nodes/_view/all_id

Then find the one with a null/blank id and delete it.

HTH,
Dan DeLeo

Once I deleted the offending node all works again! So happy to have my chef 0.8.8 running again and a big thank you to Dan DeLeo

chef-client 0.7.16 [BUG] Segmentation fault

Here is what it looked like .. it would start then die within 10 seconds:
jmiller@somerandomname:~$ sudo /etc/init.d/chef-client start
* Starting chef-client chef-client
…done.
jmiller@somerandomname:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

So I have been having problems on one of my chef boxes for the last week, it only showed up on this one system and it was driving me nuts. After a bit of time messing around it with it seems to be a known ruby issue and a updated ubuntu package finally came out!

http://tickets.opscode.com/browse/CHEF-530

Here is what it looked like .. it would start then die within 10 seconds:
jmiller@somerandomname:~$ sudo /etc/init.d/chef-client start
* Starting chef-client chef-client
…done.
jmiller@somerandomname:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Running chef client from the commandline would complete just fine.
jmiller@somerandomname:~$sudo chef-client

But the same segfault would occur when adding daemonize flag and here is a excerpt from the log files

jmiller@somerandomname:~$sudo chef-client -d -l debug

[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: —- End uname -m STDERR —-
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Ran (uname -m) returned 0
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Loading plugin virtualization
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Loading plugin linux::virtualization

I had to update the following ruby packages:

libmixlib-cli-ruby libmixlib-cli-ruby1.8 libmixlib-config-ruby libmixlib-config-ruby1.8 libmixlib-log-ruby libmixlib-log-ruby1.8 libohai-ruby libohai-ruby1.8

Now life is all good again … man that sucked.

18TB Volume and ext4 … you wish

Well I was not really tied to ext4 and it was not a big deal but come on lets stop the lies, ext4 only supports 16 binary tera bytes.

I have recently installed a few Dell MD1200 attached to R710 for long term storage and since I am using Ubuntu Karmic (9.10) I decided to go with ext4. I have read over the spec a few times quickly and had read about how“Ext4 adds 48-bit block addressing, so it will have 1 EB of maximum file system size” . I had not gotten this info from wiki page but rather the tech articles I have read. I get the raid arrays configured and try to create a ext4 fs and up pops this error “too big to be expressed in 32 bits”. I know I am running a 64bit version of Ubuntu so what gives? I double check just to confirm and sure enough “x86_64 GNU/Linux”. As I start to dig around the ugly truth pops up when I read the wiki page “The code to create file systems bigger than 16 TB is, at the time of writing this article, not in any stable release of e2fsprogs. It will be in future releases.” … future releases … ext4 has been in use for over a year now and is the default on karmic.

Well I was not really tied to ext4 and it was not a big deal but come on lets stop the lies, ext4 only supports 16 binary tera bytes and thats not likely to change any time soon.

If your looking for alternatives I suggest a good look at the tried and true xfs, and keep your eye on btrfs filesystem as it looks like it will be the first to bring the promisses of zfs to linux.

Preseeding anyone?

Preseed is a powerful tool but its a pain to work with

Today during my ubuntu install I noticed some major problems that I had not seen on the over 100 previous installs. While I install via pxe and use local mirrors for updates I failed to notice that during install the system was reaching out to security.ubuntu.com. This is not a huge problem but is network over head I would rather keep local but today it turned out to be a huge problem as security was having major issues and my 12 minute install did not complete in over an hour. Using F4 on the boot I noticed it was stuck on pulling a update from security.ubuntu.com and thats when the real fun started.

I of course had the url set for local installs but never noticed that did not cover everything.

#Use Web installation
url --url http://pxe.example.com/ubuntu-9.10-server-x86_64/ubuntu/

After a lot of digging I found the needed lines. These lines need to be right after the url according to the Ubuntu docs.

#Use Web installation
url --url http://pxe.example.com/ubuntu-9.10-server-x86_64/ubuntu/
preseed apt-setup/security_host string aptmirror.example.com
preseed apt-setup/security_path string /repo_mirror/ubuntu-9.10-server-x86_64/

Preseed is a powerful tool but its a pain to work with, the link below will give you some additional guidance.

Opscode adds training but will anyone care?

While this is a next logical step I feel that they need to focus on getting 8.0 out before they even start to worry about training. I have held off on suggesting chef to a lot of people due to tall the change coming in 8.0.

So as you can see I have enjoyed Opscodes chef a lot but now I see they added training. While this is a next logical step I feel that they need to focus on getting 8.0 out before they even start to worry about training. I have held off on suggesting chef to a lot of people due to tall the change coming in 8.0. As a user or almost 8 months now I feel the changes are so extreme that its not worth starting with chef at this point. While the recipes an basic stuff your pushing out will move forward a lot of logic changes happen in 8.0. First there is the new databags that will allow you to rethink how you use shared data. Then there is the joy of roles in roles, which I love by the way. Why these are really minor changes I hate to think about going over all my roles and recipes and reworking them for 8.0. Not because you have to but more because I like to have a common pattern in execution and I assure you that I will be using these new features in new additions to my chef tool kit. O then there is knife … umm yea world changer there. So in summary hold off on training opscode and get 8.0 out the door.