Chef 0.8.x Deb and Upstart

So my chef clients have been crashing and its always a bummer to ssh in and restart it. I could just have my monitoring system start it but why bother when Ubuntu has a wonderful and built in way to make sure the service stays up!

So my chef clients have been crashing and its always a bummer to ssh in and restart it. I could just have my monitoring system start it but why bother when Ubuntu has a wonderful and built in way to make sure the service stays up!

First I downloaded the chef recipe from opscode, then I added the following.

joshua-millers-macbook-pro:site-cookbooks jmiller$ cat chef/recipes/client-deb.rb
#
# Author:: Joshua Miller
# Cookbook Name:: chef
# Recipe:: client-deb
#
# Copyright 2008-2010, Fitsnips.net
#
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# since I have the deb installed already by this point I dont install it.
case node[:platform]
when “ubuntu”
# Upstart is on karmic and above by default … not sure about lower versions
if node[:platform_version].to_f >= 9.10

# my chef server is installed with gems, but for easy of auto install I am using debs
# in my kickstart build with a local apt-mirror. Due to that I have added a check for
# chef-server and its there we dont make any changes.
template “/etc/init.d/chef-client” do
source “chef-client-upstartjob.erb”
owner “root”
group “root”
mode 0774
backup 0
not_if do File.symlink?(“/etc/init.d/chef-server”) end
end

service “chef-client” do
provider Chef::Provider::Service::Upstart
supports :restart => true, :reload => true
end

template “/etc/default/chef-client” do
source “default-chef-client.erb”
owner “root”
group “root”
mode 644
backup 0
not_if do File.symlink?(“/etc/init.d/chef-server”) end
end

template “/etc/init/chef-client.conf” do
source “upstart-chef-client.conf.erb”
owner “root”
group “root”
mode 0644
backup 0
notifies :start, resources(:service => “chef-client”)
not_if do File.symlink?(“/etc/init.d/chef-server”) end
end

end

end

Then we create a few templates:

joshua-millers-macbook-pro:site-cookbooks jmiller$ cat chef/templates/default/upstart-chef-client.conf.erb
start on runlevel [2345]

script
exec /usr/bin/env chef-client -c /etc/chef/client.rb -i <%= @node[:chef][:client_interval] %> -s <%= @node[:chef][:client_splay] %>
end script

# Restart the process if it dies with a signal
# or exit code not given by the ‘normal exit’ stanza.
respawn

# Give up if restart occurs 10 times in 90 seconds.
respawn limit 10 90

Lets make is easy on the other admins who are not used to Upstart:

joshua-millers-macbook-pro:site-cookbooks jmiller$ cat chef/templates/default/chef-client-upstartjob.erb
#!/bin/sh -e
# upstart-job
#
# Symlink target for initscripts that have been converted to Upstart.

set -e

INITSCRIPT=”$(basename “$0″)”
JOB=”${INITSCRIPT%.sh}”

if [ “$JOB” = “upstart-job” ]; then
if [ -z “$1” ]; then
echo “Usage: upstart-job JOB COMMAND” 1>&2
exit 1
fi

JOB=”$1″
INITSCRIPT=”$1″
shift
else
if [ -z “$1” ]; then
echo “Usage: $0 COMMAND” 1>&2
exit 1
fi
fi

COMMAND=”$1″
shift

if [ -z “$DPKG_MAINTSCRIPT_PACKAGE” ]; then
ECHO=echo
else
ECHO=:
fi

$ECHO “Rather than invoking init scripts through /etc/init.d, use the service(8)”
$ECHO “utility, e.g. service $INITSCRIPT $COMMAND”

case $COMMAND in
status)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
$COMMAND “$JOB”
;;
start|stop|restart)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
PID=$(status “$JOB” 2>/dev/null | awk ‘/[0-9]$/ { print $NF }’)
if [ -z “$PID” ] && [ “$COMMAND” = “stop” ]; then
exit 0
elif [ -n “$PID” ] && [ “$COMMAND” = “start” ]; then
exit 0
elif [ -z “$PID” ] && [ “$COMMAND” = “restart” ]; then
start “$JOB”
exit 0
fi
$COMMAND “$JOB”
;;
reload|force-reload)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
reload “$JOB”
;;
*)
$ECHO
$ECHO “The script you are attempting to invoke has been converted to an Upstart” 1>&2
$ECHO “job, but $COMMAND is not supported for Upstart jobs.” 1>&2
exit 1
esac

Chef search and templates … be aware

Search return order is inconsistent, is there a way to deal with it? I really would not care except every time chef runs it restarts perbal which is a issue.

In my recipe:

search(:node, “role:WEBSERVER_ROLE”) do |n|
WEBSERVER_ROLE_host << n['ipaddress'] end search(:node, "role:APACHE_ROLE") do |n| APACHE_ROLE_host << n['ipaddress'] end template "/etc/perlbal/perlbal.conf" do source "perlbal.conf.erb" mode 0440 owner "root" group "root" variables( :WEBSERVER_ROLE_host => WEBSERVER_ROLE_host,
:APACHE_ROLE_host => APACHE_ROLE_host
)
backup 1
notifies :restart, resources(:service => “perlbal”)
end

My Template:

CREATE POOL app_pool
<% @WEBSERVER_ROLE_host.each do |n| -%>
POOL app_pool ADD <%= n %>:80
<% end -%>

CREATE POOL media_pool
<% @APACHE_ROLE_host.each do |n| -%>
POOL media_pool ADD <%= n %>:80
<% end -%>

# run A

CREATE POOL app_pool
POOL app_pool ADD 10.400.441.23:80 <<<< Problem forces restart POOL app_pool ADD 10.400.441.24:80 POOL app_pool ADD 10.400.441.25:80 CREATE POOL media_pool POOL media_pool ADD 10.400.441.27:80 POOL media_pool ADD 10.400.441.28:80 POOL media_pool ADD 10.400.441.27:80 # run B 15 mintues later CREATE POOL app_pool POOL app_pool ADD 10.400.441.25:80 <<<< Problem forces restart POOL app_pool ADD 10.400.441.23:80 POOL app_pool ADD 10.400.441.24:80 CREATE POOL media_pool POOL media_pool ADD 10.400.441.28:80 POOL media_pool ADD 10.400.441.27:80 POOL media_pool ADD 10.400.441.27:80 The fix actually is rather simple, notice the addition of .sort to my vars when passed to the template. I dont need a certain order just a consistent one so this was quick and easy. template "/etc/perlbal/perlbal.conf" do source "perlbal.conf.erb" mode 0440 owner "root" group "root" variables( :WEBSERVER_ROLE_host => WEBSERVER_ROLE_host.sort,
:APACHE_ROLE_host => APACHE_ROLE_host.sort
)
backup 1
notifies :restart, resources(:service => “perlbal”)
end