MegaCLI Raid6 Array creation

I am using Ubuntu Karmic on Dell R610 to access MD1200 storage devices and since (until recently) Openmanage was not a option for the H800 SAS Raid adaptors so I had to explore the wonderful megacli utility!

I am using Ubuntu Karmic on Dell R610 to access MD1200 storage devices and since (until recently) Openmanage was not a option for the H800 SAS Raid adaptors so I had to explore the wonderful megacli utility!

# Find unused disks

root@srv-103-27:/opt/MegaRAID/MegaCli# ./MegaCli64 -PDList -a0 | grep -B14 Unconfigured | grep -e ‘^Enclosure Device ID:’ -e ‘^Slot Number:’
Enclosure Device ID: 41
Slot Number: 11
Enclosure Device ID: 80
Slot Number: 0
Enclosure Device ID: 80
Slot Number: 1
Enclosure Device ID: 80
Slot Number: 2ID: 80
Enclosure Device ID: 80
Slot Number: 3
Enclosure Device ID: 80
Slot Number: 4
Enclosure Device ID: 80
Slot Number: 5
Enclosure Device ID: 80
Slot Number: 6
Enclosure Device ID: 80
Slot Number: 7
Enclosure Device ID: 80
Slot Number: 8
Enclosure Device ID: 80
Slot Number: 9
Enclosure Device ID: 80
Slot Number: 10
Enclosure Device ID: 80
Slot Number: 11
Enclosure Device ID: 106
Slot Number: 0
Enclosure Device ID: 106
Slot Number: 1
Enclosure Device ID: 106
Slot Number: 2
Enclosure Device ID: 106
Slot Number: 3
Enclosure Device ID: 106
Slot Number: 4
Enclosure Device ID: 106
Slot Number: 5
Enclosure Device ID: 106
Slot Number: 6
Enclosure Device ID: 106
Slot Number: 7
Enclosure Device ID: 106
Slot Number: 8
Enclosure Device ID: 106
Slot Number: 9
Enclosure Device ID: 106
Slot Number: 10
Enclosure Device ID: 106
Slot Number: 11
root@srv-103-27:/opt/MegaRAID/MegaCli#

# Create Raid 6 Volume

root@srv-103-27:/opt/MegaRAID/MegaCli# ./MegaCli64 -CfgLdAdd -r6 [80:0,80:1,80:2,80:3,80:4,80:5,80:6,80:7,80:8,80:9,80:10] -a0

Adapter 0: Created VD 5

Adapter 0: Configured the Adapter!!

Exit Code: 0x00
root@srv-103-27:/opt/MegaRAID/MegaCli#

# add dedicated hot spares, we use dedicated as they stay with the array/shelf
root@srv-103-27:/opt/MegaRAID/MegaCli# ./MegaCli64 -PDHSP -Set -Dedicated -Array5 -PhysDrv [80:11] -a0

Adapter: 0: Set Physical Drive at EnclId-80 SlotId-11 as Hot Spare Success.

Exit Code: 0x00

Chef 0.8.x Deb and Upstart

So my chef clients have been crashing and its always a bummer to ssh in and restart it. I could just have my monitoring system start it but why bother when Ubuntu has a wonderful and built in way to make sure the service stays up!

So my chef clients have been crashing and its always a bummer to ssh in and restart it. I could just have my monitoring system start it but why bother when Ubuntu has a wonderful and built in way to make sure the service stays up!

First I downloaded the chef recipe from opscode, then I added the following.

joshua-millers-macbook-pro:site-cookbooks jmiller$ cat chef/recipes/client-deb.rb
#
# Author:: Joshua Miller
# Cookbook Name:: chef
# Recipe:: client-deb
#
# Copyright 2008-2010, Fitsnips.net
#
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# since I have the deb installed already by this point I dont install it.
case node[:platform]
when “ubuntu”
# Upstart is on karmic and above by default … not sure about lower versions
if node[:platform_version].to_f >= 9.10

# my chef server is installed with gems, but for easy of auto install I am using debs
# in my kickstart build with a local apt-mirror. Due to that I have added a check for
# chef-server and its there we dont make any changes.
template “/etc/init.d/chef-client” do
source “chef-client-upstartjob.erb”
owner “root”
group “root”
mode 0774
backup 0
not_if do File.symlink?(“/etc/init.d/chef-server”) end
end

service “chef-client” do
provider Chef::Provider::Service::Upstart
supports :restart => true, :reload => true
end

template “/etc/default/chef-client” do
source “default-chef-client.erb”
owner “root”
group “root”
mode 644
backup 0
not_if do File.symlink?(“/etc/init.d/chef-server”) end
end

template “/etc/init/chef-client.conf” do
source “upstart-chef-client.conf.erb”
owner “root”
group “root”
mode 0644
backup 0
notifies :start, resources(:service => “chef-client”)
not_if do File.symlink?(“/etc/init.d/chef-server”) end
end

end

end

Then we create a few templates:

joshua-millers-macbook-pro:site-cookbooks jmiller$ cat chef/templates/default/upstart-chef-client.conf.erb
start on runlevel [2345]

script
exec /usr/bin/env chef-client -c /etc/chef/client.rb -i <%= @node[:chef][:client_interval] %> -s <%= @node[:chef][:client_splay] %>
end script

# Restart the process if it dies with a signal
# or exit code not given by the ‘normal exit’ stanza.
respawn

# Give up if restart occurs 10 times in 90 seconds.
respawn limit 10 90

Lets make is easy on the other admins who are not used to Upstart:

joshua-millers-macbook-pro:site-cookbooks jmiller$ cat chef/templates/default/chef-client-upstartjob.erb
#!/bin/sh -e
# upstart-job
#
# Symlink target for initscripts that have been converted to Upstart.

set -e

INITSCRIPT=”$(basename “$0″)”
JOB=”${INITSCRIPT%.sh}”

if [ “$JOB” = “upstart-job” ]; then
if [ -z “$1” ]; then
echo “Usage: upstart-job JOB COMMAND” 1>&2
exit 1
fi

JOB=”$1″
INITSCRIPT=”$1″
shift
else
if [ -z “$1” ]; then
echo “Usage: $0 COMMAND” 1>&2
exit 1
fi
fi

COMMAND=”$1″
shift

if [ -z “$DPKG_MAINTSCRIPT_PACKAGE” ]; then
ECHO=echo
else
ECHO=:
fi

$ECHO “Rather than invoking init scripts through /etc/init.d, use the service(8)”
$ECHO “utility, e.g. service $INITSCRIPT $COMMAND”

case $COMMAND in
status)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
$COMMAND “$JOB”
;;
start|stop|restart)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
PID=$(status “$JOB” 2>/dev/null | awk ‘/[0-9]$/ { print $NF }’)
if [ -z “$PID” ] && [ “$COMMAND” = “stop” ]; then
exit 0
elif [ -n “$PID” ] && [ “$COMMAND” = “start” ]; then
exit 0
elif [ -z “$PID” ] && [ “$COMMAND” = “restart” ]; then
start “$JOB”
exit 0
fi
$COMMAND “$JOB”
;;
reload|force-reload)
$ECHO
$ECHO “Since the script you are attempting to invoke has been converted to an”
$ECHO “Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB”
reload “$JOB”
;;
*)
$ECHO
$ECHO “The script you are attempting to invoke has been converted to an Upstart” 1>&2
$ECHO “job, but $COMMAND is not supported for Upstart jobs.” 1>&2
exit 1
esac

chef-client 0.7.16 [BUG] Segmentation fault

Here is what it looked like .. it would start then die within 10 seconds:
jmiller@somerandomname:~$ sudo /etc/init.d/chef-client start
* Starting chef-client chef-client
…done.
jmiller@somerandomname:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

So I have been having problems on one of my chef boxes for the last week, it only showed up on this one system and it was driving me nuts. After a bit of time messing around it with it seems to be a known ruby issue and a updated ubuntu package finally came out!

http://tickets.opscode.com/browse/CHEF-530

Here is what it looked like .. it would start then die within 10 seconds:
jmiller@somerandomname:~$ sudo /etc/init.d/chef-client start
* Starting chef-client chef-client
…done.
jmiller@somerandomname:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Running chef client from the commandline would complete just fine.
jmiller@somerandomname:~$sudo chef-client

But the same segfault would occur when adding daemonize flag and here is a excerpt from the log files

jmiller@somerandomname:~$sudo chef-client -d -l debug

[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: —- End uname -m STDERR —-
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Ran (uname -m) returned 0
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Loading plugin virtualization
[Thu, 11 Mar 2010 17:05:54 -0800] DEBUG: Loading plugin linux::virtualization

I had to update the following ruby packages:

libmixlib-cli-ruby libmixlib-cli-ruby1.8 libmixlib-config-ruby libmixlib-config-ruby1.8 libmixlib-log-ruby libmixlib-log-ruby1.8 libohai-ruby libohai-ruby1.8

Now life is all good again … man that sucked.

18TB Volume and ext4 … you wish

Well I was not really tied to ext4 and it was not a big deal but come on lets stop the lies, ext4 only supports 16 binary tera bytes.

I have recently installed a few Dell MD1200 attached to R710 for long term storage and since I am using Ubuntu Karmic (9.10) I decided to go with ext4. I have read over the spec a few times quickly and had read about how“Ext4 adds 48-bit block addressing, so it will have 1 EB of maximum file system size” . I had not gotten this info from wiki page but rather the tech articles I have read. I get the raid arrays configured and try to create a ext4 fs and up pops this error “too big to be expressed in 32 bits”. I know I am running a 64bit version of Ubuntu so what gives? I double check just to confirm and sure enough “x86_64 GNU/Linux”. As I start to dig around the ugly truth pops up when I read the wiki page “The code to create file systems bigger than 16 TB is, at the time of writing this article, not in any stable release of e2fsprogs. It will be in future releases.” … future releases … ext4 has been in use for over a year now and is the default on karmic.

Well I was not really tied to ext4 and it was not a big deal but come on lets stop the lies, ext4 only supports 16 binary tera bytes and thats not likely to change any time soon.

If your looking for alternatives I suggest a good look at the tried and true xfs, and keep your eye on btrfs filesystem as it looks like it will be the first to bring the promisses of zfs to linux.

Ubuntu 9.10 karmic and Chef

Quick notes on installing chef configuration management on Ubuntu 9.10 Karmic

Quick notes on installing chef configuration management on Ubuntu 9.10 Karmic, this is mostly taken directly from the chef wiki pages but kind of putting it all together and noting problems I ran into.

My automated install is a pretty tight server install:

%packages
openssh-server
curl
nfs-common
portmap
libnss-ldap
libpam-ldap
vlan

I want to install the newest version which is at opscode and not the version in karmic universe so I add the apt repo to the system.

echo “deb http://apt.opscode.com/ karmic universe” > /etc/apt/sources.list.d/opscode.list
curl http://apt.opscode.com/packages@opscode.com.gpg.key | sudo apt-key add –
apt-get update
# actually install chef-server
sudo apt-get install rubygems ohai chef chef-server

I have to manually install git for this server as its usually installed by chef

sudo apt-get -y install git-core

Now I install apache, and the apache modules

sudo apt-get -y install apache2

# module setup
for a2mod in proxy proxy_http proxy_balancer ssl rewrite headers
do
sudo a2enmod $a2mod
done

Now I create the virtual host:

Create /etc/apache2/sites-available/chef_server.repo with the following info, but replace server_fqdn with your chef fully qualified domain name.

<VirtualHost *:443>
ServerName server_fqdn
DocumentRoot /usr/share/chef-server/public

<Proxy balancer://chef_server>
BalancerMember http://127.0.0.1:4000
Order deny,allow
Allow from all
</Proxy>

LogLevel info
ErrorLog /var/log/apache2/chef_server-error.log
CustomLog /var/log/apache2/chef_server-access.log combined

SSLEngine On
SSLCertificateFile /etc/chef/certificates/server_fqdn.pem
SSLCertificateKeyFile /etc/chef/certificates/server_fqdn.pem

RequestHeader set X_FORWARDED_PROTO ‘https’

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://chef_server%{REQUEST_URI} [P,QSA,L]
</VirtualHost>

<VirtualHost *:444%gt;
ServerName server_fqdn
DocumentRoot /usr/share/chef-server/public

<Proxy balancer://chef_server_openid>
BalancerMember http://127.0.0.1:4001
Order deny,allow
Allow from all
</Proxy>

LogLevel info
ErrorLog /var/log/apache2/chef_server-error.log
CustomLog /var/log/apache2/chef_server-access.log combined

SSLEngine On
SSLCertificateFile /etc/chef/certificates/server_fqdn.pem
SSLCertificateKeyFile /etc/chef/certificates/server_fqdn.pem

RequestHeader set X_FORWARDED_PROTO ‘https’

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://chef_server_openid%{REQUEST_URI} [P,QSA,L]
</VirtualHost>

Checkout the chef repo:

cd
git clone git://github.com/opscode/chef-repo.git
cd chef-repo

Time to create your ssl cert

rake ssl_cert FQDN=chef.int.domain

Not sure what I am doing wrong here but I now run the install that for some reason does not copy the certs I just generated over … so I manually copy them over

rake install
cd /root/chef-repo/certificates
cp -a * /etc/chef/certificates/

Now we should be ready to restart apache and see if everything is working

sudo /etc/init.d/apache2 restart

We need to enable the chef virtual site

sudo a2ensite chef_server.repo
/etc/init.d/apache2 reload

now you should be able to bring up the Chef web interface in your browser, if you followed the directions in this writeup it will only work with https.

https://chef.int.domain/

Since I already have a open ldap gateway server configured I am able to log right in and confirm a running install, for more info on that see:

http://mrmiller.nonesensedomains.com/2009/09/18/chef-openid-to-ldap-gateway/

I always like to do a reboot after configuring a new host as even the best make mistakes from time to time and this way I can confirm that everything is starting/running as expected.

Next time I document my work on migrating the roles and cookbooks from my existing install on CentOS 5.3.

Additional notes:

I update my servers as part of the install, but found out that the couchdb that was with the Karmic on release would not start (local mirror was out of date). This was fixed by running

apt-get update
apt-get upgrade -y

I forgot to enable the chef virtual host at first and when I pulled up the URL in my browser got the following error: “SSL received a record that exceeded the maximum permissible length.”. Enabling the site and restarting apache fixed that right up.

Ref:

http://wiki.opscode.com/display/chef/Package+Installation+on+Debian+and+Ubuntu

http://wiki.opscode.com/display/chef/How+to+Proxy+Chef+Server+with+Apache