Its been a while

Was applying for jobs a while back trying to step back into IC roles but keep hitting the problem with my resume having to much manager experience. I keep tying to figure out how to get past that problem so until then I just keep working at the current job. Pushing ahead learning what I can until I can get that resume back to a good place and move to Austin. Fun stuff ahead!

Looking forward

So after looking back its time to look forward. One of my skills is team building, I have bee successful at bring people into companies and making them a cohesive unit. I break a few rules that a lot of managers would frown at treating my employees more like family then employees. While some may think of that as bad, look at this way how do you treat your kids? Do you treat them like family? Can they still know that your the one in charge and that you make the decisions when they need to be made? Does that mean you never have conflict? I would not say I treat the employees like my children but I do use a lot of the same tool sets. Like its easier to get some to do what you need when they feel valued and are completed on a job well done. We all want to do better work when good work is acknowledged.
Another big thing I do is go out with the team to coffee A LOT! This was something I found enjoyable as a team member back at Tagged. Every day a few of us guys would go out for coffee, walk over, sit down, talk about work or out side life. There was not set agenda it was team mates being friends. When I was finally able to bring in my first employee I was luck enough to have it be one of those guys. We continued the daily coffee run and include each new employee as we added them. On top of that we went out for lunches a few times a quarter, once again no set agenda just enjoying a meal with friends. When we added Jr team members I would pay a lot more often, and on busy days we would not make it, but as a rule every one came. If you where not a coffee drinker then you can have tea, water, jamba juice, or just enjoy the walk.

Looking back

I left after 5 years feeling like I had done a poor job. The reason, I always am working to do better and thats a strength weakness.

Over the last six months a lot has changed in my life, starting most importantly with the addition of my fourth child. Given that my personal life is not the focus of this blog thats the last I will mention of it, but had to acknowledge it.
In addition to that I changed jobs. I moved from a Rdio where I had been for five years to a online ticketing company. Stepping down from a VP position where I had full access to everything to a Director with very little view of anything above me. Both companies are about the same age but its crazy how different they are.
The new ticketing company has a history of changing people running the computers and networks, sometime so fast that tasks where left mid stream. For those of you who have not walked into a situation like that its very interesting. Unlike when I joined Rdio, I would not be starting from scratch but rather making changes in place. At Rdio, I walked in with a rack of servers and a core router that had been sitting for months but not active. Within a few weeks I had the network reconfigured, the build system up, and the rack of systems running and Chefed. Was everything perfect, hell no but it was a solid foundation. From there I built the TechOps team and we continued that solid base.
Being the originator of a environment you can see every hole, everything that is wrong. I would often look at what we had and felt like a failure. Then I would bring in candidates and they would be really impressed. We had focused on a solid core, a very tradition and structured one but solid. We had very clear network paths from X to Y, systems where well defined and single purpose. After five years I started to loose the view of how well we had actually designed the system, so I left feeling like I had done a poor job. The reason, I always am working to do better and do not settle. Strength or weakness?

Be that kind of person

I want everyone I deal with have a chance of walking away with my knowledge.

For some reason a lot of my personal focus for the last couple of years has been culture and treating people like you want to be treated. While this is great for people I admit its also driven by being selfish. I want everyone I deal with have a chance of walking away with my knowledge. This means if you ask and its at all within my power I will sit down and talk to you about anything. Often this is stuff I know really well but at times is about simple observations. Why do I do this? Simply because its what I would want others to do for me. I am not talking about total time wasters but rather if you have a goal or are trying understand I make it a priority to be supportive when possible.
I came to this conclusion when dealing with my eldest son. Often he would come to me with a statement and we would sit down and talk it out. Usually I would try to not give him the answer but walk him through the issue and try to figure out how to find the answer. In the end he not only has learned what what he was expecting to but also how to think. This same process work with co-workers.
When helping another employee I want him to walk away with not only the knowledge of how to fix a error but how to identify the issue. This includes the steps I go through the thought process in my head and often I end up learning. So why technically a fluf piece I am writing this to encourage you, next time someone ask you why how or any other question look for the opportunity to help. Notice I use the word help not teach, because the are not always the same thing.

Chef & Strainer and how my chef-repo was lacking.

Just to start I love Seths work, I know he gets pissed when people comment about his work this is not a comment on his work its a, I hope your google search comes up better then mine.

OK that said I spent a night learning how to get strainer running with travis-ci and read the instruction on the https://github.com/customink/strainer page and was like OK this should take no time at all. Turns out I was wrong. Looking back this should have taken me less then two minutes to figure out but let me be honest I blew more then a hour on this one.

For my testing I created a simple travis-chef-repo directory, dropped in a cookbook directory with the openssh cookbook and its dependency of apt and iptables. Followed the directions on https://github.com/customink/strainer to create a .travis.yml and a Strainer file. Commit it, and your off to the races. Not so quick.

First you need to make sure your Gemfile is correct in travis-chef-repo, here is my example:

jmiller11:travis-chef-repo jmiller$ cat Gemfile

source “https://rubygems.org”

gem ‘rake’
gem ‘chef’
gem ‘foodcritic’
gem ‘rspec’
gem ‘strainer’

Then create the Strainerfile in

jmiller11:travis-chef-repo jmiller$ cat Strainerfile

#
Strainerfile
knife test: bundle exec knife cookbook test $COOKBOOK
foodcritic: bundle exec foodcritic -f any $SANDBOX/$COOKBOOK

Then lets test it!

jmiller11:travistest-chef-repo jmiller$ bundle exec strainer test openssh
I could not detect if you were a chef-repo or a cookbook!
Strainer marked build OK

What the heck, why not? Maybe I have that command wrong lets try this one and maybe it will auto detect the cookbook.

jmiller11:travistest-chef-repo jmiller$ bundle exec strainer test
I could not detect if you were a chef-repo or a cookbook!
Strainer marked build OK

Or how about if we give it the path that must be it.

jmiller11:travistest-chef-repo jmiller$ bundle exec strainer test –cookbooks-path=./cookbooks/
I could not detect if you were a chef-repo or a cookbook!
Strainer marked build OK

Really, how do you tell if this is repo? Lets go look at the code:

https://github.com/customink/strainer/blob/master/lib/strainer/sandbox.rb#L54

else
Strainer.ui.warn “I could not detect if you were a chef-repo or a cookbook!”
@cookbooks = []
end

Umm yea that helps, let look at how we entered this if block:

https://github.com/customink/strainer/blob/master/lib/strainer/sandbox.rb#L54

if chef_repo?

OK so lets look at the chef_repo method:

https://github.com/customink/strainer/blob/master/lib/strainer/sandbox.rb#L244-L249

# Determines if the current project is a chef repo
#
# @return [Boolean]
# true if the current project is a chef repo, false otherwise
def chef_repo?
@_chef_repo ||= begin
chef_folders = %w(.chef certificates config cookbooks data_bags environments roles)
(root_folders & chef_folders).size > 2
end

What really, you need the following directories or are not a chef repo (.chef certificates config cookbooks data_bags environments roles)? At least its easy to fix.

mkdir .chef certificates config cookbooks data_bags environments roles

Lets test this sucker!

jmiller11:travistest-chef-repo jmiller$ bundle exec strainer test openssh
# Straining ‘openssh (v1.3.5)’
knife test | bundle exec knife cookbook test openssh
knife test | checking openssh
knife test | Running syntax check on openssh
knife test | Validating ruby files
knife test | Validating templates
knife test | SUCCESS!
foodcritic | bundle exec foodcritic -f any /Users/jmiller/Development/travistest-chef-repo/cookbooks/openssh
foodcritic | FC007: Ensure recipe dependencies are reflected in cookbook metadata: /Users/jmiller/Development/travistest-chef-repo/cookbooks/openssh/recipes/iptables.rb:20
foodcritic | Terminated with a non-zero exit status. Strainer assumes this is a failure.
foodcritic | FAILURE!
Strainer marked build as failure
jmiller11:travistest-chef-repo jmiller$

I win, lets add, commit, and push. Same “I could not detect if you were a chef-repo or a cookbook!” from travis … really whats going on here. Since I am not a git expert I realize git is not adding empty directories so lets add a file if one does not exist:

for i in certificates config .chef cookbooks data_bags environments roles; do touch $i/README.md;done

git add, commit, push and wait the 8 minutes for the gem install and success. I now see the same foodcritic errors I saw on the command line and have a failing build. Syntax check and lint tools running, time to move on.

Chef AWS

OK so you have your AWS account setup, your ready to launch a EC2 instance with Hosted Chef what do you need to know? I know this may seem simple to someone who has been doing it for a long time but it took me a few hours to figure what exactly I needed. Here is a few quick notes on what I found worked work me on my Macbook Air although it should work just fine on Linux also. I assume you have a working chef install and can upload cookbooks to your Hosted Chef server. Secondly I am using the chef-dk and had to install the knife-ec2 plugin. Assuming you have setup the chef-dk as they recommend all you should need to do is run the chef gem install command. This will install the ruby gems into your home directory at ~/.chefdk so you will not need sudo access.

jmiller11:fcs2-chef-repo jmiller$ ls -l ~/.chefdk/
total 0
drwxr-xr-x 3 jmiller staff 102 May 3 13:35 gem
jmiller11:fcs2-chef-repo jmiller$

Here is the command to install the plugin:

chef gem isntall knife-ec2

Append the following to your .chef/knife.rb

# AWS support
knife[:aws_access_key_id] = ENV[‘AWS_ACCESS_KEY_ID’]
knife[:aws_secret_access_key] = ENV[‘AWS_SECRET_ACCESS_KEY’]
# Optional if you’re using Amazon’s STS
#knife[:aws_session_token] = ENV[‘AWS_SESSION_TOKEN’]
knife[:aws_ssh_key_id] = ENV[‘AWS_MYPEM’]
knife[:region] = ENV[‘AWS_REGION’]
knife[:bootstrap_version]= ‘11.12.4-1’

Append the following to your ~/.bash_profile

AWS_ACCESS_KEY_ID=XXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXX
# note the AWS_MYPEM does not have .pem extension listed
# it found my key that was in ~/.ssh/ and is chmod 600
AWS_MYPEM=XXXXXXXX
AWS_REGION=us-east-1
# Optional if you’re using Amazon’s STS
#AWS_SESSION_TOKEN=””
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_MYPEM AWS_REGION

Source your bash profile to make sure the new variables are active:

jmiller11:fcs2-chef-repo jmiller$ . ~/.bash_profile

Ok lets test that your setup correctly by running “knife ec2 server list”, likely this will be empty for your but as long as it returns the header your fine:

jmiller11:fcs2-chef-repo jmiller$ knife ec2 server list
Instance ID Name Public IP Private IP Flavor Image SSH Key Security Groups IAM Profile State
i-b9ea45e9 base1204-1 m1.small ami-0145d268 aws-jmiller default terminated
i-88e34cd8 m1.small ami-0145d268 aws-jmiller default terminated
i-25e84775 base1204-1 m1.small ami-0145d268 aws-jmiller default terminated
i-c7e44b97 base1204-1 m1.small ami-0145d268 aws-jmiller default terminated
i-41e14e11 base1204-1 54.227.113.203 10.236.185.159 m1.small ami-0145d268 aws-jmiller www, default running
i-7dfc532d base1204-2 54.237.5.212 10.151.112.113 m1.small ami-0145d268 aws-jmiller www, default running
i-53b57800 webserver1 t1.micro ami-3202f25b me default terminated
jmiller11:fcs2-chef-repo jmiller$

Launch Command:

Ok here I am using a simple role called “base” that was uploaded to my chef hosted account all it does at this point is setup chef to run as a cron job to save memory. The ami is a ubuntu 12.04 that will be running on a m1.small instance, with the “default” and “www” security groups, with a easy to read name of “base1204-1” using the ssh key file for my aws key. I need to figure out if there is a better way then defining he ssh key on the line but this works for now.

jmiller11:fcs2-chef-repo jmiller$ knife ec2 server create -r ‘role[BASE]’ -I ami-0145d268 -f m1.small -x ubuntu -G default -N base1204-1 -i ~/.ssh/aws-jmiller

The output of the command will run and you should see something like this:

jmiller11:fcs2-chef-repo jmiller$ knife ec2 server create -r ‘role[BASE]’ -I ami-0145d268 -f m1.small -x ubuntu -G default,www -N base1204-2 -i ~/.ssh/aws-jmiller
Instance ID: i-7dfc532d
Flavor: m1.small
Image: ami-0145d268
Region: us-east-1
Availability Zone: us-east-1a
Security Groups: default, www
Tags: Name: base1204-2
SSH Key: aws-jmiller

Waiting for instance…………………
Public DNS Name: ec2-54-237-5-212.compute-1.amazonaws.com
Public IP Address: 54.237.5.212
Private DNS Name: ip-10-151-112-113.ec2.internal
Private IP Address: 10.151.112.113

Waiting for sshd….done
Connecting to ec2-54-237-5-212.compute-1.amazonaws.com
ec2-54-237-5-212.compute-1.amazonaws.com Installing Chef Client…

ec2-54-237-5-212.compute-1.amazonaws.com Chef Client finished, 7/12 resources updated in 14.133802702 seconds

Instance ID: i-7dfc532d
Flavor: m1.small
Image: ami-0145d268
Region: us-east-1
Availability Zone: us-east-1a
Security Groups: default, www
Security Group Ids: default
Tags: Name: base1204-2
SSH Key: aws-jmiller
Root Device Type: ebs
Root Volume ID: vol-cc24df85
Root Device Name: /dev/sda1
Root Device Delete on Terminate: true
Public DNS Name: ec2-54-237-5-212.compute-1.amazonaws.com
Public IP Address: 54.237.5.212
Private DNS Name: ip-10-151-112-113.ec2.internal
Private IP Address: 10.151.112.113
Environment: _default
Run List: role[BASE]

Ok thats the basics, if you get this far you might want to checkout chef-metal

How ‘DevOps’ Is Killing The Operations Engineer

Before you start to complain, I am a fan of collaboration but Devops might just be the best joke ever!

Before you start to complain, I am a fan of collaboration but Devops might just be the best joke ever! The truth is it means something different to every person. For years I have defined Devops as Engineers trying get Ops out of the way and pushing forward with out those pesky sys admins. Your think I am over blowing it? I have been in the Silicon Valley for the boom of Devops and I hear it all the time “We dont need ops, we can just have a developer do it”. The number of new startups who use AWS thus allowing them to forgo a system administrator never ceases to amaze me. My biggest problem with this is your cutting the legs out from yourself, but your assuring me job security so maybe I should keep my mouth shut.
I have been a a operations engineer for over ten years now, and honestly developers and ops engineers have different ways of functioning. To me a good software engineer has long term focus, can get deep into a project and crunch on the same code for extended durations. Give a good coder a project that will take weeks or even months and they will put there head down and solve your problem. As a generalization these people do not handle interrupt driven work well, they also often do not handle high pressure situations well.
Operations people on the other hand do the majority of their work under massive interruption and constant pressure. Tell a operations engineer the site is down and they will not focus on what the origin of the problem is, they will focus on getting the product back online and come back to fully understand why. This does not mean they do not troubleshoot but they are trying to identify the immediate cause not the who or root. One might argue this is short sited but when your stuck waiting for someone to figure out why the web severs where started your killing your customer experience. I would argue restart the web pool get the product back online and then start to look at root cause once you have identified the customer impact problem and completed the shortest path solution.
When you start off by having your engineers run operations you never allow new ops people to start from ground up and develop their skills, learning the pain points as the system grows thus ensuring when you grow to the point that you need a operations engineer the is a shortage of trained people available. One might argue that some of the developers that started the company by running operations will become your operations engineers and will cover this but to me thats like using a vice grips to remove a bolt.

Your not alone

Want to know the funny part, I have always felt like we where the only company with such a ugly Chef repo, and I was wrong! Even better it does not matter if your using Chef, Puppet, Salt, or some home grown system there is someone who is in the same boat as you.

One thing that I find often happens in at tech conferences is most of us walk away excited about the potential but totally feel defeated by seeing how well other are doing, but here is a little secret if you pull those presenters aside you will find a stack of things that they also wish where better.
I am constantly ashamed when I talk to others about the state of our chef repo, and since I started using Chef over four years ago there is a lot of history in there. For the first two years I was the only guy managing any of our Chef, along with begin the better looking half of a two man team managing our datacenter, servers, network and internal IT. The second guy on the team hated Chef and would alway push things off to me, and to be fair I did the same to him with the network. The down side of this is four and a half years later we are just now starting to get more eyes in the chef repo and over all workings of Chef.
Personally I love it when someone finds a better way to do something and when they start cussing at me about how much they hate (the company formerly know as Opscode) Chef. I have been pushing for other to be more involved, and I don’t fear the skeletons in my Chef closet, but rather ask for help in cleaning them out. I love it when the engineering team comes to me and says hey I wrote this cookbook can we talk about it, or when they say I see you did x in cookbook y why didn’t you do z! This discussion has helped move our repo forward and the pace of change is only getting faster, and in the process we are working out the bugs that a one man team does not even think about.
How did this change happen? We honestly it was a pain, not only because people did not want to learn but Chef is easy to start with but joining a existing system is a little harder. I have a personal chef account I play with all the time as I love doing this stuff but developers just want to get their code out and not worry about the infrastructure to manage it. Like everyone today a good engineers time is over committed and learning a new skill that is not directly related to the problem at hand is hard to find time for, also those Ops guys can do it for them! We finally hit a point where I could not keep up and we had a few engineers who had the desire, and while the other ops team members where doing minor changes they where still very limited in their skills. To help I wrote pages of documentation but no one would read it, they just wanted to complain about what they didn’t know! Finally I went to the VP of Eng and said lets bring in a trainer and I need your guys for three days, luckily he saw the advantage of being able to move faster and having developers help manage the infrastructure and agreed. Three days of training later we had a ton of questions and idea but we are at least on the same page now. I am not saying everything is perfect and I still get a lot of why did you do that, but we have a team working together to improve and that is what Chef is about to me.
Want to know the funny part, I have always felt like we where the only company with such a ugly Chef repo, and I was wrong! Even better it does not matter if your using Chef, Puppet, Salt, or some home grown system there is someone who is in the same boat as you. They may not talk about it on stage but pull those guys aside and while they have solved many of the problems you have, they often have problems you have already solved. The tech ops field is a focus on problem not successes, I mean how often do you hear about them unless the site goes down. The part I am trying to point out is yes, there is much focus on your failures but your not alone and maybe you should step back a second and think about your successes.

Your doing it all wrong AGAIN

Just a few mumblings as I explore the move away from roles, which I agree makes sense but I have 4 years of using roles history I have to refactor.

The joy of rapidly changing software is that if your doing it right today there is a high likely hood your doing it wrong tomorrow. Well that is one way to look at it, or you could just say “I am doing what works for me”. When I started using Opscode chef one of the things that really stood out to me was a stage conversation between Luke and Adam. Summary of that conversation as Luke stated if you don’t do it the defined way your doing it wrong, to which Adam replied unless you need to do it another way. That has pretty much summed up my whole approach and might have some part of why I enjoy chef so much and why Adam is a great front man. Nothing technical here but that is one thing to keep in mind when your working on a new chef deploy, there are best practices but they are not always right for your instance. Just a few mumblings as I explore the move away from roles, which I agree makes sense but I have 4 years of using roles history I have to refactor.