In addition to capturing the minimal steps required to install a Chef 10 Server on Ubuntu, I also wanted to capture the steps for CentOS. Unfortunately, Ruby 1.9 isn’t available in the default distribution repositories, and almost all third party repositories are quite out of date. As such, I’ll use the Ruby 1.8.7 that comes with CentOS, despite being an EOL version of Ruby.

Similar steps are also available on the Chef wiki, too.

This does not include any customization to the installation and takes everything as default. This will all change when Chef 11 is released.

These steps were performed on a default server install of CentOS 6.3.

Install required packages.

1

yum install ruby ruby-devel rubygems wget gcc gcc-c++ kernel-devel make

Update RubyGems.

1

gem update --system

Install Chef.

1

gem install chef --no-ri --no-rdoc

Configure chef-solo.

1
2

mkdir -p /etc/chef
vi /etc/chef/solo.rb

The config file should look like this:

1
2
3

file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
recipe_url "http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz"

Run chef-solo with the chef-server::rubygems-install recipe.

1

chef-solo -o chef-server::rubygems-install

The WebUI is not enabled by default. This can be done by setting an attribute, see the chef-server cookbook for more information.

Next, create a new API client that will be used to upload cookbooks.

1

knife configure -i

Change the chef_server_url to use the FQDN or IP address of the Chef Server. Change the path for the validation key. The .chef/knife.rb file should look something like this:

1
2
3
4
5
6
7
8
9

log_level                :info
log_location             STDOUT
node_name                'jtimberman'
client_key               '/root/.chef/jtimberman.pem'
validation_client_name   'chef-validator'
validation_key           '/root/.chef/validation.pem'
chef_server_url          'http://10.1.1.100:4000'
cache_type               'BasicFile'
cache_options( :path => '/root/.chef/checksums' )

Copy the validation key (/etc/chef/validation.pem) to the .chef directory.

1

cp /etc/chef/validation.pem ~/.chef

Verify that the API client and configuration file work.

1
2
3
4

knife client list
  chef-validator
  chef-webui
  jtimberman

The newly created API client should be listed. Mine is jtimberman. Now the Chef Server is ready to use.

I wanted to capture the minimal steps required to install a Chef version 10 Server using RubyGems under Ruby 1.9 (1.9.3-p0) on Ubuntu 12.04.

Similar steps are also available on the Chef wiki, too.

This does not include any customization to the installation and takes everything as default. This will all change when Chef 11 is released.

These steps were performed on a default server install of Ubuntu 12.04.

Update apt sources.

1

sudo apt-get update

Install required packages.

1

sudo apt-get install ruby1.9.1-full build-essential wget ssl-cert curl

Update RubyGems.

1

REALLY_GEM_UPDATE_SYSTEM=yes sudo -E gem update --system

Install Chef.

1

sudo gem install chef --no-ri --no-rdoc

Configure chef-solo.

1
2

sudo mkdir -p /etc/chef
sudo vi /etc/chef/solo.rb

The config file should look like this:

1
2
3

file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
recipe_url "http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz"

Run chef-solo with the chef-server::rubygems-install recipe.

1

sudo chef-solo -o chef-server::rubygems-install

The WebUI is not enabled by default. This can be done by setting an attribute, see the chef-server cookbook for more information.

Next, create a new API client that will be used to upload cookbooks.

1

sudo knife configure -i

Change the chef_server_url to use the FQDN or IP address of the Chef Server. Change the path for the validation key. The .chef/knife.rb file should look something like this:

1
2
3
4
5
6
7
8
9

log_level                :info
log_location             STDOUT
node_name                'jtimberman'
client_key               '/home/jtimberman/.chef/jtimberman.pem'
validation_client_name   'chef-validator'
validation_key           '/home/jtimberman/.chef/validation.pem'
chef_server_url          'http://10.1.1.100:4000'
cache_type               'BasicFile'
cache_options( :path => '/home/jtimberman/.chef/checksums' )

Copy the validation key (/etc/chef/validation.pem) to the .chef directory.

1

sudo cp /etc/chef/validation.pem ~/.chef

Verify that the API client and configuration file work.

1
2
3
4

knife client list
  chef-validator
  chef-webui
  jtimberman

The newly created API client should be listed. Mine is jtimberman. Now the Chef Server is ready to use.

Update Chef 10.14 is released. I removed the “--pre” from the gem install commands but otherwise left this post, since it was written by Past Me.

As you may be aware, the next release of Chef, 10.14, is in beta testing. Most publicly available baseboxes for Vagrant aren’t built for the beta. But perhaps you want to try it out?

Vagrant to the rescue! Besides the Chef solo and client provisioners, Vagrant has a shell provisioner, too. It allows you to pass in an inline command, or a shell script. This example builds on my MultiVM Vagrantfile for Chef post. You may wish to refer to the full file in that post for complete context.

First, the shell provisioner line looks like this:

1

config.vm.provision :shell, :inline => "sudo /opt/chef/embedded/bin/gem install chef --no-ri --no-rdoc"

It goes in the config block. Remember that cookbook_testers is a hash of data about the multiple VMs I’m using, so they would all get launched with this provisioner.

1
2
3
4
5
6
7
8
9
10
11
12

Vagrant::Config.run do |global_config|
  cookbook_testers.each_pair do |name, options|
    global_config.vm.define name do |config|
      ### Use shell for great justice!
      config.vm.provision :shell, :inline => "sudo /opt/chef/embedded/bin/gem install chef --no-ri --no-rdoc"
      ### chef-client (allthethings)
      config.vm.provision :chef_client do |chef|
        # ... chef provisioner is the same as before
      end
    end
  end
end

And now, a vagrant up on my Ubuntu 12.04 VM, with the less interesting output truncated:

1
2
3
4
5
6
7
8
9
10
11
12

% vagrant up precise
...
[precise] Running provisioner: Vagrant::Provisioners::Shell...
Successfully installed chef-10.14.0.beta.3
1 gem installed
[precise] Running provisioner: Vagrant::Provisioners::ChefClient...
...
[precise] Running chef-client...
...
[Fri, 10 Aug 2012 17:50:23 +0000] INFO: *** Chef 10.14.0.beta.3 ***
...
[Fri, 10 Aug 2012 17:50:37 +0000] INFO: Chef Run complete in 11.323208919 seconds

One of the most compelling features of Chef 10.14 is the introduction of “no-op” or “dry-run” mode, also called “Why Run Mode”. However, at this time, neither arbitrary command-line arguments nor why-run mode itself are supported in Vagrant. I opened a pull request for the former, which will allow the latter easily. Of course, vagrant is highly valuable for doing testing by actually running your recipes, but I think arbitrary command-line options will be a welcome feature for other purposes.

Another use case for this particular method of using Vagrant provisioners is to update RubyGems inside an “Omnibus” full stack install, to resolve this bug, or this request.

1

config.vm.provision :shell, :inline => "sudo /opt/chef/embedded/bin/gem update --system"

These two example commands can be combined in a single :inline, or as a shell script.

1

config.vm.provision :shell, :path => "gem-updater.sh"

And the shell script, gem-updater.sh in the same directory as the Vagrantfile:

1
2
3

#!/bin/sh
sudo /opt/chef/embedded/bin/gem update --system
sudo /opt/chef/embedded/bin/gem install chef --no-ri --no-rdoc

Update: This post is old and outdated. I’ll have another post in 2015 about workstation management with Chef.

I have written twice before about managing my OS X workstations with Chef. The first post has one of my highest hit counts of any blog post, so it is certainly a topic of interest to people.

This post is a rewrite of the original, and now has an accompanying Chef Repository where all the code I talk about is available.

Background

The current incarnation of this repository lives in the more general private Chef Repository I use for my home network, since I manage more than just workstations with Chef. I have used the main recipe (workstation::default) with great success on several Mac OS X systems: Macbook Pro, iMac, and Macbook Air running various versions between 10.6 and 10.8. I recently used it to configure a replacement Macbook Pro for work, and then again after upgrading to Mountain Lion.

The Setup

Also known as “bootstrapping Chef”, the system needs to be set up to run Chef.

• Opscode Hosted Chef is my Chef Server
• I run everything as a non-privileged user

Installation

I use the Opscode full-stack installer on all my systems, including OS X, because it includes everything Chef needs, including Ruby.

Whether you’re unpacking a brand new Mac, or using an existing system, use this command:

1

curl -L https://opscode.com/chef/install.sh | sudo bash

Symbolic links are created for Chef’s binaries in /usr/bin.

Mountain Lion Note: I did this on Lion before upgrading to Mountain Lion. Apple removed X11 from Mountain Lion, and the installer opens an xterm, so I don’t know how/if this works the same on a brand new Mountain Lion system.

Configuration

Next, create a configuration file for the Chef Server, and copy the validation key into place. If this is a new Mac, you’ll need to get your validation key copied to the system.

1
2
3

sudo mkdir -p /etc/chef
sudo vi /etc/chef/client.rb
cp ~/Downloads/ORGNAME-validation.pem /etc/chef/validation.pem

I am using Opscode Hosted Chef, and this is my /etc/chef/client.rb. The path options are so Chef writes its files in a location my user has write access.

1
2
3
4
5
6
7

base_dir = "/Users/USERNAME/.chef"
chef_server_url         'https://api.opscode.com/organizations/ORGNAME'
validation_client_name  'ORGNAME-validator'
checksum_path           "#{base_dir}/checksum"
file_cache_path         "#{base_dir}/cache"
file_backup_path        "#{base_dir}/backup"
cache_options({:path => "#{base_dir}/cache/checksums", :skip_expires => true})

The Repository

I have made the repository available on GitHub.

1

git clone git://github.com/jtimberman/workstation-chef-repo.git

Normally, systems that are configured with Chef wouldn’t have the Chef Repository on them. For the purpose of this post, clone the repository to the local system. Presumably, one might do further development to it.

Before we upload it and run Chef, let’s explore what is included.

Data Bags

The repository contains two data bags with a single item each. One is for the local user, the other is for the workstation setup.

The USERNAME should be changed to the local user that is being configured on the workstation. To ensure that the correct value is used, run the following and use the value returned.

1
2

% ruby -retc -e 'puts Etc.getlogin'
jtimberman

Thus, I use jtimberman for my systems.

The user data bag item is used in two cookbooks, users and workstation. This is described below under Cookbooks.

The workstation data bag item contains various data about the workstation itself, software that should be installed, property list files dropped off, etc. The JSON file in the repository contains several examples. Modify this as required for your own system.

Roles

There are three roles.

base

This is the role I apply on all my systems, not just workstations. Aside from the contents of the role file in the repository, I also set attributes across my systems for a variety of other purposes like postfix, munin, ntp and so forth. For the workstation setup purposes, it contains the attributes I use for installing Ruby under Rbenv, and the gems I want available on all my systems that aren’t project specific (I use bundler for those).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

name "base"
description "Base role for all nodes"
override_attributes(
  "ruby_build" => {
    "git_ref" => "v20120524",
    "upgrade" => true,
    "install_pkgs" => []
  },
  "rbenv" => {
    "install_pkgs" => [],
    "user_installs" => [
      {
        "user" => "USERNAME",
        "rubies" => ["1.9.3-p194"],
        "global" => "1.9.3-p194",
        "gems" => {
          "1.9.3-p194" => [
            {"name" => "bundler", "version" => "1.1.1"},
            {"name" => "git-up"},
          ]
        }
      }
    ]
  }
)

Edit the list of gems as required for your preferences. The ones included in the role are what I find useful or required for my day to day work on Chef and Chef-related projects (like opscode-cookbooks).

The base role does not have a run list. It is included instead by OS specific roles that I apply to Ubuntu or OS X systems respectively. As this is a post for my OS X workstations, let’s look at that role next.

mac_os_x

The mac_os_x role is applied on all my OS X systems. Of note, it includes the base role, and the homebrew recipe. The homebrew cookbook includes a package resource provider that replaces the Chef default for OS X, macports, with homebrew.

1
2
3
4
5
6
7

name "mac_os_x"
description "Role applied to all Mac OS X systems."
run_list(
  "role[base]",
  "recipe[build-essential]",
  "recipe[homebrew]"
)

This role probably doesn’t need to be edited.

workstation

This is the role of interest, which contains the workstation specific run list and attributes.

The role itself is very long, so I won’t include it here. You can view it in the repository.

Do note that recipe[mac_os_x::firewall] requires root access, and will prompt for the sudo password (and pause the whole run until entered).

Edit the mac_os_x settings as required for your own preferences. Edit other attributes as required for software you wish to use or install.

Cookbooks

The repository uses a number of cookbooks, most of which are published on the Chef Community site as well. I’m not going to describe all the cookbooks in this post, just the ones that are most relevant for workstation setup.

Development Essentials

These are:

• build-essential
• homebrew
• git
• ruby_build
• rbenv

On OS X, build-essential will install Kenneth Reitz’s OS X GCC installer – XCode is not required. Of course, you may have other reasons why you want to have XCode, and that is outside the scope of this repository. If so, remove the build-essential recipe from the roles.

The homebrew cookbook uses Homebrew as the default package provider on OS X. The default recipe will install Homebrew, Git with homebrew, and ensure that the formulae are updated.

IMPORTANT NOTE Homebrew recently “broke” (in my opinion) the output of brew info. I manually patched my local copy of Library/Homebrew/cmd/info.rb after discovering this halfway through setup of my new system.

The git cookbook installs git using the Git OS X installer, and the git binary will be /usr/bin/git. This is redundant with git installed from homebrew, but at some point I had issues and I don’t remember if they were resolved. If you wish to use git from homebrew, use /usr/local/bin/git instead.

The ruby_build and rbenv cookbooks are by Fletcher Nichol, and are quite excellent for installing per-user Rubies of a specific version, and gems using the rbenv_gem LWRP. The base role has the attributes set up for how I like this, YMMV.

users

I use an older, modified version of Opscode’s users cookbook, pre users_manage LWRP. The recipe adds capability for distributing arbitrary files, such as dotfiles for users. To use this, create a “files” section of the users data bag item. The USERNAME.json item includes examples of this. Each file needs to be copied to cookbooks/users/files/default/USERNAME/ as the source file name used in the data bag item.

workstation

The workstation cookbook has a recipe that does all the work of reading the workstation data bag item and setting up the system per the data available.

The README.md in the cookbook contains detailed information about its use, and the data bag item already has the structure to get started.

If the plists array is used, then each plist file should be copied into the files/default/ directory.

mac_os_x

My mac_os_x cookbook has two LWRPs that I use elsewhere in this repository:

• mac_os_x_plist – drops off property list (plist) files in ~/Library/Preferences
• mac_os_x_userdefaults – writes OS X user settings with the defaults(1) system

The plist files used by mac_os_x_plist should be added in the files/default directory of the cookbook where the resource is used in a recipe.

The mac_os_x::settings recipe will read the node['mac_os_x']['settings'] attribute for user defaults to apply.

See the mac_os_x cookbook’s README for more information.

Applications

The following are application specific cookbooks that I use:

• iterm2
• virtualbox
• ghmac
• 1password
• xquartz

The iTerm2 cookbook will set up iTerm 2 and optionally add tmux integration. I wrote about this awhile back.

We use Vagrant extensively at Opscode, which requires VirtualBox. This recipe will install it per the attributes set in the workstation role.

Install GitHub for Mac with the ghmac cookbook. Local setup for it is on your own.

I have 1password in here because I used to install it from the zip file, but I may remove this at some point since I install it from the Mac App Store now.

Note that the versions of these apps may be old, but they have Sparkle.framework or can otherwise update themselves to newer versions easily. Click the buttons, it’s cool.

I don’t have a recipe for managing application installation through the Mac App Store. It’s really not that hard to fire up the app and click the “Install” button next to the apps you want though. Seriously, it would take longer to figure out a command-line or API way to do this, if it’s even possible. Just click the button.

Others

The other cookbooks in the repository are there as dependencies and may or may not be used specifically.

Upload Repository, Run Chef

Once it is cloned, all the components need to be uploaded to the Chef Server with Knife. As that is installed with Chef, it will be available. The knife config file and user key do need to be copied to .chef in the chef-repo. If necessary, download them from Opscode Hosted Chef (or your Chef Server).

1
2
3
4

cd workstation-chef-repo
mkdir .chef
cp ~/Downloads/knife.rb .chef
cp ~/Downloads/USERNAME.pem .chef

Make your changes to the data bags and roles. I’ll wait here.

Then, upload everything.

1
2
3
4
5
6

knife data bag create users
knife data bag create apps
knife data bag from file users USERNAME.json
knife data bag from file apps workstation.json
knife role from file base.rb mac_os_x.rb workstation.rb
knife cookbook upload -a

Finally, run Chef!

1
2
3
4
5
6

% whoami
jtimberman
% chef-client
INFO: *** Chef 10.12.0 ***
... loads of output, hooray ...
INFO: Chef Run complete in 45.116912 seconds

FAQ

These aren’t necessarily questions anyone asked, but a more preemptive FAQ :).

This seems heavyweight, why all this effort?

As a sysadmin, I want to do something once and automate it afterward. That includes all the stuff I need to do to have a useful, usable work environment. This means that when I get a new computer, or have to wipe and reinstall (rare, but happens), I can get back to a productive environment very quickly.

I have three OS X systems I use regularly (work laptop, personal laptop, family iMac). Having them in a Chef Server gives me access to information about these systems easily with knife.

Also, this post is focused specifically on OS X, however this setup works pretty much as is on Linux. I simply don’t use Linux as a desktop OS, but I do have “workstation-like” systems that I SSH into, and this is generally fine for those.

Why Chef Server? Why not Chef Solo?

Honestly, I don’t actually use Chef Solo except as a way to setup a Chef Server. Since I use Chef Client/Chef Server so often, it is second nature for me to do it. You’re free to adapt this to work with Solo.

Will you support Windows with this repository?

No. I don’t use Windows as a workstation/desktop anymore.

It might just work on Windows though. It did once, but I haven’t tried in a few months.

I want to make this moar awesome, will you merge my pull request?

Thank you. I appreciate that you want to help me, or other members of the community. However I consider this pretty much “feature complete”, as it meets all my needs, and I don’t plan to merge any pull requests.

For individual cookbooks, they have their own repositories linked from their pages on the Chef Community site.

Why do you have redundancy or inconsistent use?

Such as plist file location, dmg installation, etc.

Because: Reasons. This codebase has been developed over ~2 years. It works for me.

How can I get help?

You can email me. However, as I said before this is a free time project, so I might not respond right away. If you’re an Opscode Hosted or Private Chef customer, please contact Opscode support. Finally, community based support is available through our community resources.

Further resources

If this is a topic of interest to you, I’d also like to point out a few similar projects that may be interesting. They have inspired me and things I have implemented in my own setup, so thank you Ben, Corey and Matthew and Brian at Pivotal!

• Ben Bleything’s OS X Bootstrap
• Corey Donohoe’s Cinderella
• Pivotal Labs workstation setup

I upgraded my work laptop to Mountain Lion today. It was not as smooth as previous OS X upgrades have been for me, despite my efforts in managing my workstation(s) with Chef.

I received a replacement laptop for the one that was damaged at ChefConf a couple weeks ago (champagne spill – long story, maybe for another blog post). As this is a new laptop, it is eligible for the free Mountain Lion upgrade. So on ML release day, I submitted my information to Apple for the redemption code, which I received this morning. As I’m actually on vacation this week, I thought there was no better time to upgrade.

You may recall that I have managed my workstations with Chef for quite some time. This has been all well and good so far, though the Mountain Lion installation didn’t seem to like something in my preferences along the way.

After the installation finished and the system rebooted, I logged in, expecting to be greeted with my already configured system. This was not the case, however! My desktop was that light grey of the OS X boot screen, and the Dock was not running at all. I put on my sysadmin hat (like I ever take it off?!), and started debugging. I found the issue pretty quickly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Jul 27 08:42:35 champagne.local Dock[423]: -[__NSCFBoolean isEqualToString:]: unrecognized selector sent to instance 0x7fff75d0fab0
Jul 27 08:42:35 champagne.local Dock[423]: *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFBoolean isEqualToString:]: unrecognized selector sent to instance 0x7fff75d0fab0'
        *** First throw call stack:
        (
                0   CoreFoundation                      0x00007fff8fca9716 __exceptionPreprocess + 198
                1   libobjc.A.dylib                     0x00007fff8dc63470 objc_exception_throw + 43
                2   CoreFoundation                      0x00007fff8fd3fd5a -[NSObject(NSObject) doesNotRecognizeSelector:] + 186
                3   CoreFoundation                      0x00007fff8fc97c3e ___forwarding___ + 414
                4   CoreFoundation                      0x00007fff8fc97a28 _CF_forwarding_prep_0 + 232
                5   Dock                                0x000000010da92786 Dock + 681862
                6   Dock                                0x000000010d9f10b2 Dock + 20658
                7   Dock                                0x000000010dab9aed Dock + 842477
                8   libdyld.dylib                       0x00007fff852c17e1 start + 0
        )
Jul 27 08:42:35 champagne com.apple.launchd.peruser.501[267] (com.apple.Dock.agent[423]): Job appears to have crashed: Abort trap: 6
Jul 27 08:42:35 champagne com.apple.launchd.peruser.501[267] (com.apple.Dock.agent): Throttling respawn: Will start in 1 seconds
Jul 27 08:42:35 champagne.local ReportCrash[302]: Saved crash report for Dock[423] version 1.8 (1168) to /Users/jtimberman/Library/Logs/DiagnosticReports/Dock_2012-07-27-084235_champagne.crash

This happened every second, constantly crashing and restarting, and generating a new crash report. What is the problem? Well, let’s look at one of the reports:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

Process:         Dock [21816]
Path:            /System/Library/CoreServices/Dock.app/Contents/MacOS/Dock
Identifier:      Dock
Version:         1.8 (1168)
Code Type:       X86-64 (Native)
Parent Process:  launchd [1039]
User ID:         501

Date/Time:       2012-07-27 11:26:58.819 -0600
OS Version:      Mac OS X 10.8 (12A269)
Report Version:  10

Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFBoolean isEqualToString:]: unrecognized selector sent to instance 0x7fff7b2a2ab0'
abort() called
terminate called throwing an exception

Application Specific Backtrace 1:
0   CoreFoundation                      0x00007fff8cd77716 __exceptionPreprocess + 198
1   libobjc.A.dylib                     0x00007fff8e6df470 objc_exception_throw + 43
2   CoreFoundation                      0x00007fff8ce0dd5a -[NSObject(NSObject) doesNotRecognizeSelector:] + 186
3   CoreFoundation                      0x00007fff8cd65c3e ___forwarding___ + 414
4   CoreFoundation                      0x00007fff8cd65a28 _CF_forwarding_prep_0 + 232
5   Dock                                0x000000010d1b6786 Dock + 681862
6   Dock                                0x000000010d1150b2 Dock + 20658
7   Dock                                0x000000010d1ddaed Dock + 842477
8   libdyld.dylib                       0x00007fff8d7827e1 start + 0

I’ll spare you the detail of the rest of the file. Suffice to say, it is less informative than one might want for troubleshooting the issue. I (foolishly) spent about an hour and a half trying to get to the root of the problem. I found out that the problem was isolated to my own user, and something between ~/Library/Preferences and ~/Library/Application Support. I’m not sure what the problem was – I eventually decided to just do this:

1

sudo rm -rf ~/Library/Preferences ~/Library/Application\ Support

Then I logged back in, and everything was well. I have a back up from before the upgrade (Yay, TimeMachine!), so I wasn’t concerned with losing anything important, and I knew that Chef would bring back most of my settings anyway.

The first thing I did was, of course, /usr/bin/chef-client. This worked great. Until the Dock was restarted as part of my recipes. The symptom was the same as before – the desktop went light grey, the dock wasn’t running and launchd was respawning it contnually, with the same errors from above.

I decided to have some quality time with my configuration and get to the bottom of the problem. I went through all the settings I modify through recipe[mac_os_x::settings] attributes, and did a careful comparison of manual settings through the OS X system preferences, and the files changed in ~/Library/Preferences.

Side note: This handy hint comes from Ben Bleything:

1
2
3
4

cd ~/Library/Preferences
git init
git add .
git commit -m 'initial commit'

Then, make a change in system preferences, you can use git status to see what plist files are updated. Of course, most of the plist files are binary so you can’t really diff them, but the filename will indicate which domain to use with the defaults(1) command. Handy, thanks Ben! End side note.

Anyway, it took some time, but I tuned all my configuration for the things I wanted to ensure happened on any new system, and nothing else. I believe what happened is that some setting I had is no longer supported on Mountain Lion and its presence makes Dock.app grumpy, but that is purely speculation. The end result now, though, is that I have a pretty sane set of configuration that is automatically applied in a more data driven way, and I’m not using settings that I don’t know 100% what they do.

That aside, Mountain Lion is nice so far. The GCC Installer for 10.7 seems to be working just fine, though I haven’t tried installing a new Ruby under it yet. I am looking forward to wider use and adoption of the Notification Center as a replacement for Growl.

I hope this post is helpful in some way. Unfortunately I don’t have an answer to the Dock crash problem itself, but I have now remedied the issue for my own use. I’m going to write a new post about how I’m managing my workstations, to bring the information posted previously up to date, so stay tuned.

It is quite common in Debian and Ubuntu that when installing a package that provides a daemon, said daemon is started by the init script(s) included in the package. This is a matter of Debian Policy, though I don’t interpret that section to literally mean it is required. However, it is common enough practice that several people have asked (or ranted) about the topic.

The main issue of course is that the default configuration for the software being installed may not be appropriate before starting up the service and making it available on the network. Users of other Linux distributions may be smugly smirking as their distribution doesn’t start the service on package installation.

This post isn’t about that.

Instead, this post describes how this problem is resolved using configuration management, specifically Chef. I’m also going to discuss a couple nuances about service management, so watch carefully.

For the example service I’m going to use memcached, from the memcached package. It is started on package installation as demonstrated:

1
2
3
4
5
6
7
8

vagrant@precise-housepub:~$ sudo apt-get install memcached
...
Setting up memcached (1.4.13-0ubuntu2) ...
Starting memcached: memcached.
vagrant@precise-housepub:~$ service memcached status
 * memcached is running
vagrant@precise-housepub:~$ ps awux | grep memcached
 memcache 15176  0.0  0.3 323212  1180 ?        Sl   04:32   0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1

As we can see, the memcached service is started. Of course, it is using the default configuration, which means that it has a very small memory size, and listens on localhost. While the recipe would be very simple:

1

package "memcached"

This wouldn’t be very useful for discussion, or practical use purposes. For now, I’m going to post the entire recipe I’m going to discuss, and then break it down.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

node.set['memcached']['memory_max'] = node['memory']['total'].to_i / 1024 * 0.65

package "memcached"

service "memcached" do
  supports :restart => true, :status => true
  action :enable
end

template "/etc/memcached.conf" do
  source "memcached.conf.erb"
  owner "root"
  group "root"
  mode 00644
  notifies :restart, "service[memcached]"
  variables(
    :memory_max => node['memcached']['memory_max']
    :ip_addr => node['ipaddress']
  )
end

service "memcached" do
  action :start
end

This recipe is fairly straightforward. First, it sets a node attribute based on a calculation of the amount of memory installed in the system. Then, it will install the memcached package. This of course will start up the service with the unsuitable defaults already discussed.

The first service resource occurance makes sure that the service is enabled. This is the default behavior of the package manager, but this also gives clear indication as to the intention of the recipe. Next, the configuration file is managed. The exact content of the memcached.conf.erb file isn’t particularly important. Let us presume that the variables passed in are what we care about – that we want to use 65% of the system’s total memory for memcached, and listen on the default IP address. Maybe other tuning is happening, maybe not. Of course, when the configuration is updated, we need to notify the memcached service to restart.

Finally, the memcached service is started if it is not already running. This occurs at the end to help remedy an issue where the service might have been halted, or the configuration file was rendered incorrectly (a typo?), so that we can correct such configuration problems with Chef in a single subsequent run. This uses a feature of Chef where resources can be declared multiple times with different actions (or if desired, parameters).

The first time this recipe is run on a node, memcached will be installed, started, configured, restarted. We’re not aiming to prevent the auto-start from occurring at all, but we do automate the additional steps required for handling that easily.

I created a plugin for knife that will display a specified option from Chef’s configuration object, Chef::Config. It operates with the scope of the automatically detected knife configuration file, or by passing the -c option with a configuration file.

Installation

It’s a gem.

1

% gem install knife-config

You can get the source on GitHub.

Examples

Without any options:

1
2

% knife config
WARNING: No knife configuration file found

If an config option were specified, its default value would be printed.

With a single option, chef_server_url, but no configuration file available:

1
2
3

% knife config chef_server_url
WARNING: No knife configuration file found
chef_server_url:  http://localhost:4000

In a “Chef Repository” with a .chef/knife.rb:

1
2
3

% cd chef-repo
% knife config chef_server_url
chef_server_url:  https://api.opscode.com/organizations/MY_ORG

With a few more options:

1
2
3
4
5

% cd chef-repo
% knife config chef_server_url node_name validation_client_name
chef_server_url:  https://api.opscode.com/organizations/MY_ORG
node_name:        jtimberman
validation_client_name:  housepub-validator

With a different config file (/etc/chef/client.rb on the same system):

1
2

% knife config node_name -c /etc/chef/client.rb
node_name:  imac.int.example.com

With --all, to show all the options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

% knife config --all
amqp_consumer_id:               default
amqp_host:                      0.0.0.0
amqp_pass:                      testing
amqp_port:                      5672
amqp_user:                      chef
amqp_vhost:                     /chef
authorized_openid_identifiers:
authorized_openid_providers:
cache_options:
  path:  /Users/jtimberman/.chef/checksums
cache_type:                     BasicFile
checksum_path:                  /var/chef/checksums
chef_server_url:                https://api.opscode.com/organizations/housepub
client_key:                     /Users/jtimberman/.chef/jtimberman.pem
client_registration_retries:    5
client_url:                     http://localhost:4042
[output truncated, trust me, it shows them all :)]

Show the “knife” configuration, which includes things like cloud provider authentication. This doesn’t currently support showing sub-keys (like knife[aws_access_key_id]).

1
2
3
4

% knife config knife
knife:
  aws_access_key_id:      ZOMGAWS_KEY
  aws_secret_access_key:  ZOMGAWS_SECRET

Contributing

If you’d like to contribute, that’s awesome! Please create a fork, branch, and a pull request. Reporting issues is helpful too.

Most commonly, Vagrant’s Vagrantfile describes only a single VM. That’s fine, but most environments separate functionality to different servers (e.g., database and web app). For this reason, Vagrantfiles can be set up for multi-VM arrangements.

However, I’m going to describe a different use case for multiple VMs.

Testing Multiple Distributions

As a cookbook developer for a variety of platforms and platform versions, I have to ensure changes do not break existing functionality across supported platforms – namely current releases of Ubuntu and CentOS (and their parents, Debian and RHEL).

Enter Vagrant’s Multi-VM Vagrantfile.

While I posted recently about testing with VMware Fusion, I am using Vagrant more. Primarily because Opscode uses Vagrant internally – Seth Chisamore built a multi-VM Vagrantfile for bringing up our full stack. The specifics in his Vagrantfile are tuned to that particular use case which is different than mine. However, there are similar patterns that I adapted.

Vagrantfile

The Vagrantfile configures four virtual machines:

• CentOS 5.7 and 6.2
• Ubuntu 10.04 and 11.10

I built my VMs with veewee templates that install Chef via the omnibus built chef-full package. That way I have a consistent installation that reflects what Opscode will ship as the easiest and best supported way to install Chef.

Part of the magic of this configuration is that I’m going to reuse my knife configuration. The Vagrantfile itself goes into my cookbook testing Chef Repository.

1
2
3
4
5
6

    require 'chef'
    require 'chef/config'
    require 'chef/knife'

    current_dir = File.dirname(__FILE__)
    Chef::Config.from_file(File.join(current_dir, '.chef', 'knife.rb'))

Next, I’m going to describe data about the virtual machines that I’m going to run. This is a hash of named VMs, centos5, lucid, etc. I assign their hostname, and give them a host only IP address. I also set an initial run list, since Vagrant will (noisily) complain if the run list is empty in a Chef provisioner.

Note I have a base role as a holder, the actual relevant things are in the base_redhat and base_debian roles. The details really don’t matter, though.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

cookbook_testers = {
  :centos5 => {
    :hostname => "centos5-cookbook-test",
    :ipaddress => "172.16.13.5",
    :run_list => "role[base],role[base_redhat]"
  },
  :centos6 => {
    :hostname => "centos6-cookbook-test",
    :ipaddress => "172.16.13.6",
    :run_list => "role[base],role[base_redhat]"
  },
  :lucid => {
    :hostname => "lucid-cookbook-test",
    :ipaddress => "172.16.13.10",
    :run_list => "role[base],role[base_debian]"
  },
  :oneiric => {
    :hostname => "oneiric-cookbook-test",
    :ipaddress => "172.16.13.11",
    :run_list => "role[base],role[base_debian]"
  }
}

Next, I’m going to set up the Vagrant::Config object as “global” configuration for all the VMs, and then iterate over each of the VMs described above.

1
2
3
4
5

Vagrant::Config.run do |global_config|
  cookbook_testers.each_pair do |name, options|
    global_config.vm.define name do |config|
      vm_name = "#{name}-cookbook-test"
      ipaddress = options[:ipaddress]

I disable the shared folder, since I’m going to use a Chef Server, and my recipes will download what they need from remote repositories, not my local system.

1

config.vm.share_folder("v-root", "/vagrant", ".", :disabled => true)

Set up some basic configuration for the box. Modify this to suit your environment. This section is on a per-VM basis. If particular tunables were required, I’d create additional config in the cookbook_testers hash above, and use those values here.

Note name will be a symbol, but only in some contexts of execution.

1
2
3
4

config.vm.box = name.to_s
config.vm.boot_mode = :headless
config.vm.host_name = vm_name
config.vm.network :hostonly, ipaddress

Now I set up the Chef provisioner. Again, I’m using Chef with a Server (Opscode Hosted Chef, of course). I use the chef_server_url, and validation_client_name settings from my knife.rb.

The nodes’ names will be NAME-cookbook-test, rather than their FQDN. I use this with a rake task that nukes them all from orbit consistently :).

1
2
3
4
5
6

chef.chef_server_url = Chef::Config[:chef_server_url]
chef.validation_key_path = "#{current_dir}/.chef/#{Chef::Config[:validation_client_name]}.pem"
chef.validation_client_name = Chef::Config[:validation_client_name]
chef.node_name = vm_name
chef.provisioning_path = "/etc/chef"
chef.log_level = :info

The run list is going to be combined from the run lists defined from the cookbook_testers hash above, and a shell environment variable, CHEF_RUN_LIST, which is simply a comma-separated list of run list items, similar to that used by knife bootstrap.

1
2
3

run_list = []
run_list << ENV['CHEF_RUN_LIST'].split(",") if ENV.has_key?('CHEF_RUN_LIST')
chef.run_list = [options[:run_list].split(","), run_list].flatten

To use the Vagrantfile, I export the shell variable with the role(s)/recipe(s) I am testing, then run vagrant up.

1
2

% export CHEF_RUN_LIST="recipe[apache2],recipe[apache2::mod_ssl]"
% vagrant up

Vagrant will bring up each VM one at a time, going through the full cycle of provisioning. If there’s an unhandled exception that causes Chef to exit, then Vagrant also halts execution. If vagrant up is rerun, then Vagrant continues to the next VM. To reprovision a failed VM, it can be specified:

1

% vagrant provision centos5

Without the VM name, vagrant would reprovision all the VMs. Likewise, vagrant ssh NAME can be used to open an SSH connection to the named VM. This is useful to reprovision a VM that failed early, while Vagrant is continuing on with the others.

Full Vagrantfile

The Vagrantfile is split up in the earlier section, but you can see the full thing below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

require 'chef'
require 'chef/config'
require 'chef/knife'
current_dir = File.dirname(__FILE__)
Chef::Config.from_file(File.join(current_dir, '.chef', 'knife.rb'))
cookbook_testers = {
  :centos5 => {
    :hostname => "centos5-cookbook-test",
    :ipaddress => "172.16.13.5",
    :run_list => "role[base],role[base_redhat]"
  },
  :centos6 => {
    :hostname => "centos6-cookbook-test",
    :ipaddress => "172.16.13.6",
    :run_list => "role[base],role[base_redhat]"
  },
  :lucid => {
    :hostname => "lucid-cookbook-test",
    :ipaddress => "172.16.13.10",
    :run_list => "role[base],role[base_debian]"
  },
  :oneiric => {
    :hostname => "oneiric-cookbook-test",
    :ipaddress => "172.16.13.11",
    :run_list => "role[base],role[base_debian]"
  }
}
Vagrant::Config.run do |global_config|
  cookbook_testers.each_pair do |name, options|
    global_config.vm.define name do |config|
      vm_name = "#{name}-cookbook-test"
      ipaddress = options[:ipaddress]
      config.vm.share_folder("v-root", "/vagrant", ".", :disabled => true)
      config.vm.box = name.to_s
      config.vm.boot_mode = :headless
      config.vm.host_name = vm_name
      config.vm.network :hostonly, ipaddress
      config.vm.provision :chef_client do |chef|
        chef.chef_server_url = Chef::Config[:chef_server_url]
        chef.validation_key_path = "#{current_dir}/.chef/#{Chef::Config[:validation_client_name]}.pem"
        chef.validation_client_name = Chef::Config[:validation_client_name]
        chef.node_name = vm_name
        chef.provisioning_path = "/etc/chef"
        chef.log_level = :info
        run_list = []
        run_list << ENV['CHEF_RUN_LIST'].split(",") if ENV.has_key?('CHEF_RUN_LIST')
        chef.run_list = [options[:run_list].split(","), run_list].flatten
      end
    end
  end
end

Fact: GitHub is classy. This isn’t just because Scott Chacon works there, either. Their handling of a security issue today was very professional. That said, I have some words to say about the issue itself and the aftermath, and things you as an application developer can do to help, and to avoid this kind of problem.

Disclaimer: I’m not an application developer. I am a sysadmin with a diverse background in operating system security, including previously held GSEC and GCUX certifications from the SANS Institute/GIAC.

Second disclaimer: This is not an anti-Rails post. All web frameworks need to be conscious of security, and take bug reports for security issues seriously.

Issue

Update – Preface: I am not talking about a security vulnerability (a la an exploit) in Rails. I am talking about a feature that allows automatically generated code to do things that are not secure and it is apparently on purpose and by design. This is the wrong thing to do. Deny by default with whitelisting is the right thing to do.

There is a security issue in Ruby on Rails. The bug is closed, but I haven’t dug into find out if the actual problem is fixed.

The bug itself is very serious. It allows a malicious user to send arbitrary parameters to a Rails application without requiring whitelisting up front. In fact, three months ago an issue was opened in the Rails project to force new applications to enforce whitelist mode by default. That bug was subsequently closed just a few days ago. I’m not going to do an analysis of the issue itself, you can go read the linked tickets and do further research on the issue.

Update I missed clarifying this. There is a second issue at hand. GitHub resolved the mass assignment bug by fixing their application. The second issue is that they had a vulnerability in their public key form update.

Resolution and Aftermath

You can read all about the initial resolution and retrospective on the GitHub blog. You can also read their follow-up post on responsible disclosure.

The manner in which they handled the situation is a class act: they behaved like professionals. Here’s why:

• A user reported a problem with their app.
• They worked with the user to resolve the problem.
• The same user exploited another vulnerability to prove a point to the Rails project, which is against the GitHub terms of service.
• GitHub suspended the user’s account in accordance with their terms of service.

As an unauthorized breach of a computer system, what Egor Homakov did is illegal in the United States. It was also irresponsible and unprofessional. However, his intent was not malicious. I think GitHub did the right thing by giving him another chance in reinstating Mr. Homakov’s account several hours after the incident.

GitHub has issued two apologies about this incident. First for the vulnerability existing in the first place, and second for not being clear how customers and users can responsibly disclose security vulnerabilities. They also committed to doing a security audit of their code base.

GitHub is classy.

Security

After the above, I feel compelled to say some more things about security in general. You are responsible for a lot of things regarding the web applications in your infrastructure. One of those is security, and you should do everything you can to write stable, secure code.

In a hackernews thread about this incident, Yehuda Katz said, “Not all security vulnerabilities can be protected automatically by a web framework.”

That is a fact. However, web frameworks should provide sane, secure settings by default. Those settings should be modifiable by the end-user, the developer. If a developer wishes to disable those controls, they totally have that right. I think that they need to understand the potential risk that they are accepting, and what impact that might have on the business/organization implementing the application.

This is exactly like the default setting of Red Hat Enterprise Linux to enable SELinux by default on new installations. Whether you love or hate this default, it is sane and secure. System administrators can then use the system as is, or disable SELinux if that is an acceptable level of risk.

Clearly in this incident, it is not an acceptable level of risk for GitHub, as they have repaired their application. It would have better to do that long ago, but at least it’s fixed now.

Security and convenience are quite often polar opposites and mutually exclusive. This is not always the case, but it is true much of the time, if not most of the time. The choice for Rails to not have whitelisting by default is in favor of developer convenience. Yes, it is up to the developer to make their application secure, but that was already the case. This simply creates extra work for them to do so.

Deny-all by default is the sane, correct and secure posture to take when building systems. This is the practice of many tools and operating system defaults – SELinux as mentioned, or “no open ports” per Ubuntu’s practice. You don’t have to agree with it, and you certainly can change it, but that doesn’t change the fact that it is correct and sane.

Vulnerability Disclosure

There is an entire field in information technology devoted to vulnerability disclosure. This is typically done by people performing “ethical hacking” and is one thing done when a code base goes through a security audit. This is a field that has responsible professionals participating in a variety of companies, and if it sounds interesting to you, I recommend the variety of courses offered by the SANS Institute:

• Web Application Penetration Testing (SEC542)
• Network Penetration Testing (SEC560)
• Advanced Penetration Testing (SEC660)

What You Can Do

First of all, understand the security guidelines and best practices for the programming language you’re using. Doing things that are typesafe, or avoid buffer overflows, that kind of thing. Also understand and follow the security guideslines and best practices for the web framework you’re using. The Ruby on Rails project has a fairly detailed security guide. If you’re taking shortcuts, understand the possible risks with that. If you don’t know the risks, or understand the guidelines, please ask someone in the community for help.

I strongly recommend you also learn the security guidelines and associated best practices for the operating system or distribution that your application will run on in production. If your organization has operations staff, I’m sure they can help you learn and understand. If they don’t, they’re not doing their job :).

Every organization and every application is different. The security implications are going to vary by industry. Talk to the business owners and find out what the level of security risk they are comfortable accepting.

Above all, be a professional. Don’t flippantly close security bugs. Don’t be a dick on discussions about security topics.

In other words, be classy. Like GitHub.

Thank you.

Recently, Apple did the most awesome thing for non-Xcode developers.

They made the development tools available as a standalone package!

This is awesome news for those of us who only have^Whad Xcode installed to install RubyGems that compile native extensions, or for installing software with Homebrew, MacPorts or similar.. You can download them by logging into the Developer Download site.

This appears to be work started by Kenneth Reitz with his OSX GCC Installer project. I did try that project out, but ran into issues I didn’t resolve right away, so I reverted to using Xcode proper. However with the package from Apple I don’t seem to have any issues so far.

If you already have Xcode installed, you may want to remove it first.

1
2

sudo /Developer-3.2.6/Library/uninstall-devtools
sudo /Developer/Library/uninstall-devtools --mode=all /thx

My understanding is that you can remove the /Developer* director(y|ies) when complete. I had ollllld Xcode on the system where I first did this.

Next, download and install the package from Apple. It’s about 170M and takes only a couple minutes to install; sorry I don’t have a Chef recipe for this ;).

I did run into an issue with Homebrew where it wasn’t finding the right gcc binary. I had to run the following commands to fix that issue.

1
2

sudo xcode-select -switch /usr/bin
sudo ln -sf /usr/bin/llvm-gcc-4.2 /usr/bin/gcc-4.2

You wouldn’t think that something like an 8G installation would matter in 2012. However, disk space is a precious commodity on MacBook Airs and systems that have SSDs as the root volume. This is very welcome change for me, especially since it means that future Mac OS X installations do not require a large download before I can start doing things that get my system ready to use.