jtimberman's Code Blog

Chef, Ops, Ruby, Linux/Unix. Opinions are mine, not my employer's (CHEF).

Chef 12: Fix Untrusted Self Sign Certs

Scenario: You’ve started up a brand new Chef Server using version 12, and you have installed Chef 12 on your local system. You log into the Management Console to create a user and organization (or do this with the command-line chef-server-ctl commands), and you’re ready to rock with this knife.rb:

1
2
3
4
5

node_name                'jtimberman'
client_key               'jtimberman.pem'
validation_client_name   'tester-validator'
validation_key           'tester-validator.pem'
chef_server_url          'https://chef-server.example.com/organizations/tester'

However, when you try to check things out with knife:

1
2
3

% knife client list
ERROR: SSL Validation failure connecting to host: chef-server.example.com - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

This is because Chef client 12 has SSL verification enabled by default for all requests. Since the certificate generated by the Chef Server 12 installation is self-signed, there isn’t a signing CA that can be verified, and this fails. Never fear intrepid user, for you can get the SSL certificate from the server and store it as a “trusted” certificate. To find out how, use knife ssl check.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Connecting to host chef-server.example.com:443
ERROR: The SSL certificate of chef-server.example.com could not be verified
Certificate issuer data: /C=US/ST=WA/L=Seattle/O=YouCorp/OU=Operations/CN=chef-server.example.com/emailAddress=you@example.com

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1j 15 Oct 2014
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate, you must
configure chef to trust that server's certificate.

By default, the certificate is stored in the following location on the host
where your chef-server runs:

  /var/opt/chef-server/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently: /Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm
that the server's certificate is now trusted.

(note, this chef-server location is incorrect, it’s /var/opt/opscode)

There is a fetch plugin for knife too. Let’s download the certificate to the automatically preconfigured trusted certificate location mentioned in the output above.

1
2
3
4
5
6
7
8

% knife ssl fetch
WARNING: Certificates from chef-server.example.com will be fetched and placed in your trusted_cert
directory (/Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for chef-server.example.com in /Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs/chef-server.example.com.crt

The certificate should be verified that what was downloaded is in fact the same as the certificate on the Chef Server. For example, I compared SHA256 checksums:

1
2
3
4

% ssh ubuntu@chef-server.example.com sudo sha256sum /var/opt/opscode/nginx/ca/chef-server.example.com.crt
043728b55144861ed43a426c67addca357a5889158886aee50685cf1422b5ebf  /var/opt/opscode/nginx/ca/chef-server.example.com.crt
% gsha256sum .chef/trusted_certs/chef-server.example.com.crt
043728b55144861ed43a426c67addca357a5889158886aee50685cf1422b5ebf  .chef/trusted_certs/chef-server.example.com.crt

Now check knife client list again.

1
2

% knife client list
tester-validator

Victory!

Now, we need to get the ceritficate out to every node in the infrastructure in its trusted_certs_dir – by default this is /etc/chef/trusted_certs. The most simple way to do this is to use knife ssh to run knife on the target nodes.

1
2
3
4
5
6
7
8

% knife ssh 'name:*' 'sudo knife ssl fetch -c /etc/chef/client.rb'
node-output.example.com WARNING: Certificates from chef-server-example.com will be fetched and placed in your trusted_cert
node-output.example.com directory (/etc/chef/trusted_certs).
node-output.example.com
node-output.example.com Knife has no means to verify these are the correct certificates. You should
node-output.example.com verify the authenticity of these certificates after downloading.
node-output.example.com
node-output.example.com Adding certificate for chef-server.example.com in /etc/chef/trusted_certs/chef-server.example.com.crt

The output will be interleaved for all the nodes returned by knife ssh. Of course, we should verify the SHA256 checksums like before, which can be done again with knife ssh.

Comments