Scaling Puppet With Apache And mod_passenger

As you keep on adding more and more machines to Puppet, it tends to get slower. A major reason for this is that Puppet uses Webrick by default which it ships with. While webrick is good for testing and small scale deployments, it performs poorly as the number of machines increases. So a good alternative is to use Apache or Nginx in combination with mod_passenger or Unicorn. I'll show you how to use Apache + mod_passenger in a very concise way. I think using other combinations should be equally easy.

Install httpd and mod_passenger into your puppetmaster box. Use Phusion Passenger RPM Repository for mod_passenger if it is not available for your distribution. 

# yum install mod_passenger httpd

Now you got to configure passenger. Include the passenger module in apache config and set various params. To keep the configs clean, I recommend creating a separate file and putting passenger related things there. My passenger config looks like this.

Finally let us create a virtual host for the puppet master. Remember that not only apache has to serve catalogs but it also has to take care of SSL as well. So you need to point apache to the certs and the CA created by the puppet master. You might need to install mod_ssl for turning on the SSLEngine. You can check out my apache virtual host config here. RequestHeader(s) are used to set the certificate verification result as environment variable. 

You can tweak around a bit, specially with passenger config to suit your infra needs. If at all anything is not clear please ask me in comments.

Puppet And Common Errors

Installing Puppet can be a nightmare at times especially if you are doing it for the first time. Error messages are not always obvious and would require some experience to understand. So this is my attempt to explain the errors and suggest the solutions.

Needless to say that step one would always be to ensure that the names are resolving and the puppet client and master can communicate. Also make sure that port 8140 is white listed. 

Error 1: err: Could not request certificate: getaddrinfo: Name or service not known

Probable Solution: Puppet client is not able to reach the puppet master. This usually happens when you are setting up a new environment and puppet master's name is not resolvable. If you can, put a relevant entry in your DNS and add a server variable in [agent] section in puppet.conf. Alternatively you can use /etc/hosts to point the client to the master but you'll have to add appropriate entries on the /etc/hosts of both the puppet master and client.

Error 2: Starting puppetmaster: Could not prepare for execution: Could not find a default provider for user

Probable Solution: This happens because of SELinux restrictions. You can fix this by running a "setenforce 0" which will turn off the SELinux. This is required for CA creation only. So you can turn on SELinux after the puppet master creates CA successfully.

Error 3: err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Probable Solution: Looks like your certificates have gone bad. You should remove /var/lib/puppet/ssl directory and request for new certs signed by puppet master.

Error 4: err: Could not retrieve catalog from remote server: hostname was not match with the server certificate

Probable Solution: This may happen if you are referring to the puppet master by a wrong name. In other words, the CA is not built to use this name. You can check out the correct CA name in the file /var/lib/puppet/ssl/ca/inventory.txt. You should put this name in the [agent] section assigned to server variable.

Error 5: err: Could not retrieve catalog from remote server: Connection refused - connect(2)

Probable Solution: This is happening because your puppet client is not able to connect to puppet master. One reason might be firewall which is rejecting the packets and the other reason might be that puppet master has died. So you either need to relax your firewall or make sure that your puppet master is always up and running. You may want to use daemontools or god or a similar application.

Error 6: Exiting; no certificate found and waitforcert is disabled

Probable Solution: This usually happens when a new node is introduced in the infrastructure. Issue is that this node do not have the certificate yet and since "--waitforcert" flag was not enabled, it exited immediately. If your puppet master has autosign enabled that just add the flag "--waitforcert X" with X replaced with time in seconds like 60. If autosign is not enabled then you have to sign the cert for the client manually at your puppet master.

I'll add more as I encounter them. Please let me know in comments if I am wrong anywhere. Have fun with Puppet :)