Setting up Yubikey for SSH two-factor with public key authentication

I just bought a Yubikey Neo. It is a tiny usb device which can be used for multi-factor authentication with many application. But setting it up can become tricky at times due to lack of documentation. Here is what I did to setup ssh with Yubikey two-factor for my Fedora 20 and Fedora 21:

Step1: Install the pam module for yubikey auth.
# yum install pam_yubico

Step 2: We need to create a mapping of user and the yubikey associated with it. We'll need the key id of the yubikey. To obtain that, we just need to open any text editor, plug the key in a usb slot and touch the golden button on the Yubikey. The first 12 characters is the id. Also note that, there can be multiple keys associated with one user. We'll create a file /etc/yubi-map:
# cat /etc/yubi-map
aditya:scwechdueeuv

Step 3: Now we have to add the yubico pam module to the sshd auth. Change the /etc/pam.d/sshd so that first few lines look like this:
#%PAM-1.0
auth       required     pam_sepermit.so use_first_pass
auth sufficient pam_yubico.so id=1 authfile=/etc/yubi-map debug

Note that I have added the pam_yubico as a sufficient auth and also modified the pam_sepermit to use the user's initial password.

Step 4: We'll modify the /etc/ssh/sshd_config to allow challenge response and define the authentication method.
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

Optionally, disable the password auth as well
PasswordAuthentication no

Restart the sshd.
# systemctl sshd restart

Step 5: Now, here is the real catch. Yubikey needs to contact an authentication server before it can process. As far as I understand, we cannot use Yubikey when there is no internet (alternatively, you can run an authentication server in your own infra but more on that later). This also creates a problem, which is, that SELinux denies any network request during authentication. To handle the situation, Yubikey docs suggests setting a boolen.

# setsebool -P authlogin_yubikey 1