Friday, September 5, 2014

Types of NAT and How to determine the NAT Type

I am going to do a couple of posts on NAT (Network Address Translation) to discuss their classification and how to create a NAT on a Linux machine. This post will cover NAT types.

Generally NAT is used to allow private IPs to talk to the Internet. There are certain security aspects to it as well since outsiders cannot directly access the machines inside the NAT (well, not easily at least).

In general there are 4 kinds of NAT. Let us understand them one by one.
  • Full cone NAT: This is also known as one to one NAT. It is basically simple port forwarding where there is a static binding from client ip:port to NAT's ip:port and any one from Internet can write to NAT's ip:port and it will be forwarded to the client. This kind of NAT is used very infrequently. 
  • (Address) Restricted cone NAT: In this scenario, client can only receive packets from the host where it has already sent packets before. For example, if the client from the diagram above sends a packet to server with address then NAT will accept the reply from any port of the server as long as the source IP ( remains the same.
  • Port Restricted cone NAT: In this scenario, client can only receive packets from the host where it has already sent packets before as long as they come from the same server port. For example, if the client from the diagram above sends a packet to server with address on port 5555 then NAT will only accept reply originating from port 5555 from the server. This NAT is more restricted than Address Restricted NAT.
  • Symmetric NAT: In general all the above NAT types preserve the port. For example if the client is sending a packet from to, then NAT will usually map to preserving the port number. But in Symmetric NAT a random port is chosen for every new connection.This makes port prediction very difficult and techniques like UDP hole punching fails in this scenario
How do you tell what kind of NAT you are in? I have written a set of scripts to determine that. 

Run on a publicly accessible server and on the client inside the NAT. Make sure that UDP is allowed to public server's 5005 port (or you can change the port in the code).

If you see any bug in the scripts then please let me know in the comments or on the Github

Friday, August 22, 2014

Introduction To Ansible

I recently gave a talk about Ansible at Flock, Prague. Here is a youtube video of the same.

Paul W. Frields has written a summary of the talk on the Fedora Magazine.

Wednesday, April 16, 2014

A Simple Netcat How-To for Beginners

There are tonnes of tutorials on Netcat already. This one is to remind me and my colleagues about the awesomeness of nc which we forget on regular basis.
Common situations where nc can be used:
  • Check connectivity between two nodes. I had to learn hard way that ping (read all ICMP) based protocols are not always the best way to judge connectivity. Often ISPs set ICMP to lower priority and drop it.
  • Single file transfer.
  • Testing of network applications. I have written several clients and loggers for logstash and graphite which couldn't have been easier to test without nc.
  • Firing commands to remote servers where running a conventional tcp/http server is not possible (like VMWare ESXi)
Basic Netcat servers:
  • nc -l <port>
    Netcat starts listening for TCP sockets at the specified port. A client can connect and write arbitrary strings to the socket which will be reflected here.
  • nc -u -l <port>
    Netcat starts listening for UDP sockets at the specified port. A client can write arbitrary strings to the socket which will be reflected here.
  • nc -l <port> -e /bin/bash
    Netcat starts listening for TCP sockets at the specified port. A client can connect and write arbitrary commands which will be passed to /bin/bash and executed. Use with extreme caution on remote servers. The security here is nil.
  • nc -l -k <port> -e /bin/bash
    Problem with above command is that nc gets terminated as soon as client disconnects. -k option forces nc to stay alive and listen for subsequent connections as well.
Basic Netcat Clients:
  • nc <address> <port>
    Connect as client to the server running on <address>:<port> via TCP.
  • nc -u <address> <port>
    Connect as client to the server running on <address>:<port> via UDP.
  • nc -w <seconds> <address> <port>
    Connect as client to the server running on <address>:<port> via TCP and timeout after <seconds> of being idle. I used it a lot to send data to graphite using shell scripts.

A cool example to stream any file's content live (mostly used for logs) can be found at commandlinefu.

Monday, January 20, 2014

Using OpenStack Swift as ownCloud Storage Backend

ownCloud helps us to access our files from anywhere in the world, without take the control of data from us. Traditionally server's local hard disks have been used to act as storage backend but these days, as the latency of networks is decreasing, storing data over network is becoming cheaper and safer (in terms of recovery). ownCloud is capable of using SFTP, WebDAV, SMB, OpenStack Swift and several other storage mechanisms. We'll see the usage of OpenStack Swift with ownCloud in this tutorial

At this point, the assumption is that we already have admin access to an ownCloud instance and we have set up OpenStack Swift somewhere. If not, to setup OpenStack Swift, follow this tutorial.

Step 1: External storage facilities are provided by an app known as "External storage support", written by Robin Appelman and Michael Gapczynski, which ships with ownCloud and is available on the apps dashboard. It is disabled by default, we need to enable it.

Step 2: We need to go to Admin page of the ownCloud installation and locate "External Storage" configuration area. We'll select "OpenStack Swift" from the drop down menu.

Step 3: We need to fill in the details and credentials. We'll need the following information:
  • Folder Name: A user friendly name for the storage mount point.
  • user: Username of the Swift user (required)
  • bucket : Bucket can be any random string (required). It is a container where all the files will be kept.
  • region: Region (optional for OpenStack Object Storage).
  • key: API Key (required for Rackspace Cloud Files). This is not required for OpenStack Swift. Leave it empty.
  • tenant: Tenant name (required for OpenStack Object Storage). Tenant name would be the same tenant of which the Swift user is a part of. It is created using OpenStack Keystone.
  • password: Password of the Swift user (required for OpenStack Object Storage)
  • service_name: Service Name (required for OpenStack Object Storage). This is the same name which was used while creating the Swift service
  • url: URL of identity endpoint (required for OpenStack Object Storage). It is the Keystone endpoint against which authorization will be done.
  • timeout: Timeout of HTTP requests in seconds (optional)

Just to get a better hold on things, check out the image of an empty configuration form and here is a filled up one.

Notice that if ownCloud is successfully able to connect and authorize then a green circle appear on the left side of the configuration. In case things don't work out as expected then check out the owncloud.log in the data directory of ownCloud instance.

That is it. Now ownCloud is now ready to use OpenStack Swift to store data.

Sunday, January 12, 2014

OpenStack 101: How to Setup OpenStack Swift (OpenStack Object Storage Service)

In this tutorial we'll setup OpenStack Swift which is the object store service. Swift can be used to store data with high redundancy. The nodes in Swift can be broadly classified in two categories:
  • Proxy Node: This is a public facing node. It handles all the http request for various Swift operations like uploading, managing and modifying metadata. We can setup multiple proxy nodes and then load balance them using a standard load balancer.
  • Storage Node: This node actually stores data. It is recommended to make this node private, only accessible via proxy node but not directly. Other than storage service, this node also houses container service and account service which are used for managing mapping of containers and accounts respectively. 
For a small scale setup, both proxy and storage node can reside on the same machine but avoid doing so for a bigger setup.

Step 1: Let us install all the required packages for Swift:
# yum install openstack-swift openstack-swift-proxy openstack-swift-account openstack-swift-container openstack-swift-object memcached

Step 2: Attach a disk which would be used for storage or chop off some disk space from the existing disk.
Using additional disks:
Most likely this is done when there is large amount of data to be stored. XFS is the recommended filesystem and is known to work well with Swift. If the additional disk is attached as /dev/sdb then following will do the trick:
# fdisk /dev/sdb
# mkfs.xfs /dev/sdb1
# echo "/dev/sdb1 /srv/node/partition1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
# mkdir -p /srv/node/partition1
# mount /srv/node/partition1

Chopping off disk space from the existing disk:
We can chop off disk from existing disks as well. This is usually done for smaller installations or for "proof-of-concept" stage. We can use XFS like before or we can use ext4 as well.
# truncate --size=2G /tmp/swiftstorage
# DEVICE=$(losetup --show -f /tmp/swiftstorage)
# mkfs.ext4 $DEVICE
# mkdir -p /srv/node/partition1
# mount $DEVICE /srv/node/partition1 -t ext4 -o noatime,nodiratime,nobarrier,user_xattr

Step 3 (optional): Setup rsync to replicate the objects. In case replication or redundancy is not required, then  this step can be skipped.
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/
address = <storage_local_net_ip>

max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock

max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock

max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

Note that there can be multiple account, container and object sections if we wish to use multiple disks or partitions.
Enable rysnc in defaults and start the service:
# vim /etc/default/rsync
# service rsync start

Step 4: Setup the proxy node. The default config which is shipped with the Fedora 20 is good with minor changes. Open /etc/swift/proxy-server.conf and edit the [filter:authtoken] as below:
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
admin_tenant_name = admin
admin_user = admin
admin_password = ADMIN_PASS
auth_host =
auth_port = 35357
auth_protocol = http
signing_dir = /tmp/keystone-signing-swift

Keep in mind that the admin token, admin_tenant_name and admin_user should be same which was used while setting up Keystone. If you have not installed and setup Keystone already, then check out this tutorial before you proceed.

Step 5: Now we will create the rings. Rings are mappings between the storage node components and the actual physical drive. Note that the create commands below has 3 numeric parameters in the end. The first parameter signifies the number of the swift partitions (not same as the disk partitions). Higher number of partitions ensure even distribution but also higher number of partitions put higher strain on the server. So we have to find a good trade off. The rule of thumb is to create about 100 swift partitions per drive. For that the first numeric parameter would be 7 which is (2^7=128, closest to 100). The second parameter defines the number of copies to create for the sake of replication. For a small instance with no rsync, set it to one but recommended is three. Last number is the time in hours before a specific partition can be moved in succession. Set it to a low number for testing but 24 is recommended for production instances.
# cd /etc/swift
# swift-ring-builder account.builder create 7 1 1
# swift-ring-builder container.builder create 7 1 1
# swift-ring-builder object.builder create 7 1 1

Add the device created above to the ring:
# swift-ring-builder account.builder add z1- 100
# swift-ring-builder container.builder add z1- 100
# swift-ring-builder object.builder add z1- 100

Rebalance the ring. This will ensure even distribution and minimal partition moves.
# swift-ring-builder account.builder rebalance
# swift-ring-builder container.builder rebalance
# swift-ring-builder object.builder rebalance

Set the owner and the group for the partitions
# chown -R swift:swift /etc/swift /srv/node/partition1

Step 6: Create the service and end point using Keystone.
# keystone service-create --name=swift --type=object-store --description="Object Store Service"
|   Property  |              Value               |
| description |       Object Store Service       |
|      id     | b230a3ecd12e4a52954cb24502be9d07 |
|     name    |              swift               |
|     type    |           object-store           |

Copy the id from the output of the command above and use it to create the endpoint.
# keystone endpoint-create --region RegionOne --service_id b230a3ecd12e4a52954cb24502be9d07 --publicurl "\$(tenant_id)s" --adminurl --internalurl

Step 7: Start the services and test it:
# service memcached start
# for srv in account container object proxy  ; do sudo service openstack-swift-$srv start ; done
# swift -V 2.0 -A -U admin -K pass stat
IN_PASS stat
   Account: AUTH_939ba777082a4f988d5b70dc886459e3
Containers: 0
   Objects: 0
     Bytes: 0
Content-Type: text/plain; charset=utf-8
X-Timestamp: 1389435011.63658
X-Put-Timestamp: 1389435011.63658

Upload a file abc.txt to a Swift container myfiles like this:
# swift -V 2.0 -A -U admin -K pass upload myfiles abc.txt

The OpenStack Swift is ready to use.