tag:blogger.com,1999:blog-55568547481520455632024-03-13T09:38:31.204-07:00Learning Linux System Administration<center><b>Linux - Docker - Ansible - Fedora - CentOS - Enterprise Linux - Python - TCP/IP - DevOps - System Administration - Internet - Scaling - Hacking - Load Balancing - Uptime - High Availability - Cloud - Puppet</b></center>Aditya Patawarihttp://www.blogger.com/profile/11007675457270523326noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-5556854748152045563.post-60040664459512615292014-01-20T08:00:00.000-08:002014-01-20T08:00:02.368-08:00Using OpenStack Swift as ownCloud Storage Backend<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<a href="http://owncloud.org/" target="_blank">ownCloud</a> helps us to access our files from anywhere in the world, without take the control of data from us. Traditionally server's local hard disks have been used to act as storage backend but these days, as the latency of networks is decreasing, storing data over network is becoming cheaper and safer (in terms of recovery). ownCloud is capable of using SFTP, WebDAV, SMB, OpenStack Swift and several other storage mechanisms. We'll see the usage of <a href="http://swift.openstack.org/" target="_blank">OpenStack Swift</a> with ownCloud in this tutorial<br />
<br />
At this point, the assumption is that we already have admin access to an ownCloud instance and we have set up OpenStack Swift somewhere. If not, to setup OpenStack Swift, follow <a href="http://blog.adityapatawari.com/2014/01/openstack-101-how-to-setup-openstack_12.html" target="_blank">this tutorial</a>.<br />
<br />
<b>Step 1:</b> External storage facilities are provided by an app known as "External storage support", written by Robin Appelman and Michael Gapczynski, which ships with ownCloud and is available on the apps dashboard. It is disabled by default, we need to enable it.<br />
<br />
<b>Step 2:</b> We need to go to Admin page of the ownCloud installation and locate "External Storage" configuration area. We'll select "OpenStack Swift" from the drop down menu.<br />
<br />
<b>Step 3:</b> We need to fill in the details and credentials. We'll need the following information:<br />
<ul style="text-align: left;">
<li>Folder Name: A user friendly name for the storage mount point.</li>
<li>user: Username of the Swift user (required)</li>
<li>bucket : Bucket can be any random string (required). It is a container where all the files will be kept.</li>
<li>region: Region (optional for OpenStack Object Storage).</li>
<li>key: API Key (required for Rackspace Cloud Files). This is not required for OpenStack Swift. Leave it empty.</li>
<li>tenant: Tenant name (required for OpenStack Object Storage). Tenant name would be the same tenant of which the Swift user is a part of. It is created using OpenStack Keystone.</li>
<li>password: Password of the Swift user (required for OpenStack Object Storage)</li>
<li>service_name: Service Name (required for OpenStack Object Storage). This is the same name which was used while creating the Swift service</li>
<li>url: URL of identity endpoint (required for OpenStack Object Storage). It is the Keystone endpoint against which authorization will be done.</li>
<li>timeout: Timeout of HTTP requests in seconds (optional)</li>
</ul>
<br />
Just to get a better hold on things, check out the image of an <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9vfNhMm49UNWtq28FxwdptuauM_CJWK_MKb6RSxEujFgNNNUBeG9AM7j_50w_Xa8WnK-dcblGPh4uPFHrLZiVaeB9ozfCpSm-FNb58N0WEKh47rZE4-noXGSk5sO-I9HCUpe2pDo5uWE/s1600/ownCloud_OpenStack_Swift1.png" target="_blank">empty configuration</a> form and here is a <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_m2oGU5Vzf2sA6kGDOK-AFvUXpTr0H4R4UqV6XuMf6iMgVYs2gPGJOsKraduHzvCQZ3b3EiRd9SefMk12qLJiKjZ7PndA6oEJl14kqpbe3e2Vn5P1cnYqZtM5IfYUyrsr9hFLzUR07ZY/s1600/ownCloud_OpenStack_Swift2.png" target="_blank">filled up</a> one.<br />
<br />
Notice that if ownCloud is successfully able to connect and authorize then a green circle appear on the left side of the configuration. In case things don't work out as expected then check out the owncloud.log in the data directory of ownCloud instance.<br />
<br />
That is it. Now ownCloud is now ready to use OpenStack Swift to store data.</div>
Aditya Patawarihttp://www.blogger.com/profile/04110480749979714191noreply@blogger.com1tag:blogger.com,1999:blog-5556854748152045563.post-25954819027213100052014-01-12T09:00:00.000-08:002014-01-12T09:19:09.325-08:00OpenStack 101: How to Setup OpenStack Swift (OpenStack Object Storage Service)<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
In this tutorial we'll setup <a href="http://docs.openstack.org/developer/swift/" target="_blank">OpenStack Swift</a> which is the object store service. Swift can be used to store data with high redundancy. The nodes in Swift can be broadly classified in two categories:<br />
<ul style="text-align: left;">
<li><b>Proxy Node:</b> This is a public facing node. It handles all the http request for various Swift operations like uploading, managing and modifying metadata. We can setup multiple proxy nodes and then load balance them using a standard load balancer.</li>
<li><b>Storage Node:</b> This node actually stores data. It is recommended to make this node private, only accessible via proxy node but not directly. Other than storage service, this node also houses container service and account service which are used for managing mapping of containers and accounts respectively. </li>
</ul>
<div>
For a small scale setup, both proxy and storage node can reside on the same machine but avoid doing so for a bigger setup.</div>
<div>
<br /></div>
<b>Step 1:</b> Let us install all the required packages for Swift:<br />
<code># yum install openstack-swift openstack-swift-proxy openstack-swift-account openstack-swift-container openstack-swift-object memcached</code><br />
<br />
<b>Step 2:</b> Attach a disk which would be used for storage or chop off some disk space from the existing disk.<br />
Using additional disks:<br />
Most likely this is done when there is large amount of data to be stored. <a href="http://en.wikipedia.org/wiki/XFS" target="_blank">XFS</a> is the recommended filesystem and is known to work well with Swift. If the additional disk is attached as /dev/sdb then following will do the trick:<br />
<code>
# fdisk /dev/sdb<br />
# mkfs.xfs /dev/sdb1<br />
# echo "/dev/sdb1 /srv/node/partition1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
<br />
# mkdir -p /srv/node/partition1<br />
# mount /srv/node/partition1<br />
<br />
</code>
Chopping off disk space from the existing disk:<br />
We can chop off disk from existing disks as well. This is usually done for smaller installations or for "proof-of-concept" stage. We can use XFS like before or we can use <a href="http://en.wikipedia.org/wiki/Ext4" target="_blank">ext4</a> as well.<br />
<code>
# truncate --size=2G /tmp/swiftstorage<br />
# DEVICE=$(losetup --show -f /tmp/swiftstorage)<br />
# mkfs.ext4 $DEVICE<br />
# mkdir -p /srv/node/partition1<br />
# mount $DEVICE /srv/node/partition1 -t ext4 -o noatime,nodiratime,nobarrier,user_xattr<br />
<br />
</code>
<b>Step 3 (optional):</b> Setup <a href="http://en.wikipedia.org/wiki/Rsync" target="_blank">rsync</a> to replicate the objects. In case replication or redundancy is not required, then this step can be skipped.<br />
<div>
<code>
uid = swift<br />
gid = swift<br />
log file = /var/log/rsyncd.log<br />
pid file = /var/run/rsyncd.pid<br />
address = <storage_local_net_ip><br />
<br />
[account]<br />
max connections = 2<br />
path = /srv/node/<br />
read only = false<br />
lock file = /var/lock/account.lock<br />
<br />
[container]<br />
max connections = 2<br />
path = /srv/node/<br />
read only = false<br />
lock file = /var/lock/container.lock<br />
<br />
[object]<br />
max connections = 2<br />
path = /srv/node/<br />
read only = false<br />
lock file = /var/lock/object.lock<br />
</code></div>
</div>
<br />
Note that there can be multiple account, container and object sections if we wish to use multiple disks or partitions.<br />
Enable rysnc in defaults and start the service:<br />
<code>
# vim /etc/default/rsync<br />
RSYNC_ENABLE = true
<br />
# service rsync start<br />
<br />
</code>
<b>Step 4:</b> Setup the proxy node. The default config which is shipped with the Fedora 20 is good with minor changes. Open /etc/swift/proxy-server.conf and edit the [filter:authtoken] as below:<br />
<code>
[filter:authtoken]<br />
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory<br />
admin_tenant_name = admin<br />
admin_user = admin<br />
admin_password = ADMIN_PASS<br />
auth_host = 127.0.0.1<br />
auth_port = 35357<br />
auth_protocol = http<br />
signing_dir = /tmp/keystone-signing-swift<br />
</code><br />
<div>
Keep in mind that the admin token, admin_tenant_name and admin_user should be same which was used while setting up <a href="http://docs.openstack.org/developer/keystone/" target="_blank">Keystone</a>. If you have not installed and setup Keystone already, then check out <a href="http://blog.adityapatawari.com/2014/01/openstack-101-how-to-setup-openstack.html" target="_blank">this tutorial</a> before you proceed.</div>
<div>
<br /></div>
<b>Step 5:</b> Now we will create the rings. Rings are mappings between the storage node components and the actual physical drive. Note that the create commands below has 3 numeric parameters in the end. The first parameter signifies the number of the swift partitions (not same as the disk partitions). Higher number of partitions ensure even distribution but also higher number of partitions put higher strain on the server. So we have to find a good trade off. The rule of thumb is to create about 100 swift partitions per drive. For that the first numeric parameter would be 7 which is (2^7=128, closest to 100). The second parameter defines the number of copies to create for the sake of replication. For a small instance with no rsync, set it to one but recommended is three. Last number is the time in hours before a specific partition can be moved in succession. Set it to a low number for testing but 24 is recommended for production instances.<br />
<code># cd /etc/swift<br /># swift-ring-builder account.builder create 7 1 1<br /># swift-ring-builder container.builder create 7 1 1<br /># swift-ring-builder object.builder create 7 1 1<br />
</code><br />
Add the device created above to the ring:<br />
<code># swift-ring-builder account.builder add z1-127.0.0.1:6002/partition1 100<br /># swift-ring-builder container.builder add z1-127.0.0.1:6001/partition1 100<br /># swift-ring-builder object.builder add z1-127.0.0.1:6000/partition1 100<br />
</code>
<br />
Rebalance the ring. This will ensure even distribution and minimal partition moves.<br />
<code># swift-ring-builder account.builder rebalance<br /># swift-ring-builder container.builder rebalance<br /># swift-ring-builder object.builder rebalance<br />
</code>
<br />
Set the owner and the group for the partitions<br />
<code># chown -R swift:swift /etc/swift /srv/node/partition1<br />
</code><br />
<b>Step 6:</b> Create the service and end point using Keystone.<br />
<code># keystone service-create --name=swift --type=object-store --description="Object Store Service"<br />
+-------------+----------------------------------+<br />
| Property | Value |<br />
+-------------+----------------------------------+<br />
| description | Object Store Service |<br />
| id | b230a3ecd12e4a52954cb24502be9d07 |<br />
| name | swift |<br />
| type | object-store |<br />
+-------------+----------------------------------+<br />
</code><br />
<div>
Copy the id from the output of the command above and use it to create the endpoint.</div>
<div>
</div>
<code># keystone endpoint-create --region RegionOne --service_id b230a3ecd12e4a52954cb24502be9d07 --publicurl "http://127.0.0.1:8080/v1/AUTH_\$(tenant_id)s" --adminurl http://127.0.0.1:8080/v1 --internalurl http://127.0.0.1:8080/v1</code>
<br />
<div>
<br /></div>
<div>
<b>Step 7: </b>Start the services and test it:</div>
<code># service memcached start<br /># for srv in account container object proxy ; do sudo service openstack-swift-$srv start ; done<br /># swift -V 2.0 -A http://127.0.0.1:5000/v2.0 -U admin -K pass stat
<br />
IN_PASS stat<br />
Account: AUTH_939ba777082a4f988d5b70dc886459e3<br />
Containers: 0<br />
Objects: 0<br />
Bytes: 0<br />
Content-Type: text/plain; charset=utf-8<br />
X-Timestamp: 1389435011.63658<br />
X-Put-Timestamp: 1389435011.63658<br />
</code><br />
Upload a file abc.txt to a Swift container myfiles like this:<br />
<code># swift -V 2.0 -A http://127.0.0.1:5000/v2.0 -U admin -K pass upload myfiles abc.txt</code>
<br />
<code><br /></code>
<br />
<div>
The OpenStack Swift is ready to use.</div>
</div>
Aditya Patawarihttp://www.blogger.com/profile/04110480749979714191noreply@blogger.com4tag:blogger.com,1999:blog-5556854748152045563.post-11067340853748751612014-01-11T10:22:00.000-08:002014-01-11T10:29:17.511-08:00OpenStack 101: How to Setup OpenStack Keystone (OpenStack Identity Service)<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://docs.openstack.org/developer/keystone/">OpenStack Keystone</a> is an identity or authorization service. Before we can do anything on other OpenStack components, we have to authorize ourselves and only then the operation can proceed. Let us get acquainted with some terminologies before we proceed.<br />
<ul style="text-align: left;">
<li>Token: An alphanumeric string which allows access to certain set of services depending up on the access level (role) of the user.</li>
<li>Service: An OpenStack service like Nova, Swift and Keystone itself.</li>
<li>Tenant: A group of users. </li>
<li>Endpoint: A URL (may be private) used to access the service.</li>
<li>Role: The authorization level of a user.</li>
</ul>
<div>
Let us go ahead and build the Keystone service for our use.</div>
<div>
<br /></div>
<div>
<b>Step 1:</b> Fedora 20 has OpenStack Havan in its repositories so install it is not a pain at all. Additionally, we need MySQL (replaced by MariaDB in Fedora 20) where Keystone will save its data.</div>
<div>
<code># yum install openstack-utils openstack-keystone mysql-server</code></div>
<br />
<div>
<div>
<b>Step 2:</b> Once the packages above are installed, we need to set a few things in keystone config. Find the lines and edit them to look like these:</div>
<div>
<code># vim /etc/keystone/keystone.conf</code></div>
<div>
<code>[DEFAULT]</code></div>
<div>
<code>admin_token = <i>ADMIN_TOKEN</i></code></div>
<div>
<code>.</code></div>
<div>
<code>.</code></div>
<div>
<code>.</code></div>
<div>
<code>[sql]</code></div>
<div>
<code>connection = mysql://keystone:<i>KEYSTONE_DBPASS</i>@controller/keystone</code></div>
<br />
<div>
Note that<i> ADMIN_TOKEN </i>and <i>KEYSTONE_DBPASS</i> should be long and difficult to guess. Remember that <i>ADMIN_TOKEN</i> is the almighty token which will have full access to create and destroy users and services. Also several tutorials and the official docs use command <code>openstack-config --set /etc/keystone/keystone.conf</code> to do the changes that we just did manually. I do not recommend using the command. It created duplicate sections and entries for me which can be confusing down the line. </div>
<div>
<b><br /></b>
<b>Step 3:</b> Set up MySQL/MariaDB (only required for the first run of MySQL) to set root password. </div>
<div>
<code># mysql_secure_installation</code></div>
<div>
<br /></div>
<div>
Now we need to create the required database and tables for Keystone to work. The command below will do that for us. It will ask us for root password to create the keystone use and the database.<br />
<code># openstack-db --service keystone --init --password KEYSTONE_DBPASS</code></div>
<div>
<br />
<b>Step 4:</b> Create the signing keys and certificates for the tokens.</div>
<div>
<code>
# keystone-manage pki_setup --keystone-user keystone --keystone-group keystone</code></div>
</div>
<div>
<br />
<b>Step 5:</b> Set the file owners, just in case something messed up and start the service.<br />
<code>
# chown -R keystone:keystone /etc/keystone/* /var/log/keystone/keystone.log<br />
# service openstack-keystone start<br />
# chkconfig openstack-keystone on<br />
</code>
</div>
<div>
<br />
<b>Step 6:</b> Setup the required environment variables. This will save the effort of supplying all the information every time a Keystone command is executed. Note that by default the Keystone admin port is 35357. This can be changed in <span style="font-family: monospace;">/etc/keystone/keystone.conf.</span></div>
<div>
<code># cat > ~/.keystonerc <<EOF</code></div>
<code>
</code>
<br />
<div>
<code>> export OS_SERVICE_TOKEN=ADMIN_TOKEN</code></div>
<code>
</code>
<div>
<code>> export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0</code></div>
<code>
<div>
> export OS_USERNAME=admin</div>
<div>
> export OS_PASSWORD=ADMIN_PASS</div>
<div>
> export OS_TENANT_NAME=admin</div>
<div>
> export OS_AUTH_URL=http://127.0.0.1:35357/v2.0</div>
<div>
> EOF</div>
<div>
# . ~/.keystonerc<br />
<br /></div>
</code><b>
Step 7:</b> Create the tenants, users and the Keystone service with endpoint.<br />
Creating the tenant:<br />
<div>
<code># keystone tenant-create --name=admin --description="Admin Tenant"</code></div>
<div>
<br />
Creating the admin user:</div>
<div>
<code># keystone user-create --name=admin --pass=ADMIN_PASS --email=admin@example.com</code><br />
<br />
Creating and adding admin user to admin role:</div>
<div>
<code># keystone role-create --name=admin</code></div>
<code>
</code>
<div>
<code># keystone user-role-add --user=admin --tenant=admin --role=admin</code></div>
<code>
</code>
<br />
<div>
Creating Keystone service and endpoint:<br />
<code>
# keystone service-create --name=keystone --type=identity --description="Keystone Identity Service"<br />
+-------------+--------------------------------------+<br />
| Property | Value |<br />
+-------------+--------------------------------------+<br />
| description | Keystone Identity Service |<br />
| id | c3dbb8aa4b27492f9c4a663cce0961a3 |<br />
| name | keystone |<br />
| type | identity |<br />
+-------------+--------------------------------------+<br />
<br />
</code>
Copy the id from the command above and use it in the command below:</div>
<div>
<code># keystone endpoint-create --service-id=c3dbb8aa4b27492f9c4a663cce0961a3 --publicurl=http://127.0.0.1:5000/v2.0 --internalurl=http://127.0.0.1:5000/v2.0 --adminurl=http://127.0.0.1:35357/v2.0</code><br />
<br />
<b>Step 8:</b> Test the keystone service.</div>
<div>
<code># unset OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT</code></div>
<code>
</code>
<div>
<code># keystone --os-username=admin --os-password=ADMIN_PASS --os-auth-url=http://127.0.0.1:35357/v2.0 token-get</code></div>
<code>
</code>
<br />
A token with id, validity and other information will be returned.<br />
<br />
Keystone is up and running. We'll create some services next tutorial.
</div>
Aditya Patawarihttp://www.blogger.com/profile/04110480749979714191noreply@blogger.com4tag:blogger.com,1999:blog-5556854748152045563.post-8225298459592944162014-01-11T10:15:00.000-08:002014-01-12T09:15:39.005-08:00OpenStack 101: What is OpenStack?<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://www.openstack.org/" target="_blank">OpenStack</a>, in simple words, is an open source project which facilitates us to build our own cloud computing setup. In other words, it creates an Infrastructure as a Service (IaaS) on our own infrastructure. We can have <a href="http://aws.amazon.com/" target="_blank">Amazon AWS</a> like service up and running quite fast and painlessly wherever we want. A lot of efforts have been taken to ensure that code written for Amazon AWS can be ported to any OpenStack installation easily.<br />
<br />
Below is a small comparison (not exhaustive) between major OpenStack services and Amazon AWS to give you an idea about the compatibility.<br />
<br />
<center>
<table border="2" bordercolor="#000000" cellpadding="3" cellspacing="3" style="width: 100%px;">
<tbody>
<tr>
<th>OpenStack Service</th>
<th>Amazon AWS Service</th>
</tr>
<tr>
<td>Nova</td><td>EC2</td>
</tr>
<tr>
<td>Cinder</td>
<td>EBS</td>
</tr>
<tr>
<td>Swift</td>
<td>S3</td>
</tr>
<tr>
<td>Keystone</td>
<td>IAM</td>
</tr>
<tr>
<td>Glance</td>
<td>AMI</td>
</tr>
<tr>
<td>Horizon</td>
<td>AWS Web Console</td>
</tr>
<tr>
<td>Neutron</td>
<td>EC2 network components</td>
</tr>
</tbody></table>
</center>
<br />
OpenStack 101 is a tutorial series to simplify using OpenStack and integration of OpenStack with simple applications. It'll help you create OpenStack installations for "proof -of-concept" stage or hosting small IaaS service. For most of the part I have tried to keep the tutorials as close to official documentation as possible. Let me also state this loud and clear, <a href="http://docs.openstack.org/" target="_blank">OpenStack's documentation</a> is really great. If you can, the please go through it. If you are done with "proof-of-concept" and are going to run production ready machines, then go through the official documentation. These tutorials will help you get started but are not a replacement for the docs.<br />
<br />
I am going to use <a href="http://www.openstack.org/software/havana/" target="_blank">OpenStack Havana</a> and will run it on <a href="https://fedoraproject.org/" target="_blank">Fedora 20</a> (latest at the time of writing, January 2014). All the commands and codes are tested well before putting them up here but if you see any errors, please point them out to me.<br />
<br />
Contents:<br />
<a href="http://blog.adityapatawari.com/2014/01/openstack-101-what-is-openstack.html" target="">OpenStack 101: What is OpenStack?</a><br />
<a href="http://blog.adityapatawari.com/2014/01/openstack-101-how-to-setup-openstack.html">OpenStack 101: How to Setup OpenStack KeyStone (OpenStack Identity Service)</a><br />
<a href="http://blog.adityapatawari.com/2014/01/openstack-101-how-to-setup-openstack_12.html">OpenStack 101: How to Setup OpenStack Swift (OpenStack Object Storage Service)</a></div>
Aditya Patawarihttp://www.blogger.com/profile/04110480749979714191noreply@blogger.com0