Installing ownCloud on Raspberry Pi

Presenting a ready-to-install image of ownCloud for Raspberry Pi 

A small introduction to ownCloud

ownCloud is an application which enables users to share their data without giving control to any third party posing as a facilitator. While sharing the data without loosing control is the main objective, ownCloud is much more than that. It can also rapidly sync the data, contacts, calendar events etc from several devices. It can work with several custom backends and it is highly flexible.

Many of us have a Raspberry Pi with us and we love playing with it. In past I have written posts on how to install Arch Linux on it and how to install OpenELEC to convert the Raspberry Pi into a Media Centre. This time I plan to go a little further. This time I have made a custom image which comes preinstalled with ownCloud and some tweaks to improve the ownCloud experience with Raspberry Pi. This image is based on Raspbian Wheezy.

Just follow the steps below and you'll be good to go in no time:

1. Download the archived image from in either zip format (usually for Windows) or gunzip format (usually for Linux and Unix like platforms)
   Since I am running on Linux, I would download gunzip format.
2. Extract it and put it on a SD card using dd or any other tool or command. Check out this article on elinux if you need any help for this. Although 2 GB SD card would be fine but I would recommend using 4 GB or more.
   I would run the following commands:
   $ gunzip owncloud-raspberrypi-0.1.img.gz # to extract the gz archive
$ sudo dd bs=1M if=owncloud-raspberrypi-0.1.img of=/dev/mmcblk0 # to write to the SD card. /dev/mmcblk0 can be obtained by the output of df command.
3. Put this SD card in your Raspberry Pi and boot. The default credentials are:
   user: pi
   password: owncloud
4. Run raspi-config and follow the directions to expand the filesystem to enjoy maximum disk space. Reboot, if required.
5. Run ifconfig to get the ip address of the Raspberry Pi.

That is it. Just open http://<ip_address>/owncloud and create the admin user and explore ownCloud on Raspberry Pi.

This image PHP execution time increased to 60 seconds and the upload limit has been bumped up to 500M. The Apache is set to allow .htaccess for the protection of data directory. Also SSH has been enabled by default.

The official page for the image can be found at ownCloud on Raspberry Pi. A Hacker News discussion is also going on here.

If you like this image and you are interested in knowing more about ownCloud, then please consider buying my book, Getting Started with ownCloud. It is available from Amazon.com, Amazon.co.uk, Barnes & Nobles and on Kindle.

Converting Raspberry Pi into Media Center Using OpenELEC (XMBC)

One of the most discussed and common use of Raspberry Pi is to turn it into a media center for music and videos. Raspberry Pi uses very less power so it is an ideal device for people who listen to music all the time. Talking about media center, XMBC is a very popular choice with Linux users these days. It is a full fledged media center operating system, running Linux at its core. It comes packed with all the common codecs and presents a very pleasing user interface and playlists to organize the videos and music.
Sounds good? But the catch is that it can be too heavy for a Raspberry Pi. So we need to look for a lighter alternative.

OpenELEC (Open Embedded Linux Entertainment Center) is an appliance which means that almost everything will be pre-configured. OpenELEC takes an easy XMBC and makes it even easier to install and maintain. So let us start installing it on our SD card for Raspberry Pi. Latest version of the OpenELEC can be downloaded from their downloads page in .tar.bz2 form. Once downloaded, we need to extract this archive to obtain the .img file.
$ tar -xjvf OpenELEC-RPi.arm-3.0.6.tar.bz2

After the extraction of img file is done, we need to dd this file into the SD card. To do this, put the SD card into the right slot in the computer. Run df command to see if it gets auto mounted and note the path of the device file. If it does not get auto mounted then, on terminal, type ls /dev/mm*. This will list all the memory cards in your system. Once you have got this information, now run dd to install the OpenELEC into the SD card.
# dd bs=4M if=OpenELEC-RPi.arm-3.0.6.img of=/dev/mmcblk0

This may take a few minutes. It performs byte by byte copy of the img file to the SD card. Once we are done with this step, we can just insert the SD card into the slot and fire up the Raspberry Pi. First thing which I noticed is that OpenELEC is quite fast to boot. Now just insert a thumb drive with any music or videos into the Raspberry Pi and use a standard keyboard/mouse to browse and choose the media.

I had a chance to go through Mikkel Viager's Instant OpenELEC Starter. It has much more detailed explanation about installing and maintaining OpenELEC. It also talks about installation on non-Raspberry Pi platforms and provides with handy tips to manage XMBC. I liked the feature of auto indexing of movies and tv shows and XMBC management using an Android phone remotely.

Off to watch a movie now! :-D

Deploying Big Using BitTorrent [Sharing Files Using BitTorrent]

If you just want to share some files without concern of privacy, please check out this short tutorial on bittorrent.com. This article will talk a bit about BitTorrent's basic internals and it's usage to do large code/application deploys.

Scenario: I have to do deploy some application(s) across many co-located data centers. The collective size of deploy will be of the order of tens of GB.

Conventional methods like scp, rsync and http fails:

• scp will not resume if it breaks at any point. Every time I will have to start over and over again.
• rsync works well with text files, not so well with binaries (it works nonetheless). The amount of CPU it  eats is unacceptable though.
• http can resume most of the times but as more servers try to download the application, the bandwidth limitations slow down the entire process.

Enter BitTorrent! 

• Resumes the download every time. No problem if the connection breaks.
• Does not eats my CPU.
• As more servers download, they can act as seeder and actually increase collective bandwidth.

Now let us start the technical details. For torrent to work, you will need to create a torrent file (also known as a metafile). You'll also need a tracker. Tracker keeps track of what all leechers and seeders (collectively known as peers) are there and help in general coordination by announcing the available peers periodically. Finally you will need a torrent client which can seed the files you are going to share. 

Now the problem is that BitTorrent is no longer open sourced. So either you have to get license from BitTorrent, Inc. which can be very costly (I am not sure) or you can use the older code which was once open source and still works like a charm.

For Centos/Red Hat/Scientific Linux, you should try NauLinux School repo:
# vim /etc/yum.repos.d/naulinux-school.repo:
[naulinux-school]
name=NauLinux School
baseurl=http://downloads.naulinux.ru/pub/NauLinux/6.2/$basearch/sites/School/RPMS/
enabled=0
gpgcheck=1
gpgkey=http://downloads.naulinux.ru/pub/NauLinux/RPM-GPG-KEY-linux-ink

Install bittorrent rpm package:
# yum --enablerepo=naulinux-school install bittorrent

For Fedora, you can try downloading the rpm from their build system koji and manually install it.
# yum localinstall ./bittorrent-4.4.0-16.fc15.noarch.rpm

Also install mktorrent which will be used to create torrent meta files.
# yum install mktorrent

Creating a torrent tracker
As I have mentioned before, tracker is a critical piece of the bittorrent setup. It helps in co-ordinating between the peers and maintains a list of the same. It also keeps a record of all the seeds along with the checksum of the torrent. Needless to say that without a torrent tracker, entire bittorrent setup will fail.
You can setup a tracker for yourself easily. Just run the following command on CentOS:
$ bittorrent-tracker  --port 8080 --dfile dstate --logfile tracker.log

For Fedora, you can use the bttrack command after installing the bittorent package:
$ bttrack --port 8080 --dfile dstate --logfile tracker.log

Alternatively, you can use one of the public tracker like OpenBitTorrent. This may save you sometime.

Creating a torrent metafile
Once we have the tracker up, we need to create the actual torrent file to distribute. A torrent file contains bencoded data about the files and the announce URL of the tracker along with some other information.
Creating torrent using mktorrent easy but if you prefer GUI, you can use transmission or any other bittorrent client.
$ mktorrent -a http://tracker.example.com:8080/announce -l 18 -v /path/to/the/app

Here -a specifies the tracker's announce url which we created before. -l flag specifies the size of each chunk of file which will be transferred at a time and -v flag is for verbosity.

Once the torrent metafile is created, you need to seed the torrent so that other peers can download it. I like to use rtorrent for this:
# yum install rtorrent
$ rtorrent <path to the torrent metafile>

Here is an easy-to-follow tutorial, if you are more interested in rtorrent.

Tips for peaceful life
There are certain parameters that can be tweaked for better performance. While making the torrent try adjusting the -l flag to a higher value if you have really good bandwidth. Since my deployment was for a bunch of data centers which have really good bandwidth, I usually set it up to 20.

If you do the deploys without taking out the machines from production, it is possible to limit the bandwidth usage of torrent client. This comes really handy and helps in avoiding the clogging of network pipes. Check out the tutorials and docs of your torrent client to know about these controls.

Before initiating the transfer, always make sure that you inform the relevant data center technicians and network operations guys. I did not, the first time, and due to huge spike in network, the one of the data center ops thought that we are under some sort of DOS attack and cut off connectivity to all our servers resulting in minor service disruption.

Happy deploying!

Discuss this post on Hacker News.

Arch Linux on Raspberry Pi Running XFCE [Version 2]

I have created a one-liner to install XFCE on Arch Linux, Raspberry Pi. Find it in the last line.

I wrote about installing XFCE on Arch Linux running on Raspberry Pi but it seems that those instructions are no longer valid for the new version of Arch as hosted on Raspberry Pi download page. So below are the new instructions for installing XFCE on Arch, Raspberry Pi. Please note that initial steps are similar for the previous versions of Arch too.

First off download the latest Arch Linux ARM from Raspberry Pi downloads page and unzip it to extract the img file. Once you have the img file, you need to write this on a SD Card. You can use dd command or tools like ImageWriter. There are more options available on elinux page. Let us use dd command for now:
# dd bs=4M if=~/archlinux-hf-2012-09-18.img of=/dev/mmcblk0

No, cp command is not supposed to be used here because cp copies over the file system and we have to do something at much more lower level. In case you are wondering how I got the /dev/mmcblk0 bit, I just mounted the sd card and check the output of df -h command. If you are using a sd card of more than 2G memory, then I recommend using gparted or anything else and expand the size of the file system since by default it'll be just about 2G and rest of your space will go unused. Once you are done here, insert the sd card into your Pi and fire it up.
Now you can see the awesome black login screen. The default password for root user is 'root'. Login as root and create pacman, the Arch Package Manager, database.
# pacman-key --init

Some randomness would be helpful here. So hit ALT+F2 to go to another tty and execute some random commands like ls and echo and cd etc. Switch back to the previous tty by hitting ALT+F1 and wait till the initialization of db is done. Now you can update your repositories:
# pacman -Syu

 Let us install Xorg libraries first:
# pacman -S xorg-xinit xorg-server xorg-server-utils xterm
This will get us the basic X server and related dependencies

Next, we will install XFCE:
# pacman -S xfce4
The CLI will ask you if you want to install selected packages only. I choose to install everything since they looked bare minimum anyway but you can be choosy here.

Now we may need the display drivers:
# pacman -S mesa xf86-video-fbdev xf86-video-vesa

Also we will need a login manager. I use SLiM since it is lightweight:
# pacman -S slim

Next we need to enable SLiM and graphics user mode (systemd lingo for runlevel 5):
# systemctl enable slim.service
# systemctl enable graphical.target

And we have to create a .xinitrc in the user's home directory. This file reads X server configs and starts XFCE environment:
# vim ~/.xinitrc 

#!/bin/sh 
if [ -d /etc/X11/xinit/xinitrc.d ]; then
for f in /etc/X11/xinit/xinitrc.d/*; do
[ -x "$f" ] && . "$f"
done
unset f
fi
exec startxfce4

Also we need a ~/.bash_profile to execute startx to initiate the Xserver as soon as the user (root in this case) logs in.  :
# vim ~/.bash_profile
[[ -z $DISPLAY && $XDG_VTNR -eq 1 ]] && exec startx

That is it! Reboot and enjoy XFCE on Raspberry Pi.

To save you some time, I have combined these commands in a small shell script and put it on github (fork it). So now, to install XFCE on your Pi, you need to fire just one command:

curl https://raw.github.com/adimania/arch-desktop-environments/master/XFCE-Arch-RPi.sh | bash


Discuss this post on Hacker News.

All About inodes, Hard Links and Soft Links

Open your terminal and fire "ls -i" and you will see that each file is associated with a number.
$ ls -i
2889973 users.sh 2889972 fedoraRepo.sh 2889970 sfs.sh
2889969 bigFile.sh 2889971 dbBackup.sh 2889714 tree-clone.py

Ever wondered what this number is?
Ever thought what happens when a file is deleted?
How does the system knows the owner of the file or it's last modification time?
What are hard links?
What is the difference between hard links and soft links?

I'll try to answer these questions and probably more but I want to stress on a point; every thing in Linux is a file including the directories and devices attached.
Also install a package called sleuthkit using yum or apt to obtain a tool called istat.

When a filesystem (considering ext3/ext4 for now) is created on a disk, a special data structure is created. We'll call this inode table (technically it is an array of structure). It is indexed from 1 to n where n is the maximum numbers of inodes in the filesystem. Details like maximum number of inodes are decided while creating the filesystem usually. We can run "df -i" to check out number of inodes used and available.
Now whenever we create a file or a directory, an unallocated inode number is assigned and this is where several details about the file or the directory is stored. POSIX standard requires inode to contain the following information (borrowed from Wikipedia):

• The size of the file in bytes.
• Device ID (this identifies the device containing the file).
• The User ID of the file's owner.
• The Group ID of the file.
• The file mode which determines the file type and how the file's owner, its group, and others can access the file.
• Additional system and user flags to further protect the file (limit its use and modification).
• Timestamps telling when the inode itself was last modified (ctime, inode change time), the file content last modified (mtime, modification time), and last accessed (atime, access time).
• A link count telling how many hard links point to the inode.
• Pointers to the disk blocks that store the file's contents (see inode pointer structure).

Notice that this does not include the filename. Surprised? In fact inodes never hold that information. So an obvious question arises, when we open file, how does the system know what inode it is associated with? To understand this, we need to understand what exactly is a directory. As I have mentioned, everything in Linux is a file which implies that even directory is a file. Every directory consist of a data structure, first part of which holds an inode number and the second part holds the file name. So when we try to perform any operation on a file, a recursive traversal is performed to lookup for inode number against that file name and that is how inode is obtained.

Among the information contained in inode, a link count and size is maintained. When we delete a file the link count is decreased until it reaches zero where the size of the file is also set to zero. See the example:
# ls -i abc
2891791 abc

# istat /dev/sda5 2891791
inode: 2891791
Allocated
Group: 353
Generation Id: 3534721592
uid / gid: 1000 / 1000
mode: rrw-rw-r--
Flags:
size: 6
num of links: 1

Inode Times:
Accessed: 2013-03-29 01:16:46 (IST)
File Modified: 2013-03-29 01:16:46 (IST)
Inode Modified: 2013-03-29 01:16:46 (IST)

Direct Blocks:
127754

# ln abc def
# istat /dev/sda5 2891791
inode: 2891791
Allocated
Group: 353
Generation Id: 3534721592
uid / gid: 1000 / 1000
mode: rrw-rw-r--
Flags:
size: 6
num of links: 2

Inode Times:
Accessed: 2013-03-29 01:18:41 (IST)
File Modified: 2013-03-29 01:16:46 (IST)
Inode Modified: 2013-03-29 01:18:34 (IST)

Direct Blocks:
127754

# rm abc
# istat /dev/sda5 2891791
inode: 2891791
Allocated
Group: 353
Generation Id: 3534721592
uid / gid: 1000 / 1000
mode: rrw-rw-r--
Flags:
size: 6
num of links: 1

Inode Times:
Accessed: 2013-03-29 01:18:41 (IST)
File Modified: 2013-03-29 01:16:46 (IST)
Inode Modified: 2013-03-29 01:18:57 (IST)

Direct Blocks:
127754 

So the "num of links" increased when we created a hardlink using ln command by one. When we deleted the file using rm command, the "num of links" decreased by one. If we delete the def file then the count and size will be set to zero.

# istat /dev/sda5 2891791
inode: 2891791
Not Allocated
Group: 353
Generation Id: 3534721592
uid / gid: 1000 / 1000
mode: rrw-rw-r--
Flags:
size: 0
num of links: 0

Inode Times:
Accessed: 2013-03-29 01:18:41 (IST)
File Modified: 2013-03-29 01:30:10 (IST)
Inode Modified: 2013-03-29 01:30:10 (IST)
Deleted: 2013-03-29 01:30:10 (IST)

Direct Blocks:

This brings us to our next topic of discussion, what are hard links and what are soft links? Simply putting, hard link of a file holds the inode of that file where as the soft link of the file is just a reference to another file. If we delete the original file but have a hard link to it, then we can still access the contents using the hard link. If we delete the original file, the soft link is pretty much useless since all it did was to point to the original name which contained the inode. A crude way to depict what I am saying is below. See how both "Original Name" and "Hard Link" is pointing to the inode but "Soft Link" is not.
Original Name ---------> inode <--------- Hard Link
Soft Link ----------> Original Name -----------> inode

Now, soft links have their own importance. We cannot use hard links to point to files across different filesystems but with soft links we can. This comes really handy when you want to maintain one name regardless of version differences. See how /usr/bin/python actually points to another binary.

# ls -l /usr/bin/python
lrwxrwxrwx. 1 root root 7 Jan 9 22:45 /usr/bin/python -> python2



Honestly, there are many more creative uses of links. If you are interested then I recommend that you check out how BusyBox implements a lot of commands using a single binary.
(Hint: $0 is the name of the script which is passed as a variable to the binary)

Update: As mentioned by reirob on HackerNews, there is a particular case where deleting the file does not set the "num of links" to zero. This usually happens when there is a process which is writing to the file. I have encountered this a few times myself when I delete a log file but the server writing it hasn't been restarted.

Arch Linux on Raspberry Pi Running XFCE

Instructions in this post are no longer valid. Please find the updated post here.

I recently got a Raspberry Pi from RS online store. I wanted one so bad and it took so long before I got to play with it that by the time I got it, I was pretty much drooling over it. I started off by installing Raspbian which worked out of the box (what fun it is! :( ). I then moved on to try Arch and the fun began. Arch Linux install guide at elinux is pretty good but it only helps you to get bare bones Arch up and running. After that you are on your own. So here I am going to discuss how I managed to get Arch up and running with XFCE, a login manager and a web browser.

First off, download the Arch Linux from Raspberry Pi downloads page. Raspberry Pi's processor is ARMv6 so you cannot just use any Arch variant. Once you are done with the download, you need to extract it and transfer the .img file to a sd card. Either use dd command for this or use a tool like ImageWriter. There are more options available. Check out elinux more choices. I'll use dd command here:
# dd bs=4M if=~/archlinux-hf-2012-09-18.img of=/dev/mmcblk0

No, cp command is not supposed to be used here because cp copies over the file system and we have to do something at much more lower level. In case you are wondering how I got the /dev/mmcblk0 bit, I just mounted the sd card and check the output of df -h command. If you are using a sd card of more than 2G memory, then I recommend using gparted or anything else and expand the size of the file system since by default it'll be just about 2G and rest of your space will go unused. Once you are done here, insert the sd card into your Pi and fire it up.

Now you can see the awesome black login screen. The default password for root user is 'root'. Login as root and create pacman, the Arch Package Manager, database.
# pacman-key --init
Some randomness would be helpful here. So hit ALT+F2 to go to another tty and execute some random commands like ls and echo and cd etc. Switch back to the previous tty by hitting ALT+F1 and wait till the initialization of db is done. Now you can update your repositories:
# pacman -Syu

Now first we will install the xorg libraries:
# pacman -S xorg-xinit xorg-server xorg-server-utils
This will install the X server and pull some common dependencies.

To install XFCE now, fire:
# pacman -S xfce4
It'll ask you to "Enter a selection" after giving some packages. I installed all of them since they looked quite necessary like Thunar and the top panel etc but you can be choosy if you want.

Is your GUI working? You may be missing display drivers. Install them:
# pacman -S mesa xf86-video-fbdev xf86-video-vesa

We still need a login manager. I used SLiM, the Simple Login Manager. Remember it is Pi, so we are trying to do everything lightweight.
# pacman -S slim

Reboot after this and you will be shown a GUI interface to enter your user id and password to login. Do that and open a terminal. We'll install a web browser now. You might be tempted to install Firefox or Chrome but remember, this is ARMv6 and none of the main stream browsers support this architecture out of the box. So either you can compile the binary from Firefox or Chromium code or install a browser like Midori or Arora. I installed Midori because I am more familiar with it.
# pacman -S midori

That is it. You got Arch in quite usable state with a working XFCE. Have fun!

PS: The memory footprint with XFCE up and running is about 140m for my Pi.