What Is GlusterFS Filesystem and How Does It Configure In Linux?

Hello everybody,

Today, I would like to talk about GlusterFS in Linux. Note: It’s Gluster and not Cluster. Glusterfs is a distributed file system and it’s very flexible. You can claim free spaces on each server and make it a huge virtual drive for clients. It’s flexible because you can add/remove servers and make it bigger or smaller size and also you can configure it on tcp protocol for remote access. It reminds me Cloud or Raid but much more flexible.  It’s fast too and most importantly, it’s easy administration. It has ability to balance the data and workload and also high available. It’s really cool. And yes, it’s Open Source. 

Here, I am demonstrating it with 4 CentOS 7.0 machines. 3 servers and 1 client.

At the moment that I am writing this blog, the latest version is Glusterfs 3.5.2. I am going to download the package and install it manually, however, you can install it with yum command easily. The only thing is that yum repository is not up-to-date and you may end up with lower version of Glusterfs such as 3.4.0 or something like that, but installation is much easier. I am going to install it manually for demonstration purposes.

To download and install GlusterFS, run the following commands as a root user on each server and also client:

wget http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.2/CentOS/epel-7Everything/x86_64/glusterfs-3.5.2-1.e17.x86_64.rpm

yum install gcc git nfs-utils perl rpcbind

yum install gcc git nfs-utils perl rpcbind

yum install rsyslog-mmjsonparse

yum install libibverbs

yum install librdmacm

rpm -Uvh glusterfs-3.5.2-1.el7.x86_64.rpm glusterfs-api-3.5.2-1.el7.x86_64.rpm glusterfs-api-devel-3.5.2-1.el7.x86_64.rpm glusterfs-cli-3.5.2-1.el7.x86_64.rpm glusterfs-debuginfo-3.5.2-1.el7.x86_64.rpm glusterfs-devel-3.5.2-1.el7.x86_64.rpm glusterfs-extra-xlators-3.5.2-1.el7.x86_64.rpm glusterfs-fuse-3.5.2-1.el7.x86_64.rpm glusterfs-geo-replication-3.5.2-1.el7.x86_64.rpm glusterfs-libs-3.5.2-1.el7.x86_64.rpm glusterfs-rdma-3.5.2-1.el7.x86_64.rpm glusterfs-server-3.5.2-1.el7.x86_64.rpm

                                                                           Figure 1  

After completing installation, run the following command and find out the version. Note: It’s under GNU license:

glusterfs –V

                                                                           Figure 2

Now, you need to open firewall. Run the below commands in all machines:

 

iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24011 -j ACCEPT

iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT

iptables -I INPUT -m state --state NEW -m udp -p udp --dport 111 -j ACCEPT

If you need to add more servers, you must add more port to open in firewall for each server. For example, 24012, 24013, … in this case.

Now, it’s time to start the glusterd service:

service glusterd start

or

/bin/systemctl start glusterd.service

                                                                           Figure 3

Now, we need to configure our servers. You must select one of this servers, doesn’t matter which one, to act as a master server. The first thing that you need to do is creating a Pool or Storage Pool. Log in as a root user in master server and run the following commands:

 

service glusterd  restart

gluster peer probe 192.168.157.133

gluster peer probe 192.168.157.134

You must replace the IPs above with your IP addresses. You don’t need to restart you glusterd service, but if you get “Connection failed. Please check if gluster daemon is operational” message like me as showed in figure 4, just restart glusterd service and then you should be fine.

                                                                            Figure 4

To see what’s going on, run this command:

gluster peer status

                                                                                 Figure 5

 
Now, it’s time to create a volume or a virtual volume in other words. I call it a virtual volume because client only sees one volume or disk drive, however, this volume or disk resides on all servers or 3 servers in this case. Anyway, create an empty directory on each server:
 

mkdir –p /mnt/storage.132

mkdir –p /mnt/storage.133

mkdir –p /mnt/storage.134

Then, on master server, run the below command to create a volume:

gluster volume create MyCorpVol transport tcp 192.168.157.132:/mnt/storage.132 192.168.157.133:/mnt/storage.133 192.168.157.134:/mnt/storage.134 force

I used “force” option because it kept warning me that it’s not a good idea to install on root directory or get space from system partition; but since it’s for training purposes, I just force it. Ideally, you must install it on a separate disk, anything other than system partition. See Figure 6.

OH! By the way, we have 3 types of volume:

1.  Distributed volume (command above).

Distributed volume distributes files to all servers (like Load Balancer) and because it balances the load, there is no pressure on one server and read/write files are fast. However, this is no fault tolerance. It means if one of servers goes down, you are going to lose your part of data that resides on that particular server. 

     2.  Replicated volume (see below).
 

Replicated volume writes files to all servers (replica) and you have fault tolerance now, however, the speed of writing files is going to be slow. The good thing is that you can define peer of replica. For example, if you have 4 servers, you can define 2 replicas. So, in that way, you have both fault tolerance and speed.

3        3.   Distributed Strip volume.

This is going to be very fast because gluster divides files to equal pieces and writes them to all servers at the same time. Therefore, it’s so fast. However, there is no fault tolerance.

Now, it’s time to “start” our created volume. Run the command below:

gluster volume start MyCorpVol

To see the volume’s info, run this command:

gluster volume info

                                                                           Figure 6

On client side, first we need to mount our created volume. Run the bellow command to mount it:

mkdir –p /opt/CorpStorage

mount –t glusterfs 192.168.157.132:/MyCorpVol /opt/CorpStorage

and yes; of course you can add it to /etc/fstab to mount it automatically after reboot. Now, let’s try it and see how it works. I am going to copy some files to mounted folder. Figure 7.

                                                                              Figure 7

                          

Do you want to know where those files actually have gone? Look at Figure 8, 9, and 10. Note: In this example, it’s Distribute Volume. It randomly distributes files. At this example, server 134 is empty, but it may get some files next time as files distributes to all servers.

                                                                              Figure 8

                                                                             Figure 9

                                                                              Figure 10  

Let’s create another volume with replicated mode. Run the following commands:

gluster volume create MyCorpVol2 replica 3 transport tcp 192.168.157.132:/var/storage1 192.168.157.133:/var/storage2 192.168.157.134:/var/storage3 force

gluster volume start MyCorpVol2

gluster volume info

                                                                        Figure 11   

The same deal here on the client side:

mkdir –p /opt/CorpStorage2

mount –t glusterfs 192.168.157.132:/MyCorpVol2 /opt/CorpStorage2

Now, if you copy some files on mounted directory, you should be able to see all files on all servers because the type of volume is Replicated Volume. See Figure 12, 13, 14 and 15.

                                                                             Figure 12

                                                                             Figure 13

                                                                             Figure 14

                                                                             Figure 15

When you want to add more storages (it can be on the same servers or a new server), you need to add bricks to existing volume. Run the following commands:

 mkdir –p /var/storage4

 gluster volume add-brick MyCorpVol2 replica 4 192.168.157.132:/var/storage4 force

                                                                            Figure 16 

To reduce the size of volume, you need to remove-brick command:

gluster volume remove-brick MyCorpVol2 replica 3 192.168.157.132:/var/storage4 force

                                                                                Figure 17 

 

In this example, because we are using Replication Volume, you don’t need to rebalance data and it will copy all data as soon as next file comes in to all servers. However, if you are using Distributed or Strip Volume, you need to rebalance data. Let’s try another example:

In this example, there are 2 Distributed volumes and I want to add one more volume and then rebalance data on servers. Run the following commands:

gluster peer probe 192.168.157.134    ---> to add new server to storage pool

gluster volume add-brick MyCorpVol3 192.168.157.134:/var/share3 force

gluster volume rebalance MyCorpVol3 start

gluster volume rebalance MyCorpVol3 status  ---> to see the status of volume

                                                                               Figure 19

The same deal here when you want to remove a brick:

gluster volume remove-brick MyCorpVol3 192.168.157.134:/var/share3 force

gluster volume rebalance MyCorpVol3 start

gluster volume rebalance MyCorpVol3 status

                                                                               Figure 20

To monitor your volumes and servers run the below command:

gluster volume profile MyCorpVol3  info

                                                                       Figure 21 

 

If you want to grant or deny access to a glusterfs volume for a specific client or subnet, use the following commands but replace the ip or hostname with your ip or hostname.

gluster volume set MyCorpVol3  auth.allow xxx.xxx.xxx.xxx (IP)

gluster volume set MyCorpVol3 auth.reject xxx.xxx.xxx.xxx (IP)

                                                                        Figure 22

 

And finally, if you want to delete an existing volume, you must first stop the volume and then delete it:

gluster volume stop MyCorpVol3

gluster volume delete MyCorpVol3

That’s all. I hope you enjoyed of reading this blog. Don’t forget to put your comments here.

Regards,

Khosro Taraghi

Linux as an IPv6 Router

Hello all,

First of all, I do apologize that I haven't updated my blog page since 3 months ago because I have had a very difficult situation in my life recently, but everything went well. Special thanks to all my supporters who helped me on this sticky situation.

Today, I would like to talk about IPv6 Router and how we can configure/monitor Linux(RedHat,Fedora,CentOS,SELinux) to work as an IPv6 Router. We can easily use radvd daemon
(Router ADVertisement Daemon) for this purpose. In order to install radvd daemon, run the following command after you switched to su :

su -
yum install radvd

Now you need to turn on IPv6 forwarding. Run the below command (Figure 1):

sysctl net.ipv6.conf.all.forwarding=1

                                                                        Figure 1


Configuration file is located at /etc/radvd.conf.  Figure 2 shows the content of radvd.conf :

                                                                          Figure 2

As you can see in the figure 2, all lines are commented. Based on our requirements in network, we can start to uncomment those lines.

Let's go through this lines and their definitions:

interface eth0
You need to decide which NIC or interface you want to use as a router. In this example, it assumes one interface:  ens33

                                                                           Figure 3



AdvSendAdvert on;
A flag indicating  whether  or  not  the router sends periodic router advertisements and responds to router solicitations. Router solicitations means when radvd daemon detects router network address requests from hosts.

MinRtrAdvInterval 30;
The minimum time allowed between sending unsolicited multicast router advertisements from the interface, in seconds.

MaxRtrAdvInterval 100;
The maximum time allowed between sending unsolicited multicast router advertisements from the interface, in seconds.

prefix 2001:db8:1:0::/64
 {
    AdvOnLink on;
    AdvAutonomous on;
    AdvRouterAddr off;
 };

The prefix definition specifies your IPv6 network address. To specify prefix options for a specific prefix, add them within parentheses following the prefix definition.  Here we have 3 prefix options.

AdOnLink on
According to manpage, when set, indicates  that this prefix can be used for on-link determination.
When not set the advertisement makes no statement about on-link or off-link properties of the prefix. It simply means that host requests can be received on the specified network address.

 AdvAutonomous on
When set, indicates that this prefix can be used for autonomous address configuration as specified in RFC 4862. It provides automatic address configuration.

AdvRouterAddr off
When set, indicates that the address of interface is sent instead of network prefix, as is required by Mobile IPv6. When set, minimum limits specified by Mobile IPv6 are used for MinRtrAdvInterval and MaxRtrAdvInterval.

Now, I am going to change this configuration file to meet my private network requirements, for example.

                                                                              Figure 4

In this example, for a private network 192.168.74.0 Figure 3, we use the unique-local IPv6 prefix which operates like IPv4 private network address. It's fc00:0:0:0::/64

Just a reminder from my comments in IPv6 configuration (http://ktaraghi.blogspot.ca/2014/02/ipv6-and-network-auto-configuration.html): A host in IPv6 stateless address autoconfiguration network uses its own MAC address to create a temporary link-local address (FE80:: prefix) to be able to connect to router. Then, the router sends its network prefix to replace the link-local prefix and create a full Internet address.

Now, test your configuration file by the following command:

radvd -c

If everything is fine such as syntax, we can start radvd daemon. Run the following command:

/bin/systemctl start radvd.service

to check if the service is running, run the following command:

/bin/systemctl status radvd.service

                                                                            Figure 5

Now, it's time to check and see the advertisement packets that this machine is sending out to other machines/routers. On a different machine, run the following command:

radvdump -d 4

                                                                            Figure 6

-d switch means debug mode and 4 means in verbose mode(log everything).
In Figure 6, it shows that our Linux Router is advertising the correct network address (fc00::/64); the same ipv6 network address that we configured. And That's it.

Hope you enjoyed.
Regards,
Khosro Taraghi

IPv6 and Network Auto-Configuration with IPv6

Hello All,

In order to run IPv6 in your network or personal home computer, you need to setup either DHCP server or IPv6 autoconfiguration. Today, I am going to explain IPv6 autoconfiguration in Linux. Autoconfiguration is stateless in the case of IPv6. If you get all configurations through DHCP server, it's going to be stateful. The beauty of stateless IPv6 autoconfiguration is that you don't need any independent server or DHCP server to connect to your network. It's like plug and play.

IPv6 protocol configures a host  automatically and provides all information that a host needs it. In other words, configuration information is integrated into the IPv6 protocol.With IPv4, you have to configure either manually or use DHCP server to provide configuration information. Therefore, hosts with stateless IPv6 are no longer tied to a particualr DHCP server.

Here is how IPv6 autoconfiguration works on a local network:

IPv6 stateless uses MAC Address of NIC card or network adapter to create a temporary address. Note: MAC address is a unique global address for each NIC card.Then it adds a network prefix of FE80:: to MAC address to create a link-local address (IPv4 equivalent of link-local is 169.254.0.0/16). The link-local prefix is used for physically connected hosts to LAN.

Then, it uses NDP protocol (Neighbor Discovery Protocol) to test the uniqueness of address and make sure that no host is using this address. However, this address is just useable for local network (LAN) and it cannot be routed to a larger network.

If the hosts want to reach larger networks such as Internet, the router provides the larger network address. That network address, provided by router, will replace the original link-local prefix by either a complete global Internet address or unique-local address in case of private network (IPv4 equivalent of unique-local is 10.0.0.0/12, 172.16.0.0/12, 192.168.0.0/16, RFC 1918).
Routers keep advertising this address information and NDP protocol is used to query information. Also, it can be requested specifically.

                                                                             Figure 1

                                                                             Figure 2

In short, a host in IPv6 stateless address autoconfiguration network uses its own MAC address to create a temporary link-local address (FE80:: prefix) to be able to connect to router. Then, the router
sends its network prefix to replace the link-local prefix and create a full Internet address.

Router Renumbering
What will happen for your network and IPv6 autoconfiguration if you change your ISP?
That means that your are changing your network address in IPv6. well, routers renumber/reassign the addresses on their network by RR (Router Renumbering) protocol. The router replaces network prefix with the new one. And similar to DHCP it puts a lease time for network prefix.

And that's it. In the next article,I would like to talk about Linux as an IPv6 Router. Don't miss it.
I hope you enjoyed.
Khosro Taraghi

What Is Systemd and How It Works (Part 3)

Hello everyone,
Today, I continue the systemd discussion, part3. If you haven't read or missed part 1 and 2, you can find it in the following links:
http://ktaraghi.blogspot.ca/2013/11/what-is-systemd-and-how-it-works-part-1.html and http://ktaraghi.blogspot.ca/2013/12/what-is-systemd-and-how-it-works-part-2.html.
In part 3, I would like to talk about special targets and template unit files. systemd has a set of special target files designed for special purposes. you can find a list of special targets in man page, man systemd.special, Figure 1.

                                                                      Figure 1

Some of these special targets are used for on demand services such as printer and bluetooth. So, when a printer is connected and turned on, the printer target becomes active (printer.target). Another example can be sound.target. It's activated when the system detects sound card at boot or it plugs in.
As I said, a full list of special targets are available in manpage including explanations (man systemd.special), however, I tried to explain some them:

basic.target: It loads basic system and boot-up. It pull-in mount points, sockets, and other basic initializations necessary for general purpose daemons.

                                                                          Figure 2

ctrl-alt-del.target: Activated when the user press Ctrl+Alt+Del keys and it is a link to the reboot.target which reboots the system.
default.target: It references to special target to be activated on boot (usually multi-user.target or graphical.target). It's equivalent to run level 3 or 5 in system V.
emergency.target: Starts base system with an emergency shell on the main console.
multi-user.target: Starts up command line interface and it's multi-user (Similar to runlevel 3)
graphical.target: Starts GUI interface (Similar to runlevel 5)
poweroff.target: For shutting down the system.
reboot.target: For rebooting system.
rescue.target: For setting up the base system and a rescue shell
shutdown.target: For terminating the services on system shutdown
sysinit.target: For covering early boot-up scripts
system-update.target: For an offline system update.
exit.target: For shutting down the user service manager

Template unit files

There is a special type of unit file called a template file. A template file name ends with an @ sign. If a corresponding unit file is not found for a service, systemd will check to see if there is template file that can be applied to it. For example, you don't know how many terminals you may use, therefore, they are generated automatically using the getty@.service unit file. Let's take a look at getty@.service: Figure 3

                                                                         Figure 3

Here, since it's template, %I is used to substitute for the actual service name. For example, if the service name is getty@tty2, %I is substituted with tty2.

Finally, systemctl command is used to control the systemd system and service manager. You can still use the "service" command, however, it redirects to systemctl after running "service" command. Figure 4 shows that how we can use the systemctl and service command:

                                                                           Figure 4

And pretty much that's it. Hope it was useful and you enjoyed.
Khosro Taraghi