what is split brain syndrome in clusters?

In this post we are going to see what is split brain syndrome in clusters?

Split brain syndrome is a state on a cluster where each cluster nodes are divided into small clusters and each one will believe themselves as itself only an active cluster.

Each nodes believes that all other nodes are dead and simultaneously will try to access the same data/ disks, which can be lead to data corruption. This situation will occur during the cluster reformation.

When one or more nodes fails, the cluster itself will reform the cluster with the available nodes.

Note: High Availability clusters will use some mechanism like CMAN, Pacemaker, HP ServiceGaurd, and Linux HA to avoid split brain syndrome.

Common methods to address split brain syndrome:

  1. I/O Fencing
  2. Quorum/ Local Disk
  3. Quorum Server
  4. Tie – Breakers
  5. STONITH(Shoot The Other Node In The Head)

How to boot with an old kernel in RHEL4,5,6/CentOS

In this post, we are going to see How to boot with an old kernel in RHEL4,5,6/CentOS Operating systems.

How to boot with an old kernel in RHEL4,5,6/CentOS

RedHat Operating System uses GRUB boot loader as default one globally.

We can boot update the kernel using Yum/ RPM Package management like other package upgrades which we are doing.

Use the below command to know which boot loader installed on your OS.

#grubby -bootloader-probe

Changing kernel:

/boot/grub/grub.conf is the grub configuration file.

#cat /boot/grub/grub.conf 

default=0 timeout=5 password --encrypted $6$GXGrYVEnbKXAnQoT$p64OkyclNDt4qM2q47GMsgNxJxQaclNs79gvYYsl4h07ReDtJpt5P5kQn1KQ52u2eW8pKHTqcG50ffv0UlRcW0 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Red Hat Enterprise Linux 6.4 (2.6.32-358.el6.x86_64) ===> kernel 0 root (hd0,0) kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/vg_geeklab-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_geeklab/lv_swap rd_LVM_LV=vg_geeklab/lv_root rd_NO_DM rhgb quiet initrd /initramfs-2.6.32-358.el6.x86_64.img title Red Hat Enterprise Linux 6.3 (2.6.32-279.el6.x86_64) ===> kernel 1 root (hd0,0) kernel /vmlinuz-2.6.32-279.el6.x86_64 ro root=/dev/mapper/vg_geeklab-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_geeklab/lv_swap rd_LVM_LV=vg_geeklab/lv_root rd_NO_DM rhgb quiet initrd /initramfs-2.6.32-279.el6.x86_64.img

In this, we can see Default will be 0. From this, we can understand OS will read by the default top kernel.

Whenever we are upgrading the kernel that will be coming up and considered as 0 and old kernel will be marked as 1.

So, We should change the number from “0” to “1” on below line of /boot/grub/grub.conf file using vi editor and save it.

default=1

We done the needed configuration change to boot the OS with an old kernel.

On the next boot it will take effect.

Using below command reboot the system and then check whether its booting with old kernel or new one.

#shutdown -r now

Use the below command to check the kernel versions which is in use right now

#uname -r

awk command in linux

awk command in linux

We are going to see how to use awk command in Linux in this post.

It’s a scripting language and it’s used to generate Reports and Data Manipulation.

Syntax:

#awk <option> 'criteria {action}' input_file > output_file

Awk command to print file content:

[root@localhost ~]# awk '{print}' testfile.txt
Abu 1234
Thahir 5678
Tharun 9101
Rishi 2345

Above example is only to print all the content of a file.

Awk command to print the lines which match with the given pattern:

[root@localhost ~]# awk '/Rishi/{print}' testfile.txt
Rishi 2345

awk command to split a line to fields:

$1 will be considered the first word as the first field in a line. accordingly $2,$3, etc…

[root@localhost ~]# awk '{print $2}' testfile.txt
1234
5678
9101
2345
[root@localhost ~]#

Built-in variables in awk:

NF:     We can print the last field of the lines by using NF in awk command

Example: 

[root@localhost ~]# awk '{print $NF}' testfile.txt
25000
30000
20000
15000

NR:     Using NR built-in option, we can print the specific fields along with line numbers and can print all content of a file along with the line numbers. Also, we can print the range of lines using NR in awk command.

Examples:

1. Displaying specific row with a specific field in a file

[root@localhost ~]# awk 'NR==2 {print $1,$3}' testfile.txt
Thahir 30000

2.Displaying content of a range of lines(from 2 to 4th line)

[root@localhost ~]# awk 'NR==2, NR==4 {print $1,$3}' testfile.txt
Thahir 30000
Tharun 20000
Rishi 15000

 

Thanks for reading our blog. Please drop your comments.

NFS Server Configuration RHEL 7 / Cent OS 7

Using NFS server will share the files/ Directories over the network.
Using this service will install operating system and access the remote server directory/file to read/write files and will share the file/directory to other machines.

Required Package:

libnfsidmap-0.25-15.el7.x86_64
nfs-utils-1.3.0-0.33.el7.x86_64

Installing NFS packages:

Use the below command to install all the required packages in one shot.

#yum install nfs*

 

Check whether the packages are installed by executing below command

[root@server ~]# rpm -qa | grep nfs
libnfsidmap-0.25-15.el7.x86_64
nfsometer-1.7-1.el7.noarch
nfs4-acl-tools-0.3.3-15.el7.x86_64
nfs-utils-1.3.0-0.33.el7.x86_64
nfstest-2.1.1-0.0.el7.noarch

Important configuration files:

/etc/exports :                This file contains which all are exported to remote                                                                 machines 
/etc/host.allow :           Daemon/ Client which matches the entry available in                                                        this file, will be granted access.
/etc/host.deny :           Access is denied for daemon/ client which matches in                                                     this file.
/etc/fstab :                      Will mount the shared directories/ filesystems                                                                   permanently using this file.
/etc/sysconfig/nfs:  Will manage the nfs port using this file.

Services which need to be enabled and started:

rpcbind  service
nfs-server  service

Command to enable the services:

#systemctl enable rpcbind
#systemctl enable nfs-server

[root@server ~]# systemctl enable rpcbind
[root@server ~]# systemctl enable nfs-server
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

Command to start the services:

#systemctl start rpcbind
#systemctl start nfs-server

[root@server ~]# systemctl start nfs-server

[root@server ~]# systemctl start rpcbind

create directory for sharing:

[root@server ~]# mkdir /nfshare

set the all permission to all

#chmod 777 /nfshare

Now share the directory using NFS by mentioning the directory details in /etc/exports file

#vi /etc/exports

/nfsfileshare 192.168.12.7(rw,sync,no_root_squash)

save and exit from the file

Now use the below command to make it available in network as a shared directory

[root@server ~]# exportfs -r

Enable the services permanently in firewall in this session

[root@server ~]# firewall-cmd --permanent --zone public --add-service  mountd
success
[root@server ~]# firewall-cmd --permanent --zone public --add-service rpc-bind
success
[root@server ~]# firewall-cmd --permanent --zone public --add-service nfs
success

reload the firewall  changes and make effect in this session by executing below command

[root@server ~]# firewall-cmd --reload
success

 

Now all the configuration has been done in server side and have to check and mount the shared directory in client machine.

showmount command will help us to list the directory which is shared from remote machine.

[root@node1 ~]# showmount -e 192.168.43.226
Export list for 192.168.43.226:
/nfshare *

Now we need to mount the shared directory in client. Before that we should create a directory to use that as a mount point.

[root@node1 ~]# mkdir /nfsmount

Mount the directory temporarily

#mount 192.168.43.226:/nfshare  /nfsmount

Mount permanently by editing /etc/fstab file and make entry

#vi /etc/fstab
192.168.43.226:/nfshare/ /nfsmount nfs rw,sync 0 0

save and exit from the file.

unmount the directory because we mounted temporarily before restart and after restart issue mount to check whether the shared directory is listing or not. If its not listing then there is an issue with entry in /etc/fstab file.

#umount /nfsmount

Restart the client and check using mount command

[root@node1 ~]# mount | grep /nfsmount
192.168.43.226:/nfshare/ on /nfsmount type nfs (rw,sync,vers=4,addr=192.168.43.226,clientaddr=192.168.43.67)

How to check listening ports in Linux

We should pay attention to listening ports, to validate and disable the unused ports in infra is a very important thing. In that the way, we can secure the server from the attack surface. So, Will see How to check listening ports in Linux

Known Ports:

993    – IMAPS
1194  – openVPN
1812  – RADIUS
995    – POP3s
2049  – NFS (nfsd, rpc.nfsd, rpc, portmap)
2401  – CVS server
3306  – MySql
3690  – SVN
6000-6063- X11
123    – NTP (Network time protocol used for time syncing uses UDP protocol)
137    – NetBIOS (nmbd)
139    – SMB-Samba (smbd)
143    – IMAP
161    – SNMP (For network monitoring)
389    – LDAP (For centralized administration)
443    – HTTPS (HTTP+SSL for secure web access)
514    – Syslogd (udp port)
636    – ldaps (both ctp and udp)
873    – rsync
989    – FTPS-data
990    – FTPS
20      – FTP Data (For transferring FTP data)
21      – FTP Control (For starting FTP connection)
22      – SSH (For secure remote administration which uses SSL to encrypt the transmission)
23      – Telnet (For insecure remote administration)
25      – SMTP (Mail Transfer Agent for e-mail server such as SEND mail)
53      – DNS (Special service which uses both TCP and UDP)
67      – Bootp
68      – DHCP
69      – TFTP (Trivial file transfer protocol uses udp protocol for connection less transmission of data)
80      –  HTTP/WWW(Apache)
88      – Kerberos
110    – POP3 (Mail delivery Agent)


Issue the below command as root user to check the listening ports.

Will use  netstat command to check open ports.

#netstat -tulpn

t – Will enables listing of tcp ports.
u – Will enables listing of udp ports
n – this will shows the port numbers
l – It will list only listening pockets

We should pay attention to listening ports, to validate and disable the unused ports in infra is a very important thing. In that the way, we can secure the server from the attack surface. So, Will see How to check listening ports in Linux


/etc/services – by reading this file we can read different types port/protocol combinations and applications.

#less /etc/services


lsof command:

Package for lsof:  lsof-4.87-4.el7.x86_64

To check the package avilability in linux use the below command.

#rpm -qa | grep -i lsof

To list open ports:

#lsof -i



To display all open files:

nmap command:

    Using nmap  command will check the open ports for remote server.

Using below command will check open ports in local machine.

      #nmap -sT  -O localhost



Command to check remote machine open ports:

#nmap -sT -O 192.168.0.105


Changing/ Recovering root user password in RHEL7/ Cent OS 7

We are going to see How to Changing/ Recovering root user password in RHEL7/ CentOS 7 in this post

Boot the system and wait for GRUB menu. On some Operating systems, you need to disturb the booting by pressing any keys in your system.

Changing/ Recovering root user password in RHEL7/ Cent OS 7
Highlight the first line which is pointing the Operating System entry like showed in the above image and press “e” key in your keyboard to edit manually.
Then will get the below screen.
Using up/down, left/right arrow keys move the cursor to the “rhgb quiet” which is in kernel line starting with “linux16”
Using delete/backspace keys manually delete “rhgb quiet” entry and append “init=/bin/bash” entry like below.

Than press Ctrl+x  to reboot your OS with specified option. Once rebooted, will get a shell prompt without credentials.

check whether the root filesystem has been mounted or not.

Mount the root filesystem with RW permission.

Now issue the “passwd” command to recover/change the root password. It will ask for the new password and re-enter the password for confirmation. Once provided the new password will receive the “Authentication token successfully updated”

Relabel the SELinux content using below command.
Restart the machine using the below command.
Finally, we recovered the root password. Now will restart the machine and login with a new root password.