January 17

Linux: Using rsync to mirror data between servers

Introduction
This LinuxAnswer describes how to mirror 2 systems using rsync over ssh.
I will only talk about a live server and a backup server where the backup server will connect to the live server to pull the data that is to be backed up.

Assumptions
1) You know how to open up a terminal and type a few basic commands.
2) You have a working ssh server and client installed. If not then see:
ftp://ftp.ca.openbsd.org/pub/OpenBSD…rtable/INSTALL
3) You have private/public keys generated to allow passwordless logins to the live server form the backup server.

Why would you want to?
There are many reasons so I’ll just list a few:
1) Data transfer is fast as rsync only copies modified files
2) Running it over ssh encrypts the data transfer so it is more secure than other methods

The real howto
1) Decide on the directories you need to backup on the live server assuming it is a webserver this may be “/home/httpd”
2) Decide on the options you want. The most common I would use are:
-a Archive mode this is a combination of “-rlptgoD” basically it works recursively and maintains file information such as creation dates, permissions etc. See the man page for detailed info.
-v Increase the verbosity. This will let you see what is transferred
-z Compress data so that it is a quicker transfer
–delete-after Delete any files that have been deleted on the live server
-e ssh Most importantly, run the transfer over an ssh connection
A full list can be obtains from “man rsync”.
3) Try a dry run on the backup server with “-n” to make sure any typos don’t totally screw your system. This will just show what would be done:
rsync -e ssh -avzn –delete-after user@liveserver:/home/httpd /home
4) If everything went as expected you can give it a go without -n
rsync -e ssh -avz –delete-after user@liveserver:/home/httpd /home
You should get the info about the files being transferred. Running it again should be quicker as very little has probably changed.
5) That should be it, just try creating and deleting a few files and run rsync to make sure the changes occur

Automating the process
The obvious answer running the rsync commands on the backup server via cron.
A basic example being to mirror every hour on the hour:
0 * * * * rsync -e ssh -avz user@liveserver:/home/httpd /home 2>&1 > /var/log/hourly_backup.log
Then remove deleted files every night:
30 0 * * * rsync -e ssh –delete-after -avz user@liveserver:/home/httpd /home 2>&1 > /var/log/nightly_backup.log

By: D. Ross

Category: Linux | Comments Off on Linux: Using rsync to mirror data between servers
January 17

Linux: Installing DMKS on Red Hat/Centos

DKMS:
– Dynamic Kernel Module Support (DKMS) is a framework used to generate Linux kernel modules whose sources do not generally reside in the Linux kernel source tree. DKMS enables kernel device drivers to be automatically rebuilt when a new kernel is installed.
– An essential feature of DKMS is that it automatically recompiles all DKMS modules if a new kernel version is installed. This allows drivers and devices outside of the mainline kernel to continue working after a Linux kernel upgrade.
– Another benefit of DKMS is that it allows the installation of a new driver on an existing system, running an arbitrary kernel version, without any need for manual compilation or precompiled packages provided by the vendor.
– DKMS was written by the Linux Engineering Team at Dell in 2003. It is included in many distributions, such as Ubuntu, Debian, Fedora, and SuSE. DKMS is free software released under the terms of the GNU General Public License (GPL) v2 or later.
– DKMS supports both the RPM and DEB package formats out-of-the-box. (from Wikipedia)
____________________________________________________

I was trying to install guest additions on my Centos operating system but faced a lot of problems. Though the idea was very simple as all you have to do was to install DKMS package  on your centos operating system and run the install virtual box guest additions setup, but the main problem is that dkms package is not available on your centos,  it is a third party repository. So I believe there are a lot of new users who face this issue (I being one of them).

 

Following steps will help in installing guest additions on your centos.

 

Step1: update everything( though not really required but still I took this step 1st)

 

Step2: make a directory rpm using the following commands and go in that directory and download the rpm package from this link or goto http://pkgs.repoforge.org/rpmforge-release/ and download the appropriate package.

 

1
2
3
4
5
6
7
8
9
$ mkdir rpm
$ cd rpm
$ rpm -i rpmforge-release-0.5.2-2.el5.rf.*.rpm
$ yum install htop

 

now if you get an error something like this

 

1
2
3
error: Failed dependencies:
rpmlib(FileDigests) <= 4.6.0-1 is needed by rpmforge-release-0.5.2-2.el6.rf.i686
rpmlib(PayloadIsXz) <= 5.2-1 is needed by rpmforge-release-0.5.2-2.el6.rf.i686

 

That means you have installed your centos virtual machine from cloudera which is centos5 and you have downloaded rpm package for centos6 so all you have to do is to change that package and download package for centos5. You can also check if you are running a 32 bit machine or a 64 bit machine as there are two packages one is for 32 bit machine and the other for 64 bit. To check which machine you are running just type the following command

 

1
$uname -i

 

if you get i386 or i686 that means you are running 32 bit machine and if you get x86_64 that means you are running a 64 bit machine.

 

Step3. Install kernel-devel

 

1
$ sudo yum install kernel-devel

 

Step4. So almost everything is done and you are ready to install  dkms package

 

1
sudo yum install dkms

 

if everything goes fine dkms package will install successfully, without any issues.

 

Step5. This will be the final step

 

Insert VboxGuestAdditions.iso and go to the folder which will be probably in

 

/media/VboxGuestAdditions

 

and run the following command

 

1
$ sh ./VboxLinuxAdditions.run

 

This will successfully install Guest Additions on Centos.

By: Saad

Category: Linux | Comments Off on Linux: Installing DMKS on Red Hat/Centos
January 17

Linux: Compiling ZFS on Red Hat/Centos

 

 

Installing ZFS on a CentOS 6 Linux server

 

The ZFS file system for Linux comes as source code, which you build into loadable kernel modules (this is how they get around the license incompatibilities). The implementation also contains the userland utilities (zfs, zpool, etc.) most Solaris admins are used to, and they act just like their Solaris counterparts! Nice!

 

Testing occurred on a CentOS 6 machine, specifically 6.5:

 

$ cat /etc/redhat-release
CentOS release 6.5 (Final)

 

Install dependencies:

 

$ yum install gcc kernel-devel zlib-devel libuuid-devel libblkid-devel libselinux-devel parted lsscsi rpm-build

 

Once these are installed you can retrieve and build spl and zfs packages from:

 

http://zfsonlinux.org/download.html

Once downloaded do the following:

 

$ tar xfvz spl-0.6.0-rc14.tar.gz

$ cd spl-0.6.*

 

$ ./configure

$ make rpm

 

$ rpm -Uvh *.x86_64.rpm

 

Preparing...                ########################################### [100%]
   1:spl-modules-devel      ########################################### [ 33%]
   2:spl-modules            ########################################### [ 67%]
   3:spl                    ########################################### [100%]

 

$ wget http://github.com/downloads/zfsonlinux/zfs/zfs-0.6.0-rc6.tar.gz

 

$ tar xfvz zfs-0.6.0-rc14.tar.gz

$ cd zfs-0.6.*

 

$ ./configure

$ make rpm

 

$ rpm -Uvh *.x86_64.rpm

 

Preparing...                ########################################### [100%]
   1:zfs-test               ########################################### [ 17%]
   2:zfs-modules-devel      ########################################### [ 33%]
   3:zfs-modules            ########################################### [ 50%]
   4:zfs-dracut             ########################################### [ 67%]
   5:zfs-devel              ########################################### [ 83%]
   6:zfs                    ########################################### [100%]

 

If everything went as planned you now have the ZFS kernel modules and userland utilities installed! To begin using ZFS you will first need to load the kernel modules with modprobe:

 

$ modprobe zfs

 

To verify the module loaded you can tail /var/log/messages:

 

Feb 12 17:54:27 centos6 kernel: SPL: Loaded module v0.6.0, using hostid 0x00000000
Feb 12 17:54:27 centos6 kernel: zunicode: module license 'CDDL' taints kernel.
Feb 12 17:54:27 centos6 kernel: Disabling lock debugging due to kernel taint
Feb 12 17:54:27 centos6 kernel: ZFS: Loaded module v0.6.0, ZFS pool version 28, ZFS filesystem version 5

 

And run lsmod to verify they are there:

 

$ lsmod | grep -i zfs

 

zfs                  1038053  0
zcommon                42478  1 zfs
znvpair                47487  2 zfs,zcommon
zavl                    6925  1 zfs
zunicode              323120  1 zfs
spl                   210887  5 zfs,zcommon,znvpair,zavl,zunicode

 

To create our first pool we can use the zpool utilities create option:

 

$ zpool create mysqlpool mirror sdb sdc

 

The example above created a mirrored pool out of the sdb and sdc block devices. We can see this layout in the output of `zpool status`:

 

$ zpool status -v

 

  pool: mysqlpool
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	mysqlpool   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	    sdc     ONLINE       0     0     0

errors: No known data errors

 

Awesome! Since we are at pool version 28 lets disable atime updates and enable compression and deduplication:

 

$ zfs set compression=on mysqlpool

 

$ zfs set dedup=on mysqlpool

 

$ zfs set atime=off mysqlpool

 

For a somewhat real world test, I stopped one of my MySQL slaves, mounted the pool on /var/lib/mysql, synchronized the previous data over to the ZFS file system and then started MySQL. No errors to report, and MySQL is working just fine. Next up, I trash one side of the mirror and verified that resilvering works:

 

$ dd if=/dev/zero of=/dev/sdb

 

$ zpool scrub mysqlpool

 

I let this run for a few minutes then ran `zpool status` to verify the scrub fixed everything:

 

$ zpool status -v

 

  pool: mysqlpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scan: scrub repaired 966K in 0h0m with 0 errors on Sun Feb 12 18:54:51 2012
config:

	NAME        STATE     READ WRITE CKSUM
	mysqlpool   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sdb     ONLINE       0     0   175
	    sdc     ONLINE       0     0     0

By: Matty
Modified By: nighthawk

Category: Linux | Comments Off on Linux: Compiling ZFS on Red Hat/Centos
January 7

Linux: Changing / Disable SE Linux configuration from the command line

From the command line, you can edit the /etc/sysconfig/selinux file. This file is a symlink to /etc/selinux/config. The configuration file is self-explanatory. Changing the value of SELINUX or SELINUXTYPE changes the state of SELinux and the name of the policy to be used the next time the system boots.

[root@host2a ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=permissive
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.
SELINUXTYPE=targeted

# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0

Reboot for changes to take effect immediately
[Ref.] Disable SE Linux
Category: Linux | Comments Off on Linux: Changing / Disable SE Linux configuration from the command line
December 30

Linux: Using the dynamic DNS editor: nsupdate

Invoking nsupdate

If one just types “nsupdate” one arrives in a basic command-line environment for sending “update” commands to DNS, but there are two options that you probably want to use when you invoke nsupdate.

The first option “-v” is to specify that communication with the DNS is to be done with TCP, not UDP. This is a good idea if the update requests are potentially longer than 512 bytes, which is often the case.

The second option “-k” is to specify where the encryption key files are to be found. A securely-configured DNS will not accept updates from just anyone – the update requests must be cryptographically signed with an appropriate key. There may be several keys that permit one to update a DNS. One may have created a root-readable key for manual administration of the DNS contained in the files /etc/bind/admin-updater.{key,private}. Note that it is important that the “key” file and “private” file have the same stem path since one cannot be used without the other.

Therefore, the way to invoke nsupdate is:

nsupdate -v -k /etc/bind/admin-updater.key

To quit an nsupdate session, just hit CTRL-D or type “quit” and hit RETURN.

Deleting a record

Unconditionally deleting a DNS record (eg. the CNAME record www.example.com) is straight-forward. At the nsupdate prompt, type:

> update delete www.example.com cname
> send

When one types “send” and hits RETURN, the update request is built, signed, and sent to the appropriate DNS server. If your key was authorised to make the request, the DNS will update its database, update its journal files, increment the serial number of the SOA record, and send a change notification to any slave DNSes. The slaves will start AXFRs (or IXFRs) to collect the updates from the master DNS.

Multiple updates

To save time and reduce DNS traffic (due to excessive slave DNS AXFR/IXFR requests), multiple update requests can be sent simultaneously in one batch:

> update delete www.example.com cname
> update delete www1.example.com a
> update delete www2.example.com a
> update delete www3.example.com a
> send

There are a couple of points to keep in mind. All updates in one “send” batch must be for the same zone. For example, one cannot mix changes for example.com and 1.16.172.in-addr.arpa in the same “send“.

In addition, there seems to be an undocumented limit in nsupdate of how many updates can be in one batch, probably due to a fixed input buffer. I have found that 2000 requests at a time sometimes do not pass. I no longer attempt more than 1000 for every “send“, just to be safe.

Adding records

Here are examples of how to add A, CNAME, and PTR records. One must specify the TTL (time-to-live) of records (in seconds) when they are added.

> update add www1.example.com 86400 a 172.16.1.1
> update add www.example.com 600 cname www1.example.com.
> send

> update add 1.1.16.172.in-addr.arpa 86400 ptr www1.example.com.
> send

Note that I have taken care to use two separate “send” commands to handle the A and PTR updates of www1.example.com since the changes apply to two different zones, example.com and 1.16.172.in-addr.arpa.

Conditional updates

It is possible to program an update conditional on the presence or absence of DNS records (prerequisites), but I have never needed this for manual administration of a DNS. See “man nsupdate” and/or RFC2136 for more information on the possibilities.

Non-interactive usage

One of the most interesting ways of using nsupdate is non-interactive, by specifying a file containing a batch of commands or simply piping them in on STDIN. For example, one could use a text editor or a script to create a file “batch.txt” with the contents:

update delete www.example.com cname
update delete www1.example.com a
update delete www2.example.com a
update delete www3.example.com a
send

and then run it by doing:

nsupdate -v -k /etc/bind/admin-updater.key batch.txt

Here is another non-interactive example using pipelines. Suppose that we want to delete all of the A records in example.com starting with “www”. We could do:

( host -t a -l example.com | grep -i '^www' | 
   awk '{ print "update delete "$1" a" }' ; echo send ) | 
   nsupdate -v -k /etc/bind/admin-updater.key

By:Rossen
Category: Linux | Comments Off on Linux: Using the dynamic DNS editor: nsupdate