November 19

Linux: Finding text within multiple files in multiple directories and replacing the text

I like to start these type of recursive runs in the root directory of the search. So in our test case I would do the following:

Eg. Find the port number 8080 in multiple .xml files and replace it with the port number 15001.

  1. cd /path/rootdirectoryofsearch

*note – Change which files get examined by changing the extension type:

  1. Then run the command
    With Backup .mybackup
    find . -type f -name “*.xml” -exec sed -i.mybackup -e ‘s/8080/15001/g’ {} +
  2. Delete backups:
    find . -name “*.mybackup” -type f – delete
  3. With Backup
    find . -type f -name “*.xml” -exec sed -i” ‘s/8080/15001/g’ {} +
Category: Linux | Comments Off on Linux: Finding text within multiple files in multiple directories and replacing the text
November 19

Linux: How can you distinguish between a crash and a reboot on RHEL7

(1) auditd logs

auditd is amazing. You can see all the different events that it logs by checking ausearch -m. Apropos to the problem at hand, it logs system shutdown and system boot, so you can use the command ausearch -i -m system_boot,system_shutdown | tail -4. If this reports a SYSTEM_SHUTDOWN followed by a SYSTEM_BOOT, all is well; however, if it reports 2 SYSTEM_BOOT lines in a row, then clearly the system did not shutdown gracefully, as in the following example:

[root@a72 ~]# ausearch -i -m system_boot,system_shutdown | tail -4
----
type=SYSTEM_BOOT msg=audit(09/20/2016 01:10:32.392:7) : pid=657 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg=' comm=systemd-update-utmp exe=/usr/lib/systemd/systemd-update-utmp hostname=? addr=? terminal=? res=success' 
----
type=SYSTEM_BOOT msg=audit(09/20/2016 01:11:41.134:7) : pid=656 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg=' comm=systemd-update-utmp exe=/usr/lib/systemd/systemd-update-utmp hostname=? addr=? terminal=? res=success' 

(2) last -x

Same as above, but with the simple last -n2 -x shutdown reboot command. Example where system crashed:

[root@a72 ~]# last -n2 -x shutdown reboot
reboot   system boot  3.10.0-327.el7.x Tue Sep 20 01:11 - 01:20  (00:08)    
reboot   system boot  3.10.0-327.el7.x Tue Sep 20 01:10 - 01:20  (00:09)    

Or where system had a graceful reboot:

[root@a72 ~]# last -n2 -x shutdown reboot
reboot   system boot  3.10.0-327.el7.x Tue Sep 20 01:21 - 01:21  (00:00)    
shutdown system down  3.10.0-327.el7.x Tue Sep 20 01:21 - 01:21  (00:00)    

(3) create your own service unit

This is IMHO the best approach because you can tailor it to whatever you want. There are a million ways to do this. Here's one I just made up. This next service only runs at shutdown.

[root@a72 ~]# cat /etc/systemd/system/set_gracefulshutdown.service
[Unit]
Description=Set flag for graceful shutdown
DefaultDependencies=no
RefuseManualStart=true
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/bin/touch /root/graceful_shutdown

[Install]
WantedBy=shutdown.target
[root@a72 ~]# systemctl enable set_gracefulshutdown.service 
Created symlink from /etc/systemd/system/shutdown.target.wants/set_gracefulshutdown.service to /etc/systemd/system/set_gracefulshutdown.service.

Then when the system boots, this next service will only start if the file created by the above shutdown service exists.

[root@a72 ~]# cat /etc/systemd/system/check_graceful.service 
[Unit]
Description=Check if system booted after a graceful shutdown
ConditionPathExists=/root/graceful_shutdown
RefuseManualStart=true
RefuseManualStop=true

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/rm /root/graceful_shutdown

[Install]
WantedBy=multi-user.target
[root@a72 ~]# systemctl enable check_graceful
Created symlink from /etc/systemd/system/multi-user.target.wants/check_graceful.service to /etc/systemd/system/check_graceful.service.

So at any given time I can check if the previous boot was done after a graceful shutdown by doing systemctl is-active check_graceful, e.g.:

[root@a72 ~]# systemctl is-active check_graceful && echo YAY || echo OH NOES
active
YAY
[root@a72 ~]# systemctl status check_graceful
● check_graceful.service - Check if system booted after a graceful shutdown
   Loaded: loaded (/etc/systemd/system/check_graceful.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2016-09-20 01:10:32 EDT; 20s ago
  Process: 669 ExecStart=/bin/rm /root/graceful_shutdown (code=exited, status=0/SUCCESS)
 Main PID: 669 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/check_graceful.service

Sep 20 01:10:32 a72.example.com systemd[1]: Starting Check if system booted after a graceful shutdown...
Sep 20 01:10:32 a72.example.com systemd[1]: Started Check if system booted after a graceful shutdown.

Or here's after an ungraceful shutdown:

[root@a72 ~]# systemctl is-active check_graceful && echo YAY || echo OH NOES
inactive
OH NOES
[root@a72 ~]# systemctl status check_graceful
● check_graceful.service - Check if system booted after a graceful shutdown
   Loaded: loaded (/etc/systemd/system/check_graceful.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Tue 2016-09-20 01:11:41 EDT; 16s ago
           ConditionPathExists=/root/graceful_shutdown was not met

Sep 20 01:11:41 a72.example.com systemd[1]: Started Check if system booted after a graceful shutdown.

(4) journalctl

It is worth mentioning that if you configure systemd-journald to keep a persistent journal, you can then use journalctl -b -1 -n to look at the last few (10 by default) lines of the previous boot (-b -2 is the boot before that, etc). Example where the system rebooted gracefully:

[root@a72 ~]# mkdir /var/log/journal
[root@a72 ~]# systemctl -s SIGUSR1 kill systemd-journald
[root@a72 ~]# reboot
...
[root@a72 ~]# journalctl -b -1 -n
-- Logs begin at Tue 2016-09-20 01:01:15 EDT, end at Tue 2016-09-20 01:21:33 EDT. --
Sep 20 01:21:19 a72.example.com systemd[1]: Stopped Create Static Device Nodes in /dev.
Sep 20 01:21:19 a72.example.com systemd[1]: Stopping Create Static Device Nodes in /dev...
Sep 20 01:21:19 a72.example.com systemd[1]: Reached target Shutdown.
Sep 20 01:21:19 a72.example.com systemd[1]: Starting Shutdown.
Sep 20 01:21:19 a72.example.com systemd[1]: Reached target Final Step.
Sep 20 01:21:19 a72.example.com systemd[1]: Starting Final Step.
Sep 20 01:21:19 a72.example.com systemd[1]: Starting Reboot...
Sep 20 01:21:19 a72.example.com systemd[1]: Shutting down.
Sep 20 01:21:19 a72.example.com systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Sep 20 01:21:19 a72.example.com systemd-journal[483]: Journal stopped

If you get good output like that, then clearly the system was shutdown gracefully. That said, it's not super-reliable in my experience when bad things happen (system crashes). Sometimes the indexing gets weird
Category: Linux | Comments Off on Linux: How can you distinguish between a crash and a reboot on RHEL7
November 19

Linux: RHEL issue with duplicate repositories

I ran into an issue with one of our servers that dealt with duplicate repositories when running “yum check-update”
Update notice FEDORA-EPEL-2018-20225d1828 (from epel) is broken, or a bad duplicate, skipping.

My fix was to check in /etc/yum.repo
I found that one of my co workers had setup two other .repo files that we conflicting with the one created when registering the Red Hat server with Satellite.
I did the following:

  1. renamed the additional repositories to a .old extension
  2. yum clean all
  3. yum check-update
Category: Linux | Comments Off on Linux: RHEL issue with duplicate repositories
November 19

Linux: Blacklist a country using Firewalld

Create the blacklist:

firewall-cmd –permanent –new-ipset=blacklist –type=hash:net –option=family=inet –option=hashsize=4096 –option=maxelem=200000

  • –permanent = use to make changes to the permanent configuration –new-ipset = name of the new IP/net blacklist –type = storage hash type, “net” is for subnets, while “ip” for individual ip addresses –option=family = IPv4 or IPv6 network, inet is for IPv4 –option=hashsize = the initial hash size of the list –option=maxelem = max number of elements

Download net blocks:

wget http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
tar -vxzf all-zones.tar.gz

Choose which countries you would like to block, ipdeny.com provides net blocks by country. The above command will download all country zones together in one archive. Once extracted you should end up with various files, each named after a country, for example “cn.zone” for China. I can’t tell you what to block, it all depends on what kind of service you provide and the location of your “real” requests. Personally, I run many major European sites and based on my logs, I block the following countries: ar bd bg br by cn co il in ir kp ly mn mu pa sd tw ua ro ru ve vn

After block the above countries, SPAM and hacking attempts dropped to nearly zero. Pretty much anything else comes via a European or American proxy, but that is easy to mitigate, once I file an abuse report to their network provider, the proxy is usually shut down rather quickly. While orchestrated and methodical hacks won’t be mitigated by a simple country block list, everything else will be blocked, especially spam.

Populate the blacklist:

firewall-cmd –permanent –ipset=blacklist –add-entries-from-file=./cn.zone

The above command will load a country zone file to our blacklist. Make sure to change the path and filename to your chosen country zone file. You may also add individual IP addresses or net blocks by yourself, from the shell or by using a tool like fail2ban, with the following simple shell script (for example, save it as ~/bin/ban):

firewall-cmd –permanent –ipset=blacklist –add-entry=$1
firewall-cmd –ipset=blacklist –add-entry=$1

Run it like this:

ban 192.168.1.0/24

Redirect the blacklist to the drop zone

firewall-cmd –permanent –zone=drop –add-source=ipset:blacklist
firewall-cmd –reload

Category: Linux | Comments Off on Linux: Blacklist a country using Firewalld
November 19

Linux: Issue with an interrupted upgrade

  1. Reboot gave display this message: “Kernel Panic – Not Syncing: VFS: Unable To Mount Root FS”
  2. Was able to boot into a past kernel. This gave me hope that it was just going to be a problem relating to the latest kernel installation
  3. I discovered that the latest kernel files were not all completely created. Particularly boot/initramfs-4.1.12-124.24.3.el6uek.x86_64.img
  4. I used dracut to build the image file:
    dracut /boot/initramfs-4.1.12-124.24.3.el6uek.x86_64.img 4.1.12-124.24.3.el6uek.x86_64
  5. I also ran yum-complete-transaction to force yum to finish installing the rest of the files that were interrupted
  6. I rebooted again and got the same error.
  7. We discovered that the /boot/grub/grub.conf file did not create correctly. Problem: title Oracle Linux Server Unbreakable Enterprise Kernel (4.1.12-124.24.3.el6uek.x86_64) root (hd0,0) kernel /vmlinuz-4.1.12-124.24.3.el6uek.x86_64 ro root=/dev/mapper/vg_servername-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_servername/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_servername/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet numa=off transparent_hugepage=never '''initrd /initramfs-4.1.12-61.1.28.el6uek.x86_64.img''' It appears that during the Kernel configuration build since /boot/initramfs-4.1.12-124.24.3.el6uek.x86_64.img was missing, it pointed to an incoreect initramfs file. This file boots the hard drive. Corrected: title Oracle Linux Server Unbreakable Enterprise Kernel (4.1.12-124.24.3.el6uek.x86_64) root (hd0,0) kernel /vmlinuz-4.1.12-124.24.3.el6uek.x86_64 ro root=/dev/mapper/vg_servername-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_servername/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_servername/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet numa=off transparent_hugepage=never initrd /initramfs-4.1.12-124.24.3.el6uek.x86_64.img
Category: Linux | Comments Off on Linux: Issue with an interrupted upgrade