Install RPM packages to a Crashed server + RAID (software ) in rescue mode

(1) Boot in to the rescue mode. Since the server have packages dependency problems, it will not allow you to chroot.

(2) Configure network in rescue mode.

++++++++++++++++++++++++++++++++++
#ifconfig eth0 <IP-Addr> netmask <net_mask>
#route add default gw 192.168.1.254 eth0
++++++++++++++++++++++++++++++++++

(3) Examine the RAID array using mdadm and create a temperory RIAD config file as follows.

#mdadm --examine --scan > /etc/mdadm.conf

(4) Assemble the raid array which has the broken OS installation. Here it is md4

#mdadm --assemble --scan /dev/md4

Verify the status in /proc/mdstat

(5) Dwonload the missing RPMS

(6) Mount the assembled RAID array to a temperory mount point

# mount /dev/md4 /old_drive

(7) Now, install the RPMS as follows.

#rpm -ivh --force --noscripts --root=/old_drive *.rpm

(8) Now you can unmount crashed array and reboot the server.

Perl/Calfbot (spamming script) infection

Perl/Calfbot

The presence of a /tmp/... file reveals if a server is infected and the file creation timestamp will accurately reflect the infection time. However if the server is rebooted or the C&C server sends a KILL command, the file will still be present but the malware will not be running anymore. In order to confirm an active infection, one must test the presence of a lock on /tmp/... using the following command:

flock --nb /tmp/... echo "System clean" || echo "System infected"

If one is infected, lsof can be used to see what process owns that lock:

lsof /tmp/...

The following can also validate that the targets of the /proc/$pid/exe symbolic links are the real crond:

pgrep -x "crond" | xargs -I '{}' ls -la "/proc/{}/exe"

Anything looking like "/tmp/ " (with a space) in the output is very suspicious.

pgrep requires the procps package. If you can’t install the package, replace:

pgrep -x crond

with

ps -ef | grep crond | grep -v grep | awk '{print $2}'

Routing multiple network interfaces inside a Xen PV CentOS VPS

1) Comment the "GATEWAY=" line in the /etc/sysconfig/network file.

Let us first set some symbolical names. Let $IF1 be the name of the first interface and $IF2 the name of the second interface. Then let $IP1 be the IP address associated with $IF1 and $IP2 the IP address associated with $IF2. Next, let $P1 be the IP address of the gateway at Provider 1, and $P2 the IP address of the gateway at provider 2. Finally, let $P1_NET be the IP network $P1 is in, and $P2_NET the IP network $P2 is in.

2) Creates two additional routing tables, say T1 and T2. These are added in /etc/iproute2/rt_tables. 

echo 1 T1 >> /etc/iproute2/rt_tables
echo 2 T2 >> /etc/iproute2/rt_tables

Eg:

================================
[root@test ~]# cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 T1
2 T2
================================

3) Then you set up routing in these tables as follows:

       ip route add $P1_NET dev $IF1 src $IP1 table T1
       ip route add default via $P1 table T1
       ip route add $P2_NET dev $IF2 src $IP2 table T2
       ip route add default via $P2 table T2

4) Next you set up the main routing table. Note the `src' arguments, they make sure the right outgoing IP address is chosen.

         ip route add $P1_NET dev $IF1 src $IP1
         ip route add $P2_NET dev $IF2 src $IP2

5) Then, your preference for default route:

         ip route add default via $P1

6) Next, you set up the routing rules. These actually choose what routing table to route with. You want to make sure that you route out a given interface if you already have the corresponding source address:

         ip rule add from $IP1 table T1
         ip rule add from $IP2 table T2

7) Put the above commands in the /etc/rc.local file.

Example:
########################
ip route add 210.61.156.0/24 dev eth0 src 210.61.156.253 table T1
ip route add default via 210.61.156.254 table T1
ip route add 59.125.26.0/24 dev eth1 src 59.125.26.34 table T2
ip route add default via 59.125.26.254 table T2

ip route add 210.61.156.0/24 dev eth0 src 210.61.156.253
ip route add 59.125.26.0/24 dev eth1 src 59.125.26.34

ip route add default via 210.61.156.254

ip rule add from 210.61.156.253 table T1
ip rule add from 59.125.26.34 table T2
#########################

8) Reboot the VM and check if eth0 and eth1 are responding to ping requests.

Reference:

http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html

http://www.linuxhorizon.ro/iproute2.html

HP Smart Array RAID status on CentOS

(1) You can download the rpm from here:

 wget ftp://ftp.hp.com/pub/softlib2/software1/pubsw-linux/p1257348637/v71527/hpacucli-9.10-22.0.x86_64.rpm

Check the controller status

-----------------------------

#hpacucli controller all show status

Check the logical drive

------------------------

#hpacucli controller slot=0 logicaldrive all show status

Check the physical drives

--------------------------

hpacucli controller slot=0 physicaldrive all show status

Adaptec Raid Monitoring

You can download the source from here

http://download.adaptec.com/raid/storage_manager/asm_linux_x64_v7_31_18856.tgz

/usr/StorMan/arcconf getconfig 1


/opt/adaptec-monitor.pl
===================
#!/usr/bin/perl -w
# adaptec-monitor.sh - Monitors Adaptec Hardware RAID

use strict;
open(my $arcout, "/usr/StorMan/arcconf getconfig 1|") or die $!;
my @arcout;
while(<$arcout>) {
chomp;
push(@arcout,$_);

}
close($arcout);

my $cont_trouble;
my %disk_trouble;
my $disk_number;

for(@arcout) {
if(/Status of logical device\s+:\s+(Degraded|Missing|SMART-ERROR|Offline|Warning|Critical|Bad|Unknown|Device-Error)/) {
$cont_trouble = $1;
}
if(/Device #(\d+)/) {
$disk_number = $1;
}
if (/State\s+:\s+(Degraded|Missing|SMART-ERROR|Offline|Warning|Critical|Bad|Unknown|Device-Error)/) {
$disk_trouble{ $disk_number } = $1;
}
}

if($cont_trouble) {
&mailit($cont_trouble);
}

while ( my ($key, $value) = each(%disk_trouble) ) {
&mailit($key,$value);
}

sub mailit {

my $title='RAID Issue!';
my $to='<email_id>';
my $from= 'root@server';
my $subject='RAID issue detected on hostname';

open(MAIL, "|/usr/sbin/sendmail -t");

## Mail Header
print MAIL "To: $to\n";
print MAIL "From: $from\n";
print MAIL "Subject: $subject\n\n";
## Mail Body
if ($_[1]) {
print MAIL "Disk Status: #$_[0]: $_[1]\n";
}
else { print MAIL "Controller Status: $_[0]\n" }
close(MAIL);

}


[root@op1 backup]# crontab -l
* */1 * * * /usr/bin/perl /opt/adaptec-monitor.pl