Skip to main content

Posts

Showing posts from November, 2009

net-snmp-lvs module fails under high load

I'm using the net-snmp-lvs module to interface LVS statistics to SNMP so I can graph them (I'm using OpenNMS ). I have a virtual HTTP service that is balanced across eight real servers. In testing, everything seemed to work just fine and I got some nice graphs that show the Connection Rate, Packet Rate, and Byte Rate for the virtual service and each of the real servers. This morning, we attempted a cutover, ie. we re-directed real traffic to the new service. Sadly, our perimeter firewall hit > 90% CPU so we had to revert. But, in the time that we were live, I noticed that the Connection Rate statistics were missing for both the virtual service and the real servers for the period in which the service was under high load: Notice the gap in the Connection Rate graph when the Packet & Byte rate graphs show high values. I am currently investigating the cause of this issue.

Rebuilding ourdelta MySQL SRPM packages on CentOS 5

ourdelta provide MySQL packages for various platforms, built with assorted performance/feature patchsets . Sadly, like the percona builds, the RPM packages for RHEL/CentOS are not upstream-compatible, ie. they package MySQL differently. I was planning to re-build the ourdelta packages to use the upstream RPM package layout but I've decided to stick with re-building the percona packages as I've already done the work for that. Anyway, in case it helps someone, here's how to rebuild the ourdelta packages from the SRPM: rpmbuild --rebuild \   MySQL-OurDelta-5.0.87.d10-65.el5.src.rpm \   --define 'ourdelta 1' \   --define 'mysqlversion 5.0.87' \   --define 'elversion 5' \   --define 'patchset d10'

Configuring lots of Dell DRACs at once

Having got racadm working on my workstation (see my previous post ), the next step is to perform initial DRAC configuration, ie. change the root password, set the SSL cert values, etc. First I checked that all DRACs were pingable: for h in $(seq -w 1 34); do hn=b0$h.drac.example.com if ping -q -c 1 $hn >& /dev/null ; then echo OK else echo failed fidone Next, I created a drac config file (named drac.cfg) containing the settings that are common to all devices: [cfgLanNetworking]cfgDNSDomainName=drac.example.com[cfgUserAdmin]# cfgUserAdminIndex=2cfgUserAdminUserName=rootcfgUserAdminPassword=secret[cfgOobSnmp]cfgOobSnmpAgentEnable=1cfgOobSnmpAgentCommunity=my_community_name[cfgRacSecurity]cfgRacSecCsrKeySize=1024# cfgRacSecCsrCommonName=cfgRacSecCsrOrganizationName=example.comcfgRacSecCsrOrganizationUnit=Web ServicescfgRacSecCsrLocalityName=My CitycfgRacSecCsrStateName=My StatecfgRacSecCsrCountryCode=IEcfgRacSecCsrEmailAddr=contact@example.com I then ran ...

Install Dell racadm on Fedora 11

I wanted to install racadm on my local (non-Dell) workstation running Fedora 11. I tried installing via the repos as per the instructions , but without success. I eventually managed by downloading manually: wget http://linux.dell.com/repo/hardware/OMSA_6.1/platform_independent/rh50_64/racadm/mgmtst-racadm-6.1.0-648.i386.rpmyum localinstall mgmtst-racadm-6.1.0-648.i386.rpm The next problem was that racadm wouldn't run – it failed with the error: ERROR: Failed to initialize transport Running under strace showed that it seemed to be having problems finding a suitable ssl library – probably because racadm is i686 and I'm running it on x86_64. Using yum provides I determined that a suitable ssl lib was shipped with Adobe Reader, installed in /opt/Adobe/ on my workstation. The following couple of symlinks fixed it: ln -s ../../opt/Adobe/Reader9/Reader/intellinux/lib/libssl.so.0.9.8 /lib/i686/ln -s ../../opt/Adobe/Reader9/Reader/intellinux/lib/libcrypto.so.0.9.8 /lib/...

Check file system integrity of multiple xen domU guests

I needed to check the integrity of the file systems on several xen domU guests while the guests were shutdown, ie. I needed to do it from the dom0. I use LVM logical volumes for the block devices for the guests disks named $host-disk0 . These are stored in a volume group named vg_guests . I use kpartx to access the partitions on the block device. Each guest disk has a small physical partition for /boot ; the rest of the disk is allocated to a 2nd partition which is used as an LVM volume group named vg_$host . Here's a script I knocked up to do the job: for host in host1 host2 host3 ; do # create devices from the LVs kpartx -av /dev/mapper/vg_guests-$host--disk* # Activate the VGs for the host for vg in $(vgs --noheadings | grep $host | awk '{print $1}' ) ; do echo Activating $vg vgchange -ay $vg done # check the file systems for p in $(/dev/mapper/vg_$host* | grep -v swap); do e2fsck -p $p done # Deactivate the VGs for ...