Wednesday, December 11, 2013

Setting LDOM auto-boot to false and to be persistent across a CDOM reboot

I ran into this problem recently where you set auto-boot to false (be it via ldm command on the cdom, or eeprom on the host), and once the control domain is powered cycled, the logical domains revert back to their previous state of auto starting. The correct method is as follows:

List current vars for your ldom
# ldm list-variable
it should show - auto-boot?=true (or the var doesn't exist at all)

set host not to autoboot
# ldm set-variable "auto-boot?=false"

view current spconfig
# ldm list-spconfig

save new spconfig
# ldm add-spconfig

Apply new config
# ldm set-spconfig

view spconfig settings for next reboot
# ldm list-spconfig

view current vars
# ldm list-variable CUSRGNAS01DR1
auto-boot?=false

Shutdown hosts from the OS
# init 5

Shutdown CDOM
# init 0

Power off CDOM from the ILOM
-> stop /SYS

Check it's powered off
-> show /SYS power_state

Power on from the ILOM
-> start /SYS

view current spconfig
# ldm list-spconfig

List current vars for your ldom
# ldm list-variable

Check LDOM state
# ldm list

It show as active, but if you telnet to the console port, you'll see it sitting at the OK prompt.

Wednesday, November 27, 2013

Alarm on high non-global zone memory usage

We use a lot of Solaris non-global zones for various reasons. Our tools admin enabled memory usage alarms on these ng zones and they alerted incorrectly (due to the misleading internal resource stats). We still use native resource counters to alarm on global zones, but scripted the following bit of perl to alarm when ng zones exceed a defined threshold:

use strict;
use warnings;
use Time::localtime;

my $ram_threshold = "90";

my $zones = `prstat -Z 1 1 | /usr/sfw/bin/ggrep -A 99 ZONEID | egrep -vi 'total|global|zoneid' | awk '{ print \$5","\$8 }'`;
my @zoneSplit = split("\n",$zones);

foreach my $zone (@zoneSplit) {
                my @zone_info = split(",",$zone);
                my $zone_ram = substr($zone_info[0], 0, -1);
                my $zone_name = $zone_info[1];

                if ($zone_ram gt $ram_threshold) {
                                $ConsoleMessage->Severity("Normal");
                                $ConsoleMessage->Application("zone_pool");
                                $ConsoleMessage->Object("object");
                                $ConsoleMessage->MsgText("$zone_name memory exceeded $ram_threshold, value is $zone_ram");
                                $ConsoleMessage->Send();
                }

}

Tuesday, November 19, 2013

Catalogue your UNIX/Linux servers via a shell script

Taking my last post one step further, I've written a script that cuts up my entire UNIX/Linux server list, and outputs flat files containing all HP servers, Linux servers, V100 or V125 SunFire servers, Dell, HP Proliant servers, all offline and online servers, and servers by business unit.

This is incredibly useful for automating a lot my work, as often I will need to (for example) disable a service on my Dell Linux servers (but not my HP Linux servers). Configuration changes, etc.

For those wondering, yes I have setup a puppet master server, and hope to manage all 370+ branch servers with this soon.

Here's the script:


#!/bin/bash

# Delete our server lists
rm -f ../server_lists/*

# Remove any non branch servers from the list

cat serverlist | egrep -vi '^cusrv|^musrv|^dusrv' > serverlist.tmp
mv serverlist.tmp serverlist

for x in $(cat serverlist)

do
  echo ""
  echo ${x}:

  # Ping host

  hostAlive=$(ping ${x} 2)

  case "$hostAlive" in

    *alive*)  echo "Host is online, checking OS..."
  
          # Add to online list
          echo ${x} >> "../server_lists/online"
  
          # Determine Business Unit
          bu=$(echo ${x} | awk '{print substr($0,0,2)}')

          case $bu in

            cg|CG)  # Add to CG list 
                echo ${x} >> "../server_lists/cg"
                ;;

            cd|CD)  # Add to CD list

                echo ${x} >> "../server_lists/cd"
                ;;

            tl|TL)  # Add to TL list

                echo ${x} >> "../server_lists/tl"
                ;;

            mp|MP)  # Add to MP list

                echo ${x} >> "../server_lists/mp"
                ;;

            mt|MT)  # Add to MT list

                echo ${x} >> "../server_lists/mt"
                ;;

            cy|CY)  # Add to CY list

                echo ${x} >> "../server_lists/cy"
                ;;

            *)    # Add to other list

                echo ${x} >> "../server_lists/other"
                ;;
          esac

          # Check operating system

          os=$(ssh -oStrictHostKeyChecking=no root@${x} "uname")

          case $os in


            SunOS)  echo "SunOS detected"


            # Add to Solaris list

            echo ${x} >> "../server_lists/solaris"  

            # Check hardware

            hardware=$(ssh -oStrictHostKeyChecking=no root@${x} "/usr/sbin/prtdiag | head -1 | cut -d: -f2")

            case $hardware in

              *V100*)  echo "V100 detected"
                  echo ${x} >> "../server_lists/v100"  
                  ;;
            
              *V125*)  echo "V125 detected"
                  echo ${x} >> "../server_lists/v125"
                  ;;
            esac
            ;;

            Linux)  echo "Linux detected"


                # Add to Linux list

                echo ${x} >> "../server_lists/linux"

                # Check hardware

                hardware=$(ssh -oStrictHostKeyChecking=no root@${x} "dmidecode | grep -i vendor | cut -d: -f2")
                # ANOTHER METHOD - "lshal | grep system\.hardware\.product | cut -d\' -f2")

                case $hardware in

                  *HP*)  echo "HP ProLiant detected"
                      echo ${x} >> "../server_lists/proliant"
                      ;;

                  *Dell*) echo "Dell detected"

                      echo ${x} >> "../server_lists/dell"
                      ;;
                esac  
                ;;

            *)      echo "Some other os detected, not interested"

                ;;

          esac

          ;;

    *unknown*)  echo "Unable to reach host"


          # Add to offline list

          echo ${x} >> "../server_lists/offline"
          ;;

    *answer*)  echo "Unable to reach host"


    # Add to offline list

          echo ${x} >> "../server_lists/offline"
          ;;

    *)    echo "Unable to reach host"


        # Add to offline list

        echo ${x} >> "../server_lists/offline"
        ;;
  esac
done

The output looks like:

# ls -la
drwxr-xr-x   2 root     other        512 Nov 19 16:01 ./
drwxr-xr-x  15 root     other        512 Nov 19 11:47 ../
-rw-r--r--   1 root     other         84 Nov 19 15:57 cd
-rw-r--r--   1 root     other         62 Nov 19 15:57 cg
-rw-r--r--   1 root     other        532 Nov 19 16:00 dell
-rw-r--r--   1 root     other       1738 Nov 19 16:10 linux
-rw-r--r--   1 root     other       1617 Nov 19 16:00 mp
-rw-r--r--   1 root     other         56 Nov 19 16:00 mt
-rw-r--r--   1 root     other       1272 Nov 19 16:02 offline
-rw-r--r--   1 root     other       8732 Nov 19 16:10 online
-rw-r--r--   1 root     other       1175 Nov 19 16:10 proliant
-rw-r--r--   1 root     other       6963 Nov 19 16:10 solaris
-rw-r--r--   1 root     other       6913 Nov 19 16:10 tl
-rw-r--r--   1 root     other       6498 Nov 19 16:10 v100
-rw-r--r--   1 root     other        465 Nov 19 16:10 v125

Now if I want to run a command against all v125 UNIX servers for example, I can do something like:

# for x in $(cat v125); do ssh root@${x} "foo"; done

Shell script to determine OS

I look after hundreds of UNIX and Linux machines, and often need to roll a change out to them, but the change might be slightly different based on the server's operating system. I often use the following script to determine the OS, and execute the appropriate commands on each server (populate serverlist with your list of servers to check):

#!/bin/bash

for x in $(cat serverlist)
do
        echo Checking ${x}:

        # Determine OS

        os=$(ssh root@${x} "uname")

        case $os in

        SunOS)  echo "SunOS detected, doing something.."
                ssh -oStrictHostKeyChecking=no root@${x} "something Solaris specific"
                ;;

        Linux)  echo "Linux detected, doing something"
                ssh -oStrictHostKeyChecking=no root@${x} "something Linux specific"
                ;;

        *)      echo "Something unheard of"
                ;;

        esac

done

Thursday, November 14, 2013

Motivation via /etc/motd

It's about the time of year when everything and everyone bugs me. To amplify this typical November feeling, the company I work for was purchased a couple of years back, and are really ramping up the outsourcing (sorry....cost reduction initiative). So things generally suck anyhow.

To combat the feelings of despair, and total lack of motivation..I thought it would be a nice idea to remind myself how many days until I go on Christmas break. My main Ops server doesn't have a lot of GNU packages (like gnu date) so I'll use perl.



Script looks like this:


#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
use POSIX;

my @today = localtime();
my $time = timelocal(@today);

my @holiday = (0, 30, 15, 23, 11, 2013);
my $holidayTime = timelocal(@holiday);

print "\nDays Until Holidays = " . floor((((($holidayTime - $time) / 60) / 60) / 24)) . "\n\n";

exit 0;


Crontab looks like this:

# Update MOTD

0 1 * * * /usr/bin/perl /usr/local/scripts/motd/motd.pl > /etc/motd 2>&1


Output looks like this:



Wednesday, October 30, 2013

Thinking Outside the Protocol

I work remotely from the rest of my team and bosses (some based in NZ, most based in Sydney, Australia). This presents quite a few challenges day-to-day. I often need to move ISO's around for various reasons. We don't have a massive amount of "available" disk in our company, so we try not to replicate I.T related files too heavily.

So, the problem....I noticed when trying to straight SMB copy large (eg. 3.6GB) files from our Data Center to the site I work out of (1000km from the DC) our riverbeds don't handle the requests well, they tend to flood and drop my transfer rate down to 1-1.5KB/sec! Oh, and this also impacts the site now as the riverbed is no longer doing it's job properly. Rather than bothering the network guys to fix this (they are aware of it), I thought of a neat way to work around it.

We have a management server for us sysadmins located in the Data Center. I simply copied my ISOs there, turned on IIS, and added the following MIME Type:

Extension: .iso
MIME Type: application/octet-stream

Punch the URL to the file into Free Download Manager, and I'm now downloading over HTTP at ~700KB/sec without impacting the site (as this is properly QOS'd on our riverbeds).

Happy Days!

Saturday, September 14, 2013

Renaming Photos with Powershell

As I've mentioned previously, my wife and I have twins. One name Lucy, the other Abigail. We tend to get a little confused who is who, when reviewing photos so my wife asked me to come up with a simple way of renaming photos on the fly.

She normally plugs her phone or camera in to her laptop and copies the photos to a specific directory on our main backup share. I created 4 folders within the initial directory titled 'Abi', 'Lucy','Abi & Lucy', 'Lucy & Abi' to denote who is the only twin in the photo or left to right who is in the photo. She then runs the following script I wrote to append the first letter of their name to the end of the photo:


Function RenameFile($path,$newExt){

     Get-ChildItem -path $path | 

     Foreach-Object { 

          $baseName = $_.name.remove($_.name.length - $_.extension.length)
          Rename-Item -Path $_.fullname -newname ($baseName + " - " + $newExt + $_.extension) 
     }
}

RenameFile -path "H:\Photos\Abigail & Lucy\Abi" -newExt "A"
RenameFile -path "H:\Photos\Abigail & Lucy\Lucy" -newExt "L"
RenameFile -path "H:\Photos\Abigail & Lucy\Abi & Lucy" -newExt "AL"
RenameFile -path "H:\Photos\Abigail & Lucy\Lucy & Abi" -newExt "LA"

Tuesday, July 16, 2013

Using AdBlock to hide Dennis Reno's smug face

I use Chrome (where possible), and I hate ads. So I use the AdBlock plugin. It does a fantastic job at blocking most ads, but it also does a great job at removing all those annoying web parts you can live without.

For example, I spend a lot of my professional life logging into the My Oracle Support (MOS) website. Anyone that uses MOS, or deals with Oracle support will understand when I say...I hate Oracle, I hate their support model, but mostly, I hate staring at Dennis Reno's smug face every time I login.

Adblock allows me to simply select the bits I class as "ads" and hide them. To do this you select "block an ad on this page". Next you select the web part you class as an ad (in this case, Reno's face) click "looks good" then "block it!" and....









Much better!

Tuesday, July 9, 2013

Using Powershell to find duplicate directory names

I keep all my transcoded DVD's, in their own properly named directories, but spread across various volumes on my HTPC. Over the years I've moved my library across many disks and I knew I had at least a few duplicates because of this. There are many ways to skin a cat, but here's my taxidermy efforts:



Friday, July 5, 2013

Raspberry Pi Proxy Project

I'm a proud father of twin 8 month old girls. With fatherhood comes many, many fears, one of which is online safety. I was recently contemplating how (when the time comes) will I keep my girls safe from the dark areas of the internet. It was grounds for a proof of concept proxy, to at least understand what is possible, how easy is it to manage, and could it ultimately achieve a level of transparency between what my girls are doing online and what their parents are aware of.

Don't get me wrong, no acl can be a substitute for good parenting and an open, loving relationship between parents and their kids. But, I'm a child of the internet...and I can appreciate the value a proxy can bring the parents of connected children these days.

I'll be setting this up from a windows machine. There are a couple of things you'll need to get started:




I wrote the raspbian image to my SD card, booted up my rpi and performed the following:

Update repository and install squid:
$ sudo apt-get update
$ sudo apt-get install squid3

Modify squid configuration:
$ sudo vi /etc/squid3/squid.conf
Uncomment the following lines:
http_access allow localnet
acl localnet src 192.168.0.0/16

Restart squid service:
$ sudo service squid3 restart

Now I point my browser to 192.l68.1.5 on port 3128 (default squid port) and I'm now running through my pi proxy. To confirm this is working you can tail the access log like so:

$ sudo tail -f /var/log/squid3/access.log

All in all very easy to setup to this basic level. Next I'd like to play around with turning this into a transparent proxy, and generating decent reporting.

Friday, June 28, 2013

Repurposing an old laptop

I was looking to install a HTPC in the bedroom. I fall into a category not disimilar to the hobo-geeks. I have enough money to buy tech goodies, but am too tight fisted to spend it! I tried Raspberry Pi running raspbmc and openelec, but despite all the reviews I read, all the tips and advice, it just didn't cut the mustard for me. It's nowhere quick enough to tolerate my media library, and it's too flaky when it comes to SD card corruption.

What to do? I had an old Dell D620 laying around (i don't often through away machines) and thought it would do the job perfectly. I set about removing the screen, and extracting the wifi module/antenna to reuse. Here it is topless, and all screwed back together:


Next I needed an appropriate image that's easy to install, configure, quick to boot and reliable. I chose XBMCbuntu. You can get it here - http://mirrors.xbmc.org/releases/XBMCbuntu/. I added  an old MCSE remote I had laying around which worked out of the box. The finished result was a very quick booting, powerful (enough) PC capable of 720p playback via wireless. Easy to use, and achieved a high WAF.



Configuring SNMP on multiple Solaris 10 machines

There was a requirement to configure SNMP on dozens of Solaris 10 servers. I took one server, modified the /etc/sma/snmp/snmpd.conf file, tested it worked using snmpwalk, then wrote the following script to deploy that snmpd.conf as a template to every server in servers.list. Here's the script:

#!/bin/bash

for x in $(cat servers.list)
do
     echo "Server: $x"
     ssh root@${x} "mv /etc/sma/snmp/snmpd.conf /etc/sma/snmp/snmpd.orig"
     scp ./snmpd.conf root@${x}:/etc/sma/snmp/snmpd.conf
     ssh root@${x} "svcadm disable snmpdx; svcadm refresh snmpdx; svcadm enable sma; svcadm refresh sma; sleep 3; svcs -v sma"
done

Thursday, June 27, 2013

A practical use for Raspberry Pi

There are hundreds of Raspberry Pi projects online, each one interesting. I purchased my rpi a while ago in hopes it would replace my bedroom HTPC...unfortunately they're just not great at handling large media libraries. I did however find a very practical use for it when preparing for a family holiday.

We had rented an apartment at Noosa for the week and I knew whilst the accommodation was very nice, they only offered basic pay-tv. So I installed the latest openelec image using this guide:

http://squirrelhosting.co.uk/hosting-blog/hosting-blog-info.php?id=9

I took the rpi and a spare external HDD with me loaded up with must watch holiday movies. For anyone interested the reason I chose openelec over raspbmc was because I tested both quite thoroughly and OpenElec dealt with displaying thumbnails much better (smoother) than raspbmc in XBMC's default confluence interface.

When we booked I asked what type of TVs they have in the living/bedroom and was delighted to find out they were relatively new Samsung models so the CEC function worked great. So I didn't even need to take a remote with me!

Here's what it looked like:



Australian Radio Plugins for XBMC

My wife was using the ABC website to listen to Triple J on our HTPC. I did a little research and realised writing a plugin to do this within XBMC was pretty straight forward. While I was at it I wrote one to listen to RRR too.

EDIT: I've just created and uploaded a plugin for Double J. Thanks everyone for the feedback and suggestions.

Double J Plugin - Download here
Triple J Plugin - Download here
RRR Plugin - Download here


As Zan has written in the comments section,  to add these plugins to the home screen (as displayed below) perform the following:

System -> Settings
Appearance -> Skin Settings
Add-on Shortcuts
Home Page Music Submenu
Addon 1 Triple J
Addon 2 Three Triple R


It looks like this:



Scripting LUN path count and device info

During a recent project I cut hundreds of LUNs to present to a few Solaris 10 servers. I wanted a way to check the total number of LUNs for each server, and that each LUN was properly dual pathed. To do this, I wrote the following shell script. It will list out the LUN device path and number of paths, comma seperated, and the total number of LUNs at the end.

Remember if you're running this on a machine which has hundreds of LUNs presented to it, you're best directing the output to a file to prevent the screen buffer refresh slowing down the return rate of your results.

#!/bin/ksh

a=0
print "Device_Path,Path_Count"

# probe for LUNS
luxadm probe|grep Logical|
while
     read CRAP DEV_PATH
do

     # increment LUN count
     let a+=1

     # print device path
     print -n "${DEV_PATH:##Path:},"

     # check we have dual paths on each LUN
     pathCount=`luxadm display ${DEV_PATH:##Path:}|grep State|wc -l|tr -d ' '`

     if [ $pathCount -lt 2 ]; then
          errorMessage=" - WARNING: LUN does not contain redundant path"
     fi

     # check for path status other than online
     offlinePath=`luxadm display ${DEV_PATH:##Path:}|grep State|grep -v ONLINE`

     if [ -z $offlinePath ]; then
          offlinePath=""
     else
           errorMessage="$errorMessage - WARNING: 1 or more paths detected as NOT ONLINE"
     fi

     # print number of paths and path error if an offline path was found
     print "$pathCount $errorMessage"

done

# print number of LUNS discovered
echo "$a LUNS presented."

Monitoring Solaris zpools with HP OMW


Recently, we've had a few degraded zpools go unnoticed. They went unnoticed because our HP agents weren't monitoring their state (seems fair enough). After having a poke around what logs were available natively (not much) I broached the subject with our Tools Admin. He's always happy to help, so we knocked up the following perl script that gets executed every 15 minutes and reports in the event a zpool is in any state other than ONLINE.

use strict;

sub get_zp_list {
     my $pool_names = `zpool list -H -o name`;
     if ($pool_names eq "no pools available\n") {
          return $pool_names;
     }
     my @pool_names = split("\n",$pool_names);
     return @pool_names;
}

sub get_zp_health {
     my $pool_name = shift;
     my $result = `zpool list -H -o health $pool_name`;
     chomp $result;
     return $result;
}


#START PROCESS>>>

if ( ! -f "/sbin/zpool") {
     $ConsoleMessage->Severity("Minor");
     $ConsoleMessage->Application("zfspool");
     $ConsoleMessage->MsgText("zpools not supported in this sun release");
     $ConsoleMessage->Send();
     exit 1;
}

zpcheck();

sub zpcheck {
     my @zpools = get_zp_list();
     if ($zpools[0] eq "no pools available\n") {
          $ConsoleMessage->Severity("Minor");
          $ConsoleMessage->Application("zfspool");
          $ConsoleMessage->MsgText("no pools defined!");
          $ConsoleMessage->Send();
          return 0;
     }

     foreach my $pool (@zpools) {
          my $health = get_zp_health($pool);
          if ( $health eq "ONLINE" ) {
               # do nothing
          } else {
               $health = "unknown" if (!$health);
               # alarm in HP OMW
               $ConsoleMessage->Severity("Major");
               $ConsoleMessage->Application("zfspool");
               $ConsoleMessage->Object("$pool");
               $ConsoleMessage->MsgText("DNP- Problem with zpool $pool detected, status is $health");
               $ConsoleMessage->Send();
          }
     }
}

Launch SSH connections like RDP

I'm a sysadmin for a living, primarily dealing with UNIX and Windows servers.  Seeing as there's an everyday requirement to cross between these two platforms I've tried many solutions to better manage the connection process. I don't like any of them, and I'd much prefer to just launch a connection to a server when needed, do what's required, then exit.

To achieve this I'll use mstsc /v server_name to connect via RDP to a Windows Server, or Putty to connect to a UNIX or Linux server. Establishing Putty connections take too long so I sought for a shortcut similar to mstsc from the run dialog box, here's a quick and easy solution:

1) Modify your PATH variable
* Right click Computer and select Properties
* Select Advanced System Settings on the left pane
* Click the Advanced tab
* Click Environment Variables
* Select Path from System  Variables and click Edit
* At the end of the Variable value field add ;Your_Path_To_Putty
* Click OK and close the system properties window

2) Add a putty shortcut
* Browse to your putty path
* Right Click > New > Shortcut
* Enter the path to putty followed by -ssh (eg. C:\Putty\putty.exe -ssh)
* Click Next
* Enter a shortcut name of ssh and click Finish
* Right click your new shortcut and select Properties
* Select the Shortcut tab
* Change the Run dropdown box to reflect Maximized

Now bring up the run dialog box and type ssh server_name. If you followed along, Windows should launch a putty SSH session to your server.