emunctory: October 2007

Tuesday, October 30, 2007

The horse hair has snapped

"The Sword of Damocles is an often-used allusion ... epitomizing the imminent and ever-present peril faced by those in positions of power. More generally, it is used to denote a precarious situation and sense of foreboding, especially one in which the onset of tragedy is restrained only by a delicate trigger or chance. It can also be seen as a lesson in the importance of understanding someone's experience."

If thats all to cryptic then let me clarify - yes, the uni is discontinuing my position, will examine options for my redeployment and where or rather when, no opportunities can be found I will be retrenched.

Monday, October 29, 2007

I have had a wonderful weekend, thanks for asking. House dad on Saturday building a cubby hose from couch cushions whilst Alex did the weekly shopping and chose some shoes and then a short bush walk in the National Part out the back of the house in the afternoon. Sunday we trekked to the zoo, late due to a conspiracy between Adrian not sleeping most of the early early morning and refusing to let daylight savings thieve an hour of snoozes.

And now I am at work.

Waiting for the all staff meeting at 1:30 to discuss the merger of the division of the university I work for with another and find out whether I will still have a job and thus whether I will be looking for work over Christmas or enjoying the time off.

Tuesday, October 23, 2007

Animeme

Wow, I've been memed,

Thanks Mark for taking me with my first meme. I'm thankful it's a small one. I do wonder how these things start, do they reproduce, recombine, mutate, and evolve? What are the selection pressures acting on a blog meme? Is extinction forever or are they periodically revivified? Will the far right Christian lobby try to prevent meme evolution being taught in favor of meme design? I'm sure somebody studying meme fitness will investigate it one day? This meme has a name - "Animeme". Is it common to name a meme or are they more like the Gary Larson Commic book I have that had an index that was empty for every letter except for the letter 'T' which has all the cartoones indexed under their respective titles "The one about ......."

The meme:

An interesting animal I had
As a boy living on the edge of a National Park in Sydney's north I enjoyed almost daily interaction with the snakes, spiders, goanas, owls and all the other critters that invaded my space much as I was invading theirs. I kept a tank of red crowned toadlets for a few years. When I learned that they were endangered I carried their spawn, which they always created in vast quantities in my fish tank, back out into the bush whenever it rained. I hope I have helped this little species on the brink to hold on for a while longer.

An interesting animal I ate
As a geologist doing exploration work in the remote outback I had the opportunity to do a bush survival course where we mostly learned how to fix a broken 4x4, do remote first aid, fix a broken radio, burn spinafex to get attention, build a shelter and collect some water we did get to try some of the spiders, ants, and delicious Witchetty grubs.
The most memorable bush tucker was a meal around the camp fire prepared by an aboriginal camp hand. Over a few beers and what I thought was an unusually flaky but somewhat tough smoked fish One of the drillers asked him, "Hay cookie - you didn't eat that goanna you were feading at lunch did ya?" "Nah mate", replied the cook, "you eat him now".
On a less exotic note, depending on your perspective, you can roll up to the Australian Hotel, right next to the approach to the Harbor Bridge in Sydney and have kangaroo, emu & crocadile on a pizza - now that's Aussie.

An interesting animal in the museum
That would have to be Ekaltadeta the flesh eating kangaroo but it's a close competition. Check out some of the competitors

An interesting thing I did with or to an animal
I would not could not with a goat, I would not could not in a boat...
If using conodonts to date rocks around Parks doesn't cut it then I think that, when I was working as an environmental scientist and I used nematodes to characterize soil environments in and around one of Australia's most toxic polluted sites that that was pretty interesting. No?

An interesting animal in its natural habitat
Having one of these on you shoe right outside the front door is pretty interesting but I think that the female form of homo sapiens australes in it's native habitat at the beach is more so.

I'm tagging Rowena, andLeesha.

Thursday, October 11, 2007

Never let it be said

Never let it be said that I don't share my inner most self with you.

I've a form of Arthritis that's a right pain in the arse (SI joint to be exact). These were part of a set of about 400 scans taken a few weeks ago as part of an effort to try and better manage it's progression. Just all part of the fun of being human I guess.

Wednesday, October 10, 2007

How to plot how busy you server is from the user perspective

Do you run a mission critical application on an Apache web server?
if not stop reading - I'll post cute picks of kids or my MRI scan later.

click here if your a gheek and want to read more

How to plot how busy you server is from the user perspective.
The assumption I am making here is that the slower the server is the more user connections I will see because they spend more time connecting because the server is slower etc. etc.

In effect I'm going to plot the "requests being processed" against time.

So you've got mod_status installed and you have a cron job gziping and taring wgets of the server_status page

on a typical day head -n30 looks like this:


Apache Server Status for XXXXX.XXXXX.edu.au

Server Version: Apache/2.0.55 (Unix) DAV/2 mod_ssl/2.0.55 OpenSSL/0.9.6m mod_jk/1.2.6
Server Built: Jan 4 2006 17:25:42

Current Time: Wednesday, 10-Oct-2007 15:23:08 EST
Restart Time: Tuesday, 09-Oct-2007 17:24:28 EST
Parent Server Generation: 1
Server uptime: 21 hours 58 minutes 40 seconds
Total accesses: 1756415 - Total Traffic: 28.1 GB
CPU Usage: u1452.51 s1680.81 cu6053.03 cs0 - 11.6% CPU load
22.2 requests/sec - 372.4 kB/second - 16.8 kB/request
129 requests currently being processed, 41 idle workers

K_KCKKKWKWKKRWCKKKKKKKKKKKCK_KKRK_KKKKK_K._K.K.KKKKKWKCK.._KKKK_
.K_._._CKK__KK_K.KKK_KKWKKKK_KK..KK._.KK.K___KK.WK_KKCK_K_K_.KK_
.KKKKKK__WR_K.KK.K_KKWKKR.K._K__K..KKK_K_K_W.KK.K...............
....................................._..........................
....................._.................................K........
................................................................
..........................................K................_...K
......_.._..........................................K...........
................................................................
................K........K......................................
...........W....................................................
..............W.................................................
................................................................
.........._.................................................._..
.................................K..............................
........K...........................................K...........

And when you are being subjected to a DOS attack it looks more like this


Current Time: Monday, 08-Oct-2007 15:59:21 EST
Restart Time: Sunday, 09-Sep-2007 14:19:25 EST
Parent Server Generation: 29
Server uptime: 29 days 1 hour 39 minutes 56 seconds
Total accesses: 38331840 - Total Traffic: 326.1 GB
CPU Usage: u749.13 s842.35 cu2791.89 cs0 - .175% CPU load
15.3 requests/sec - 136.1 kB/second - 8.9 kB/request
1024 requests currently being processed, 0 idle workers

GRRRGRRRRRRRRRRGRRRRRRRRRRRRRRRRRKRRRRRRRRRRRRKRRKRRRRGRRRRRRRRR
RRRWGRGRRRRRRRRRRRRRRRRKRRRRRRRRRRRRGRKRRRRRRRRRRRGRRRRRRRRRWRRR
RRRGRRRRRGRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRGRGRKRRRCRRRKKRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRGRRRRRRRRRRRRRRRRRRKRRRRRRRRRRKRRRRR
RRRRRRRRRRRRRRRRRRRKKRRRRRRKRRRRKRRRRRRRRRRRRRRKRRRRRRRGRRRKRRRR
RRGRRRWRRGRRRRRRRRRKRGRRRRRRGGRRRRRRRRRRRRRRRRKRRRRRRRRRRKRRRRKR
RRKRRRRRRRRRRRRRRRRRRRRRRRRGRRRRRRRRRRRRRRRKRRRRKRRRRRRRRKRRRRRR
KRRRRRRRRRRKRRRRRRRRRRRRRRRRKRKRKRRRRRRRRRRRRRRRKKRRCRRRRRRRRKRW
RRKRCRKRRKRRRRRRRRRRRRRWRKKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRRRRRRRRRRRRRRRCRR
RRRKRRRKRRRRRRRRWRRRRRKRRRRRRRRRRRRRRRRRRRRRRWRRRRRRRRRRRRRRRRRR
KRRRRRRGRRRKKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKKRRRRRRRRRRRR
RRRRRRRRRRRRRRKRKRRRRKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRWRKRRRRRRRRRR
KRRRRRKRRRKRRKRRRKRKRRRKRRRRRKRKRKRRRKRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRKKRRRRRRRRRRRRRRRRKRRRRKRRRRRRRRRRRRKKRRRRRRGRRRRRRRRRRRKRRK
KRKRRKRKRRRRRRRKRKRRRRKWKRRRRRRRKRRKRKRKRKRRRCRRRCKRCKRRRKRKRKKC

where all those R's are ..Reading..

Method

Write a script that loops through all your tar files untars and ungzips them then greps the head to get the date and the requests line. We gzip and tar the original pages into a directory - opt/servstat and then tar together whole days so mine looks like this but your mileage may vary:


#!/bin/bash

# Clear the screen in preparation for script output
clear
echo "--------------------------------------------------"

dir=`pwd`
startdir=$dir

if [ -d "tmp" ] ; then
 rm -r tmp
fi

if [ -d "~/Desktop/out" ] ; then
 rm -r ~/Desktop/out
fi
mkdir tmp
mkdir ~/Desktop/out

for file in $*; do
 (
  cd tmp

  tar zxvf ../$file
  (
   cd opt/servstat
   gunzip *.gz
   sudo chown stephen\: *
   mv *.txt ~/Desktop/out   
   head -n17 /home/stephen/Desktop/out/*.txt | grep -i Current >> ~/Desktop/outputFile
   rm ~/Desktop/out/*.txt 
  )
 )
 
done

cd $startdir

exit 0

end result being I get a file some 100K lines ling that looks like this;


<dt>Current Time: Wednesday, 10-Oct-2007 08:35:00 EST</dt>
<dt>69 requests currently being processed, 36 idle workers</dt>
<dt>Current Time: Wednesday, 10-Oct-2007 08:40:00 EST</dt>
<dt>103 requests currently being processed, 21 idle workers</dt>
<dt>Current Time: Wednesday, 10-Oct-2007 08:45:00 EST</dt>
<dt>78 requests currently being processed, 46 idle workers</dt>
<dt>Current Time: Wednesday, 10-Oct-2007 08:50:00 EST</dt>
<dt>80 requests currently being processed, 44 idle workers</dt>
<dt>Current Time: Wednesday, 10-Oct-2007 08:55:01 EST</dt>
<dt>111 requests currently being processed, 52 idle workers</dt>

you can get rid of the HTML if you want sed -e 's/<[^>]*>//g' outputFile > outputFileNoHTML

but chances are you want to keep it so that you can use it with sed to get the time, requests currently being processed, and idle workers all on the same line

if you want a csv to put into excel use this almighty "one liner"

cat outputFile | sed -e 's/<dt>Current Time: /\n/g' | sed -e :a -e '$!N;s/\n</ /;ta' -e 'P;D'| sed '/^$/d' | sed -e 's/ EST<\/dt> dt>/,/g' | sed -e 's/ requests currently being processed, /,/g' | sed -e 's/idle workers<\/dt>//g' > outputFile.csv

To Generate a tab seperated file use this:
cut -d ',' -f 2,3,4 aug.csv | sed 's/2007 /2007-/g' | sed 's/,/\t/g' > aug.tab
cut -d ',' -f 2,3,4 sep.csv | sed 's/2007 /2007-/g' | sed 's/,/\t/g' > sep.tab

or do the whole thing
cut -d ',' -f 2,3,4 outputFile.csv | sed 's/2007 /2007-/g' | sed 's/,/\t/g' > out.tab

now grep out the months if you want or deal with the whole file, i.e for September
grep Sep outputFile.csv > sep.csv
or
grep Sep outputFile.tab > out.csv

I'm using gnuplot to generate the plots. Here is the load file you want, in this case for all of August



# Gnuplot script file for plotting data from server_status
# This file is called   aug.plot
unset log    # remove any previous log-scaling
unset key    # remove any previous key
unset label    # remove any previous labels
unset title    # remove any previous title
set xtic auto    # set xtics automatically
set ytic auto    # set ytics automatically
set yrange [0:1028]   # Change the y-axis range
set xtics rotate                        # rotate the x-axis lables so they are verticle
set terminal jpeg size 1280, 780 # make a pretty picture for the output
set output 'aug.jpg'                    # name of the output file 
set xdata time                          # set x axis to be time scale
set timefmt "%d-%b-%Y-%H:%M:%S"         # set the input format of the time
set format x "%d/%m"                    # set the output format of the time label
set grid                                # show a grid in the picture
set title "Number of user connections"  # the title of the output plot
set xlabel "Date/Time"                  # label the x axis
plot "aug.tab" using 1:2                # generate the plot using column 1  for x and 2 for y

if you want to plot a single day you want different x-axis labeling, use:
set format x "%d/%m %H:%M"

from within gnuplot to generate the plot from this plot file enter: load 'aug.plot'

You should now get a pretty picture.

Mine look like this:

For the year since the end of July