Skip navigation
Currently Being Moderated

Dev chat 11/20/2008

VERSION 1 
Created on: Sep 14, 2009 11:20 AM by Noel Brockett - Last Modified:  Sep 14, 2009 11:20 AM by Noel Brockett

[2008-11-20 10::03:52] jb: they aren't here yet though

[2008-11-20 10::04:00] perr0: they are late :)

[2008-11-20 10::04:09] jb: theres mray..

[2008-11-20 10::04:39] mrayzenoss: I'm here

[2008-11-20 10::06:22] cgibbons: crickets

[2008-11-20 10::06:45] mrayzenoss: cgibbons can answer your WMI questions

[2008-11-20 10::06:48] jb: mray, my question is referring to a thread I have in the zenoss-dev forums/list regarding graphing the change in a datasource(integer) from poll to poll specifically using winperf..

[2008-11-20 10::07:08] adytum-bot: Title: Forums :: View topic - Graphing Averages (Day/Week/Month) (at forums.zenoss.com)

[2008-11-20 10::07:28] mrayzenoss: yeah, I remember asking you to bring it up there. Still no progress?

[2008-11-20 10::07:39] jb: well, it seems like I have made some progress using a regular SNMP datasource

[2008-11-20 10::08:30] cgibbons: was that using the zenpack in enterprise that does winperf or the community one, jb?

[2008-11-20 10::08:37] jb: im using enterprise..

[2008-11-20 10::08:50] cgibbons: k, and with what version? (sorry if this is in the thread, multi-tasking)

[2008-11-20 10::09:35] jb: 2.2.4

[2008-11-20 10::09:36] perr0: mrayzenoss: to start off I have a simple question. I installed zenoss on a box which **was** also being used a syslog server. We learned that this is no fun especially when we have thousands of syslog messages coming through. Is there a way to disable this in zenoss but then would defeat one of the purposes.

[2008-11-20 10::10:49] jb: cgibbons: the thread has a screenshot of the graph, and of the graph defintino

[2008-11-20 10::11:30] cgibbons: k. i'm not sure about the rrd stuff myself, but i can throw out some info on the winperf info. in 2.2.4 and earlier collection was based upon the typeperf utility (which you can manually run on the windows server). in 2.3.x it's based upon direct polling of the data, so we've been working hard on making sure the data is accurate. it should also be accurate with typeperf as well but i'm not...

[2008-11-20 10::11:46] mrayzenoss: perr0: what's the issue specifically? That zensyslog is interfering with syslog?

[2008-11-20 10::12:03] cgibbons: lemme look closer @ the thread

[2008-11-20 10::12:08] mrayzenoss: perr0: or that performance sucks?

[2008-11-20 10::12:14] * __pok joined the chat room.

[2008-11-20 10::12:37] __pok: goodday

[2008-11-20 10::13:01] perr0: the perfomance is degraded badly so we are in the process of setting up syslog on another box

[2008-11-20 10::13:47] jb: cgibbons: when using the "Test" function on the datasource, it is accurate.. but somehow by the time it gets to the graph, its not.. the goal here is basically take the difference between POLL A and POLL B to get the change..

[2008-11-20 10::13:57] * kells joined the chat room.

[2008-11-20 10::14:01] cgibbons: hey kells

[2008-11-20 10::14:21] kells: Hey Chad!

[2008-11-20 10::14:25] mrayzenoss: perr0: so the bottleneck for syslog events is probably the mysql database

[2008-11-20 10::14:58] kells: Hmmm... I don't think the mysqltuner reference made it into the Admin Guide yet. :(

[2008-11-20 10::15:15] mrayzenoss: kells: that's where I was headed :)

[2008-11-20 10::16:02] adytum-bot: Title: MySQLTuner - MySQLTuner (at wiki.mysqltuner.com)

[2008-11-20 10::16:18] perr0: mrayzenoss: thank you, I did that already but it didn

[2008-11-20 10::16:50] * ericnewton joined the chat room.

[2008-11-20 10::16:59] jb: this has to be somethign simple.

[2008-11-20 10::17:05] cgibbons: quite likely

[2008-11-20 10::17:05] jb: since it works with snmp datasources

[2008-11-20 10::17:22] * chudler joined the chat room.

[2008-11-20 10::17:33] jb: is winperf returning multiple values per poll?

[2008-11-20 10::17:37] mrayzenoss: hmmm... could you open a ticket as an enhancement? Outline what you did and maybe we can improve syslog processing performance at some point

[2008-11-20 10::17:46] cgibbons: hurm you know I think it returns multiple ones

[2008-11-20 10::17:52] jb: that could be the problem

[2008-11-20 10::18:20] cgibbons: yeah. can you run zenwinperf from the commannd line to see what kind of output you get there?

[2008-11-20 10::18:34] jb: let me see if I can parse thru that again.

[2008-11-20 10::18:51] cgibbons: 2.3.1 should be out in a couple of days hopefully :)

[2008-11-20 10::19:33] perr0: mrayzenoss: where you talking to me about syslog?

[2008-11-20 10::19:42] mrayzenoss: perr0: yes please

[2008-11-20 10::19:49] jb: i'll probably upgrade when 2.3.1 comes out

[2008-11-20 10::19:53] perr0: that would actually be great, I will do that cause it really hurts

[2008-11-20 10::20:11] jb: cgibbons: should zenwinperf ever actually complete?

[2008-11-20 10::20:14] cgibbons: the winperf stuff is completely different, and a should be a whole lot better. the internals are different so it scales better, etc.

[2008-11-20 10::20:19] jb: it seems to just sit indefintely

[2008-11-20 10::20:22] mrayzenoss: perr0: so you're doing what order of magnatude? tens a second? hundreds?

[2008-11-20 10::20:37] cgibbons: depends upon how you run it. zenwinperf run -v10 -d blahblah; would be ideal, no -c flag.

[2008-11-20 10::20:45] jb: thats what I'm doing..

[2008-11-20 10::21:06] perr0: mrayzenoss: syslog messages? probably hundreds

[2008-11-20 10::21:24] mrayzenoss: yeah, we probably have some work to do there, sorry

[2008-11-20 10::21:37] jb: cgibbons: sort of hard to understand the output.. i don't see actual values in the logs

[2008-11-20 10::22:00] cgibbons: shoot. i didn't work with the older version much so i don't know off the top of my head how it works. i was hoping it would dump the typeperf output.

[2008-11-20 10::22:26] garnica: wow..... I am still using 2.1.3 'cause I am still planning upgrade to 2.3... any suggestion ??

[2008-11-20 10::22:36] jb: heh.. prepare for the worst.

[2008-11-20 10::22:44] mrayzenoss: garnica: The upgrade path is 2.1.3->2.2.4->2.3

[2008-11-20 10::22:48] perr0: mrayzenoss: np, I am so thankful for zenoss. You guys do great work

[2008-11-20 10::22:52] * dorferiferon joined the chat room.

[2008-11-20 10::22:54] __pok: cgibbons: so you're saying I could just as well just wait a few days instead of upgrading from 2.2.4 to 2.3.0 today?

[2008-11-20 10::22:54] mrayzenoss: perr0: Thanks!

[2008-11-20 10::23:24] garnica: mrayzenoss, ok I''ll check it out

[2008-11-20 10::23:26] cgibbons: 2.3.1 is just the enterprise release of 2.3.0, with a few fixes in the enterprise zenpacks over 2.3.0

[2008-11-20 10::23:29] jb: cgibbons: if winperf is returning multiple values, then I think this will screw everything up..

[2008-11-20 10::23:53] * Bulwinkle left the chat room.

[2008-11-20 10::23:54] perr0: if you guys had a wish list of redbull or something I'd send you some

[2008-11-20 10::24:10] * thutchis joined the chat room.

[2008-11-20 10::24:24] cgibbons: yeah. i think ecn will know for sure if he wakes up :) let me see if i've got a 2.2.4 instance somewhere to look

[2008-11-20 10::24:31] ericnewton: huh, wha?

[2008-11-20 10::24:32] jb: ok :)

[2008-11-20 10::24:43] mrayzenoss: perr0: maybe we can catch up at a conference or something. :)

[2008-11-20 10::24:48] adytum-bot: Title: Forums :: View topic - Graphing Averages (Day/Week/Month) (at forums.zenoss.com)

[2008-11-20 10::24:52] jb: ecn: using 2.2.4 enterprise/winperf

[2008-11-20 10::25:11] garnica: what is the recommended os for running zenoss ?? RHEL 5 ?? Cent OS ?? Fedora ??

[2008-11-20 10::25:24] jb: garnica: rhel/cent will be fine..

[2008-11-20 10::25:29] mrayzenoss: garnica: Most of our enterprise customers use rhel/centos

[2008-11-20 10::25:57] cgibbons: ecn: we're trying to figure out if the old winperf collector maybe returned multiple values at once?

[2008-11-20 10::26:02] garnica: ok

[2008-11-20 10::26:13] ericnewton: define "returned"

[2008-11-20 10::26:37] jb: when zenwinperf queries a winperf datasource, is it returning a single value?

[2008-11-20 10::26:59] ericnewton: yes, a single value

[2008-11-20 10::27:13] jb: hm, well then I have no idea where its getting these numbers

[2008-11-20 10::27:48] ericnewton: is it the case that windows is turning your counter into a gauge value? I don't think I understand what the problem is

[2008-11-20 10::28:10] jb: the problem is outlined with a screenshot in that thread

[2008-11-20 10::34:31] ericnewton: jb: is it possible that the rrd file was created before with the wrong spec?

[2008-11-20 10::34:52] jb: ericnew: i created a new graph definition to test that..

[2008-11-20 10::35:09] ericnewton: with a new datapoint name?

[2008-11-20 10::35:25] jb: hm, good point.. I deleted the datapoint and recreated it, but I think I used the same name.

[2008-11-20 10::35:46] * __pok left the chat room.

[2008-11-20 10::37:23] * kim0 left the chat room.

[2008-11-20 10::37:40] jb: I'll recreate it now.

[2008-11-20 10::38:05] kells: Looking for some feedback on the Dev Guide, specifically the 2.3 version. Thoughts?

[2008-11-20 10::39:33] ericnewton: cool... ian just twittered that Zenoss is #1 at SourceForge

[2008-11-20 10::39:46] mrayzenoss: ericnewton: That's just currently for the day

[2008-11-20 10::39:59] jb: kells: we need details on how to create custom modelers!

[2008-11-20 10::40:17] perr0: mrayzenoss: I have to have zenoss pro or enterprise before I can raise a ticket?

[2008-11-20 10::40:22] kells: So the section on that in the guide didn't help?

[2008-11-20 10::40:23] ericnewton: mray: right... still cool

[2008-11-20 10::40:30] jb: it's just a placeholder right now

[2008-11-20 10::40:38] kells: In 2.3.1?

[2008-11-20 10::40:44] jb: ooh, i havent seen 2.3.1

[2008-11-20 10::40:46] mrayzenoss: perr0: only for support. You can open a defect http://dev.zenoss.org/trac/

[2008-11-20 10::40:47] kells: Sorry, in the 2.3 Dev Guide?

[2008-11-20 10::40:55] jb: let me grab it.

[2008-11-20 10::42:00] mrayzenoss: perr0: no guarantees that it will get worked on immediately, but at least it will be under consideration and get reviewed. I'm doing a little syslog and not seeing issues, but I don't know about other users and their scalability

[2008-11-20 10::42:06] kells: There might be a refresh of the Dev Guide in both PDF and HTML when 2.3.2 comes out. The Dcobook source is there (sans XSL :( ) so it can be created for the impatient. ;)

[2008-11-20 10::42:29] jb: this is in the dev guide, right?

[2008-11-20 10::42:37] ericnewton: perr0: you are having issues with syslog scalability with a hundred messages per second?

[2008-11-20 10::42:48] * mloven joined the chat room.

[2008-11-20 10::42:52] kells: In the last week or so there's been further doc updates with typo fixes, diagrams and (today) some more stuff that will be coming up in 2.4

[2008-11-20 10::43:03] * tr_ joined the chat room.

[2008-11-20 10::43:37] kells: Section 10: Adding a new Device Type

[2008-11-20 10::43:42] perr0: ericnewton: yes, the box is down right now so I can't tell you the exact number of messages. I know by the days end, the log is almost 10 gigs

[2008-11-20 10::43:57] jb: chapter 10?

[2008-11-20 10::44:03] kells: There's also two new sections: Chapter 9 Zenoss Daemons and Chapter 10: Add a Performance Daemon

[2008-11-20 10::44:12] perr0: not 100% sure at this time

[2008-11-20 10::44:24] jb: ah god it.

[2008-11-20 10::44:27] * pjz joined the chat room.

[2008-11-20 10::44:43] ericnewton: perr0: ok... interesting... the other event collectors handle up to 2K events/sec

[2008-11-20 10::44:45] jb: kells: I'll take a look at that.. thanks.

[2008-11-20 10::45:03] ericnewton: perr0: usually on some fast hardware (like the Zenoss large appliance)

[2008-11-20 10::45:04] pjz: Howdy, I just set up Zenoss and am getting a ton of false alarms because some of the machines I'm monitoring are virtual and so a bit slow to respond... how do I up the timeout threshold ?

[2008-11-20 10::45:04] kells: Just to warn you that the formatting for the Dev Guide is still absolutely terrible. :)

[2008-11-20 10::45:52] ericnewton: pjz: SNMP timeouts?

[2008-11-20 10::46:01] pjz: ericnewton: yes

[2008-11-20 10::46:08] ericnewton: there's a zProperty for that

[2008-11-20 10::46:15] pjz: Message:

[2008-11-20 10::46:18] tr_: How does zenoss map an SNMP trap to a given EventClass? I have a problem with a trap being identified with an invalid EventClass

[2008-11-20 10::46:38] pjz: okay, how do I find it?

[2008-11-20 10::46:49] jb: ok, come on graphs.

[2008-11-20 10::46:51] perr0: mrayzenoss: I don't expect this to get worked on right away. I am happy with moving zenoss to a different box. We tried doing a few steps such as the mysql tuner script, we disabled rdns in syslog, but to no avail the .python.bin was eating up all the CPU

[2008-11-20 10::46:58] pjz: ericnewton: is that zCommandTimeout?

[2008-11-20 10::47:06] * daans left the chat room.

[2008-11-20 10::47:16] ericnewton: zSnmpTimeout

[2008-11-20 10::47:42] * atrawog joined the chat room.

[2008-11-20 10::47:45] jb: kells: so is there planned native AIX support since the documentation uses it?

[2008-11-20 10::47:49] mrayzenoss: perr0: Yeah, a ticket will at least get it on our radar, maybe ericnewton will check it out in his free time :p

[2008-11-20 10::47:53] jb: that would be great :)

[2008-11-20 10::48:21] ericnewton: pjz: the default is 2.5 seconds

[2008-11-20 10::48:48] pjz: ericnewton: yeah, I upped that to 5

[2008-11-20 10::48:55] kells: jb: In the 2.4 release there is a plan to add more support for the Big 3 Unices

[2008-11-20 10::49:10] jb: big 3 unices?

[2008-11-20 10::49:18] kells: Solaris, AIX, HPUX

[2008-11-20 10::49:22] perr0: I hate to pest, but I am on the link you provided, can't login. I would hate to let this go and not provide good information on this concern

[2008-11-20 10::49:25] jb: cause my AIX is a hackjob right now

[2008-11-20 10::49:29] ericnewton: perr0: it would be nice to know which python program was the problem child: zenhub or zensyslog

[2008-11-20 10::49:36] jb: i'l love to redo it

[2008-11-20 10::49:51] kells: AIX's snmpd is, errmmm, a little interesting to deal with at times.

[2008-11-20 10::50:13] jb: heh yes.

[2008-11-20 10::50:16] mrayzenoss: perr0: username zenoss

[2008-11-20 10::50:21] kells: Personally, I've had a few issues with it going down, but it may have been my environment. :)

[2008-11-20 10::50:26] jb: which is why I use net-snmp and proxy certain mib's to it

[2008-11-20 10::50:38] pjz: ericnewton: oh, oop, okay, so I'm also getting SMTP timeouts waiting for a 220 response

[2008-11-20 10::50:42] mrayzenoss: perr0: password zenoss

[2008-11-20 10::50:54] kells: I haven't tried the AIX 6.1 series snmpd though, so maybe that's better. :)

[2008-11-20 10::51:01] jb: kells: so the AIX implementation would use strictly AIX's snmpd, no net-snmp?

[2008-11-20 10::51:03] kells: jb: Are you using VIO servers?

[2008-11-20 10::51:03] jb: i haven't either..

[2008-11-20 10::51:10] ericnewton: perr0: zenhub scalability is improved in 2.3.1

[2008-11-20 10::51:10] perr0: mrayzenoss: thank you much

[2008-11-20 10::51:13] jb: not monitoring them currently though

[2008-11-20 10::51:15] * tr_ left the chat room.

[2008-11-20 10::51:46] ericnewton: pjz: is this a command test (a nagios style plugin?)

[2008-11-20 10::51:50] kells: They're running the native AIX snmpd, so the ZenPack.community.AIX zenpack might help.

[2008-11-20 10::51:52] * tr_ joined the chat room.

[2008-11-20 10::52:01] * pens left the chat room.

[2008-11-20 10::52:20] jb: kells: also using some custom scripts to get the serial number/AIX version.. so please keep this in mind when developing native support.. these are important :)

[2008-11-20 10::52:20] pjz: ericnewton: uhh... it's whatever default smtp monitoring shows up when you tell zenoss something is a linux host

[2008-11-20 10::52:26] kells: The native AIX, Solaris, HPUX support is geared towards SSH-modeling, not SNMP. (Only so many hours in a day. :) )

[2008-11-20 10::52:28] jb: piping them in via net-snmp

[2008-11-20 10::53:31] kells: serial #, AIX version is gathered by the community.AIX zenpack. Getting it to work for some environments may be a challenge sometimes, though. ;)

[2008-11-20 10::53:45] jb: yeah i helped with that modeler code

[2008-11-20 10::54:08] * tr_ left the chat room.

[2008-11-20 10::54:15] kells: "Implementation is trivial" is a phrase that *seems* helpful, but... :)

[2008-11-20 10::54:16] * tr__ joined the chat room.

[2008-11-20 10::54:30] kells: jb: Ahh, I *thought* that might have been you. Thanks!

[2008-11-20 10::54:42] jb: oh hold up

[2008-11-20 10::55:24] pjz: ericnewton: or, hrm, maybe I did set this up. It's not a nagios test, it's /IpService/Privileged/smtp

[2008-11-20 10::56:11] jb: kpg123: I have been thinking about wiping my existing AIX/net-snmp/snmpd plan and trying the zenpack..

[2008-11-20 10::56:23] ericnewton: pjz: right, I'm just trying to figure out the answer

[2008-11-20 10::56:31] jb: err. kells

[2008-11-20 10::56:33] atrawog: jb: Writing an AIX modeler is pretty easy, what's tough is that there isn't support for volume group / logical volume / file system relationships in Zenoss.

[2008-11-20 10::56:52] jb: yeah

[2008-11-20 10::56:57] kells: jb: Since I joined Zenoss, I've switched to kells :)

[2008-11-20 10::57:05] jb: kells: I wasn't aware you had joined zenoss :)

[2008-11-20 10::57:25] kells: Thanks! It's been very fun! Lots of work, but very fun too!

[2008-11-20 10::57:29] jb: kells: are you based out on anapolis?

[2008-11-20 10::57:36] kells: Calgary, Alberta, Canada :)

[2008-11-20 10::57:56] jb: ah ok.. i'll be at zen offices in february, and I'd love to try and nail some of this AIX stuff out

[2008-11-20 10::58:37] kells: atrawog: Unfortuntaely, the storage stuff is probably out of scope for the moment in 2.4. (Only so many hours in the day. :( )

[2008-11-20 10::58:54] jb: ericnewton/cgibbons: works like a charm now.. used different datasource names!

[2008-11-20 10::59:13] mrayzenoss: jb: awesome, could you followup on your forum thread with the fix?

[2008-11-20 10::59:17] perr0: another quick question, how often are the install guides updated? Is the forum thought of the aid for users with their installation questions?

[2008-11-20 10::59:18] jb: yep.

[2008-11-20 10::59:45] * tr__ left the chat room.

[2008-11-20 11::00:06] mrayzenoss: perr0: The install guides get revved every time there's a new release and there's something new. So 2.3 added a bunch of stuff, and 2.3.2 will have the OSX stuff integrated into it

[2008-11-20 11::00:09] ericnewton: pjz: zStatusConnectTimeout

[2008-11-20 11::00:34] mrayzenoss: perr0: And yes, the forums are for Core support, with the IRC sessions every 2 weeks

[2008-11-20 11::00:58] kells: Please take any comments about the 2.4 directions as being directions (of course) and not contractually binding etc :)

[2008-11-20 11::01:30] jb: sure

[2008-11-20 11::01:41] kells: Matt Ray's working on the full list :)

[2008-11-20 11::01:44] jb: would you say its production-ready?

[2008-11-20 11::03:25] kells: I really hesitate to say that because I haven't put in some patches I've received from people and I haven't had an opportunity to fully exercise a lot of different things (ie it needs some TLC). I would think that it might be worth a try to see (in a limited deployment) how well it works for you.

[2008-11-20 11::03:41] jb: sure

[2008-11-20 11::04:02] perr0: good to know. Well, I just want to say thanks again to Eric and mrayzenoss for the great work. Cheers!

[2008-11-20 11::04:05] kells: There are currently no plans for a TSM ZenPack. It needs some vocal enterprise supporters. :)

[2008-11-20 11::04:52] jb: sure

[2008-11-20 11::05:09] pjz: ericnewton: thanks@

[2008-11-20 11::06:43] ericnewton: go to that device

[2008-11-20 11::06:47] jb: i think thats probably on the /Devices/Server template?

[2008-11-20 11::06:52] kells: jb: I would love to create a TSM zenpack, but there's 250+ tickets waiting for me at the moment. :) And after the 2.4 release it will probably be a lot easier to do. Also, when TSM 6x comes out using the DB2 backend, that will also make things much easier to deal with at that point.

[2008-11-20 11::06:59] ericnewton: pull the menu, and find "Templates

[2008-11-20 11::07:16] jb: kells: awesome.. we are a TSM on AIX shop..

[2008-11-20 11::07:16] kells: ie better ODBC support

[2008-11-20 11::07:29] jb: i wonder if the windows version of TSM provides perfmon counters

[2008-11-20 11::07:36] pjz: ericnewton: okay, found it, and /Devices/Server template

[2008-11-20 11::07:44] kells: I'm pretty sure that it does

[2008-11-20 11::08:02] jb: yeah

[2008-11-20 11::08:18] ericnewton: pjz: all set then?

[2008-11-20 11::08:25] jb: is TSM info available via snmp in AIX?

[2008-11-20 11::08:30] kells: IIRC, Windows has been the primary platform for development of TSM since about the 4.x series.

[2008-11-20 11::08:38] mrayzenoss: jb: If you really need a TSM ZenPack, you could probably contact Client Services and they'd work on it

[2008-11-20 11::08:48] jb: ok

[2008-11-20 11::09:15] kells: jb: There's some interesting things about the TSM SNMP support especially under AIX.

[2008-11-20 11::09:22] jb: oh yeah?

[2008-11-20 11::09:30] pjz: ericnewton: yah, that's the right place, I see

[2008-11-20 11::09:50] * cgibbons left the chat room.

[2008-11-20 11::09:51] kells: a) the variables avaiable aren't the ones that I want (that's probably always the case)

[2008-11-20 11::09:56] mrayzenoss: Unless there's any more questions, I'm gonna let ericnewton off the hook to get back to work. kells and cgibbons were volunteers

[2008-11-20 11::10:06] pjz: ericnewton: what's the 'Escalate count' ? does it have to exceed the threshold that many times before it warns?

[2008-11-20 11::10:40] ericnewton: pjz: it just raises the priority by one after that many times

[2008-11-20 11::10:40] kells: b) the "run an arbitrary command and get back the results" part seems useful, but doesn't seem to be that much better that just running the command through dsmc

[2008-11-20 11::11:08] jb: no.

[2008-11-20 11::11:36] pjz: ericnewton: priority or severity?

[2008-11-20 11::11:41] ericnewton: severity

[2008-11-20 11::12:09] pjz: got it! Thanks!

[2008-11-20 11::12:12] * Freezerburn joined the chat room.

[2008-11-20 11::12:26] Freezerburn: omg guys I just got zenoss up

[2008-11-20 11::12:35] mrayzenoss: Freezerburn: congrats

[2008-11-20 11::13:30] kells: Doh! Have fun, matt!

[2008-11-20 11::13:32] mrayzenoss: Thanks everybody

[2008-11-20 11::13:53] ericnewton: yes, I have a meeting (and lunch) to get to, too

[2008-11-20 11::14:28] * ericnewton left the chat room.

[2008-11-20 11::14:50] kells: It's probably time for me to head back to the bugs, too!

[2008-11-20 11::14:57] jb: alrighty

[2008-11-20 11::14:58] kells: Thanks jb, atrawog!

[2008-11-20 11::14:59] jb: thanks guys.

[2008-11-20 11::15:11] atrawog: goodby

[2008-11-20 11::15:14] * kells left the chat room.

Comments (0)