48,899 Views 10 Replies Last post: Oct 9, 2009 11:24 AM by evilk RSS
ccole Rank: White Belt 18 posts since
Aug 18, 2008
Currently Being Moderated

Sep 24, 2008 3:34 PM

Per filesystem thresholds, is this a hack?

My management is very interested in Zenoss, but we've got a few challenges. The first challenge is that we have a number of Windows machines that have large shadow copy volumes. Those are intended to fill up, and alerting for those isn't particularly useful. The second challenge is that a one-size-fits-all threshold doesn't actually fit all.

The solution we've fixed upon is to put a custom threshold into the volume name and work off that. Here's the code used in the /Perf/Filesystem event transform (based off another user's change that made for more understandable messages)


import re

perDevicePattern = re.compile(r'zz(\d{3})')
fs_id = device.prepId(evt.component)
for f in device.os.filesystems():
    if f.id != fs_id: continue
    p = (float(f.usedBytes()) / f.totalBytes()) * 100
    freeAmtGB = (float(f.totalBytes() - f.usedBytes())) / 1024 / 1024 / 1024
    # Regardless of severity, this will be the message
    evt.summary = "Disk space low: %3.1f%% used (%3.2f GB free)" %  (p,freeAmtGB)
    # Set a default for critical @ 95%
    perDeviceThreshold = 95.0
    # This is where we change to a per device threshold
    if perDevicePattern.search(f.id):
        perDeviceThreshold = float(perDevicePattern.search(f.id).groups()[0])
    if p >= perDeviceThreshold: evt.severity = 5
    break


So, is this a hack? Or is there a better way to accomplish this?
Chet Luther ZenossEmployee 1,218 posts since
May 22, 2007
Currently Being Moderated
1. Sep 30, 2008 6:54 PM in response to: ccole
RE: Per filesystem thresholds, is this a hack?
This is a great solution, but it suffers from a subtle but serious problem that will wreak havoc with a distrbuted collector setup. The call to f.usedBytes() within the transform will make an XML-RPC call to the collector responsible for this device to get the last polled usedBytes. This causes a problem with distribtued collector setups because the event transform will be evaluated within the zenhub daemon, which is also responsible for handling this XML-RPC request. So what ends up happening is that zenhub blocks because it is to busy asking for the information to respond with it.

Long-winded explanation aside, there is a better way. You can extract the current usedBlocks from the summary of the message so we don't have to run through this XML-RPC nonsense. Take a look at your transform below that has been altered to use this mechanism instead. Please note that I haven't actually tested this, but it should at least be enough to get you going.

If you don't have distributed collectors you may also want to take this approach because it performs better. The reason being is that it doesn't have to again look into the RRD file for the current value of usedBlocks.


import re

fs_id = device.prepId(evt.component)
for f in device.os.filesystems():
    if f.id != fs_id: continue
    
    # Extract the percent and free from the summary
    m = re.search("threshold of [^:]+: current value ([\d\.]+)", evt.summary)
    if not m: continue
    usedBlocks = float(m.groups()[0])
    p = (usedBlocks / f.totalBlocks) * 100
    freeAmtGB = ((f.totalBlocks - usedBlocks) * f.blockSize) / 1073741824
    
    # Make a nicer summary
    evt.summary = "Disk space low: %3.1f%% used (%3.2f GB free)" %  (p,freeAmtGB)
    
    # This is where we change to a per device threshold
    perDeviceThreshold = 95.0
    m = re.search("zz(\d{3})", f.id)
    perDeviceThreshold = m and float(m.groups()[0]) or 95.0
    if p >= perDeviceThreshold: evt.severity = 5
    break
Matt Ray ZenossEmployee 1,981 posts since
Apr 5, 2008
Currently Being Moderated
2. Oct 22, 2008 6:24 PM in response to: Chet Luther
RE: Per filesystem thresholds, is this a hack?
This thread made the Tip of the Month.
http://blog.zenoss.com/2008/10/22/tip-of-the-month-per-filesystem-thresholds/

Thanks,
Matt Ray
Zenoss Community Manager
akoutlid Rank: White Belt 6 posts since
Jun 2, 2009
Currently Being Moderated
3. Sep 18, 2009 6:20 AM in response to: Matt Ray
Re: RE: Per filesystem thresholds, is this a hack?
The code that is probably the most useful part of this post seems to have been lost in translation - any chance it can be recovered?
Pim Rupert Rank: White Belt 11 posts since
May 30, 2009
Currently Being Moderated
4. Sep 23, 2009 10:52 AM in response to: akoutlid
Re: RE: Per filesystem thresholds, is this a hack?
Nick Yeates ZenossEmployee 30 posts since
Mar 30, 2009
Currently Being Moderated
5. Sep 24, 2009 10:50 AM in response to: Pim Rupert
Re: RE: Per filesystem thresholds, is this a hack?

Actually, the code posted by Chet is likely more what people want.

 

import re 
 
 fs_id = device.prepId(evt.component) 
 for f in device.os.filesystems(): 
     if f.id != fs_id: continue 
      
     # Extract the percent and free from the summary 
     m = re.search("threshold of [^:]+: current value ([\d\.]+)", evt.summary) 
     if not m: continue 
     usedBlocks = float(m.groups()[0]) 
     p = (usedBlocks / f.totalBlocks) * 100 
     freeAmtGB = ((f.totalBlocks - usedBlocks) * f.blockSize) / 1073741824 
      
     # Make a nicer summary 
     evt.summary = "Disk space low: %3.1f%% used (%3.2f GB free)" %  (p,freeAmtGB) 
      
     # This is where we change to a per device threshold 
     perDeviceThreshold = 95.0 
     m = re.search("zz(\d{3})", f.id) 
     perDeviceThreshold = m and float(m.groups()[0]) or 95.0 
     if p >= perDeviceThreshold: evt.severity = 5 
     break
evilk Rank: White Belt 23 posts since
May 8, 2009
Currently Being Moderated
6. Oct 9, 2009 8:49 AM in response to: Nick Yeates
Re: RE: Per filesystem thresholds, is this a hack?

Hi,

 

Where this event transform script is supposed to go? I tried to create EventClass mapping to /Perf/Filesystem (EventClassKey: 'usedBlocks_usedBlocks|high disk usage') but the events don't match the mapping. I think that the mapping is not executed because when filesystem performance template threshold it defines the eventclass /Perf/FileSystem.

jmp242 ZenossMaster 2,295 posts since
Mar 7, 2007
Currently Being Moderated
7. Oct 9, 2009 8:53 AM in response to: evilk
Re: Per filesystem thresholds, is this a hack?

You should put the transform in the existing /Perf/Filesystem event class.

--

James Pulver

Information Technology Area Supervisor

LEPP Computer Group

Cornell University

 

 

 

kwikman wrote, On 10/9/2009 8:49 AM:

Hi,

 

Where this event transform script is supposed to go? I tried to create EventClass mapping to /Perf/Filesystem (EventClassKey:         'usedBlocks_usedBlocks|high disk usage') but the events don't match the mapping. I think that the mapping is not executed because when filesystem performance template threshold it defines the eventclass /Perf/FileSystem.

>

evilk Rank: White Belt 23 posts since
May 8, 2009
Currently Being Moderated
8. Oct 9, 2009 8:59 AM in response to: jmp242
Re: Per filesystem thresholds, is this a hack?

Hi James,


I've done that, but what I should put into the mapping's 'Event Class Key'? Threshold generated events do not have eventClassKey value.

jmp242 ZenossMaster 2,295 posts since
Mar 7, 2007
Currently Being Moderated
9. Oct 9, 2009 9:02 AM in response to: evilk
Re: Per filesystem thresholds, is this a hack?

What eventclasskey? You aren't creating a new mapping. Go to

/Perf/Filesystem, click the dropdown, More, Transform.

--

James Pulver

Information Technology Area Supervisor

LEPP Computer Group

Cornell University

 

 

 

kwikman wrote, On 10/9/2009 8:59 AM:

Hi James,

 

I''ve done that, but what I should put into the mapping's 'Event Class Key'? Threshold generated events do not have eventClassKey value.

>

evilk Rank: White Belt 23 posts since
May 8, 2009
Currently Being Moderated
10. Oct 9, 2009 11:24 AM in response to: jmp242
Re: Per filesystem thresholds, is this a hack?

Hi again,

 

Now I got it working. I did not realize that there was transform selection for all events of the EventClass. Thanks for help.

More Like This

  • Retrieving data ...

Bookmarked By (0)