Skip navigation
1 2 3 Previous Next 60026 Views 31 Replies Latest reply: Nov 6, 2013 11:59 AM by John Komara RSS
rhester Newbie 1 posts since
Mar 5, 2007
Currently Being Moderated

Mar 5, 2007 10:39 AM

Linux memory thresholds

I'm sure someone else has had and solved this problem, but short of writing a plugin to handle it, I'm at a bit of a loss.

Linux "free memory" is actually the sum of three components in SNMP and Zenoss - real memory, cached memory, and buffers.

When setting up a memory threshold in Zenoss for a Linux server, you want the threshold to be against the sum of those three, not just real memory (because a very active server could have virtually no "real memory" free because it's all tied up in gigs of cache and buffers, which is a completely normal situation and that memory will be reclaimed by applications as needed).

Unfortunately, Zenoss only allows thresholding against a single datapoint, making it essentially useless for thresholding memory usage on Linux servers (Sun, for instance, doesn't have this problem as it reports free memory in one "chunk").

Has anyone else already solved this problem? It seems a shame to even consider writing a plugin to handle this, having to deploy it to each server, and wasting the additional CPU cycles to return data that Zenoss already has.

Rodney
  • ecn Rank: Green Belt 306 posts since
    Feb 7, 2007
    Currently Being Moderated
    1. Mar 12, 2007 10:50 AM (in response to rhester)
    RE: Linux memory thresholds
    Known limitation:

    http://dev.zenoss.org/trac/ticket/718
  • laguest Rank: White Belt 39 posts since
    Oct 1, 2007
    Currently Being Moderated
    2. Oct 1, 2007 7:30 PM (in response to ecn)
    RE: Linux memory thresholds
    Has anyone got this to work yet?
    If so could you pass along the info.

    Thanks
  • vincem Rank: White Belt 20 posts since
    Jan 14, 2009
    Currently Being Moderated
    3. Mar 13, 2009 2:15 PM (in response to laguest)
    RE: Linux memory thresholds
    Anyone find a way to resolve this ?
  • HummerBoy Rank: Green Belt 116 posts since
    Dec 17, 2008
    Currently Being Moderated
    4. Jun 9, 2009 11:17 PM (in response to vincem)
    RE: Linux memory thresholds
    I want a fix for this too pretty please.
  • amitk Rank: White Belt 26 posts since
    Apr 13, 2009
    Currently Being Moderated
    5. Nov 23, 2009 10:41 AM (in response to HummerBoy)
    Re: RE: Linux memory thresholds
    Is this working for anyone?
  • dfischer Rank: White Belt 6 posts since
    Jun 28, 2008
    Currently Being Moderated
    6. Nov 23, 2009 12:38 PM (in response to rhester)
    Re: Linux memory thresholds

    We gave up on this and no longer monitor any memory on linux boxes.  Instead we monitor the state of SWAP.  As long as that isn't being used much then we do not care at all about the memory usage.

  • medievalhellspawn999 Newbie 1 posts since
    Jan 14, 2010
    Currently Being Moderated
    7. Dec 20, 2010 4:50 PM (in response to dfischer)
    Re: Linux memory thresholds

    I'm about to give up on Zenoss all together because of this annoying bug.  Come on now.  If you are always going to report red on all my linux boxes, then what is the point of monitoring?  And adding the ignore is not the answer for we rely on the memory monitoring for leaks.

  • amitk Rank: White Belt 26 posts since
    Apr 13, 2009
    Currently Being Moderated
    8. Dec 20, 2010 8:44 PM (in response to medievalhellspawn999)
    Re: Linux memory thresholds

    This is working perfect for me. I've used this in "/Events/Perf/Snmp" transform.

    import re

    m = re.search("threshold of [^:]+: current value ([\d\.]+)", evt.message)

     

    if m and device:

                            fbc = device.hw.memAvailReal + device.hw.memBuffer + device.hw.memCached

                            total = device.hw.totalMemory

                            tot_25pc = (total * 25)/100

                            if fbc < tot_25pc:

                                           evt._action = "drop"

  • James Stewart Rank: Green Belt 91 posts since
    Dec 1, 2010
    Currently Being Moderated
    9. Aug 18, 2011 12:30 AM (in response to amitk)
    Re: Linux memory thresholds

    How can this work? As far as I can see the various memory data points gathered via SNMP are not accessible via zendmd, and thus not accessible from within an event transform.

     

    For example, trying to access device.hw.memAvailReal in an event transform results in...

     

    Problem on line 5: AttributeError: memAvailReal

     

    the same for any given linux server in zendmd:

     

    >>> device.hw.memAvailReal

    Traceback (most recent call last):

       File "<console>", line 1, in <module>

    AttributeError: memAvailReal

     

    Are you sure this is working? Is there something I'm missing?

  • ebogaard Rank: White Belt 41 posts since
    Jun 9, 2011
    Currently Being Moderated
    10. Mar 23, 2012 1:21 PM (in response to rhester)
    Re: Linux memory thresholds

    Sorry for opening this thread again, but to me it seems this problem still isn't solved: Zenoss cannot monitor the real memory usage of a Linux installation. This is really annoying.

     

    Even though it is quite easy to with a formula like this (assuming 90% memory usage is your threshold):

    memAvailReal + memBuffer + memCached > memTotalReal * 0,1

     

    The problem here is the left side of this fomula: is there any way to combine several datapoints in an alias or something like that?

  • James Stewart Rank: Green Belt 91 posts since
    Dec 1, 2010
    Currently Being Moderated
    11. Mar 26, 2012 5:28 PM (in response to ebogaard)
    Re: Linux memory thresholds

    ebogaard...

     

    I looked into this fairly extensively and from what I can gather, there is no way for the zenoss snmp performance engine to perform operations across values obtained by polling multiple OIDs. As such, there's no out-of-the box way to perform the required equation.

     

    In the end, I wrote a python script that collects the various values via snmp and derives the actual memory usage. I use a command data source to run this against my linux servers.

     

    If it would be useful I can create a zenpack that pulls together my script, the command data source and the graph output?

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    12. Mar 26, 2012 11:47 PM (in response to ebogaard)
    Re: Linux memory thresholds

    ebogaard:

     

    You can do this one of three ways.

     

    1) Ryan Mattes comibined datasource zenpack. Don't use this in medium or larger scenarios.

    2) Write a command datasource that fetches the values and adds them as needed

    3) Write a script that gets the added values and puts them in a custom oid. Poll this oid.

     

    Best,

    --Shane (Hackman238)

  • ebogaard Rank: White Belt 41 posts since
    Jun 9, 2011
    Currently Being Moderated
    13. Mar 27, 2012 2:45 AM (in response to James Stewart)
    Re: Linux memory thresholds

    James, Shane, thanks for you suggestions!

     

    James, if you have a script in place and can make a Zenpack from it, that would be great.

    Also checked out the 'Forumla Data Source' from Ryan Mattes that Shane suggested and that looks like a good suggestion as well, although this might not work on 3.2.1 (which I use) and be kinda hard on the Zenoss server.

     

    I'll try both.

  • linuxdynasty ZenossMaster 156 posts since
    Apr 10, 2007
    Currently Being Moderated
    14. Mar 27, 2012 1:19 PM (in response to ebogaard)
    Re: Linux memory thresholds

    You can use my script, for monitoring Memory or swap percentage used...

    Example how to use the script below...

    zenoss@localhost:/usr/local/zenoss/zenoss/libexec$ python getMemPercentage.py -d server1

    OK |swapPercentageAvail=96 swapPercentageUsed=3 memPercentageAvail=30 memPercentageUsed=69

     

     

    https://github.com/linuxdynasty/Linuxdynasty/blob/de98e1b4d0946bd7c569d1f6952aaa926f4df032/asanabria/Scripts/python/getMemPercentage.py

1 2 3 Previous Next

More Like This

  • Retrieving data ...