Skip navigation
10917 Views 11 Replies Latest reply: Jun 8, 2012 2:34 PM by Tyler Fisher RSS
isenberg Rank: White Belt 7 posts since
Jan 28, 2008
Currently Being Moderated

Jan 30, 2009 8:33 AM

Zenoss floods snmpd with too many requests per second

Every few hours a day the snmpd on one Debian GNU/Linux 4.0 32bit server is shown as down via Zenoss. On that Linux server /var/log/daemon.log shows the line "last message repeated 1503 times" at that very same time.

Is it possible to prevent Zenoss to try reconnect to a service without pause after a failure? At least some seconds pause should be used here.

Zenoss 2.3.2 Virtual Appliance.

/var/log/daemon.log:

Jan 30 12:46:40 snmpd[3663]: Connection from 172.17.130.15
Jan 30 12:51:40 snmpd[3663]: Connection from 172.17.130.15
Jan 30 12:55:02 snmpd[3663]: Connection from 172.17.130.15
Jan 30 12:55:07 last message repeated 1503 times
Jan 30 12:56:40 snmpd[3663]: Connection from 172.17.130.15
Jan 30 13:01:40 snmpd[3663]: Connection from 172.17.130.15
  • Wouter DHaeseleer ZenossMaster 204 posts since
    Jun 22, 2007
    I'm seeing the same issue here on some server

    any ideay why that is caused?
  • Matt Ray Rank: Zen Master 2,484 posts since
    Apr 5, 2008
    That sounds pretty heinous. So far I don't think we've seen that, any
    more details? Anyone else having similar issues?

    Thanks,
    Matt Ray
    Zenoss Community Manager
    community.zenoss.com
    mray@zenoss.com



    On Jan 30, 2009, at 7:53 AM, netdata wrote:

     

     

    I'm seeing the same issue here on some server

    any ideay why that is caused?







    _______________________________________________
    zenoss-users mailing list
    zenoss-users@zenoss.org
    http://lists.zenoss.org/mailman/listinfo/zenoss-users


    _______________________________________________
    zenoss-users mailing list
    zenoss-users@zenoss.org
    http://lists.zenoss.org/mailman/listinfo/zenoss-users
  • hbokh Rank: White Belt 16 posts since
    Jan 10, 2009
    Doh! I've created a similar post here...
    http://forums.zenoss.com/viewtopic.php?t=9279

    Should have done a search first, right? Sorry me.

    But yes indeed: same issue here. No more Zenoss until I can explain what's going on... sad
  • nikal Rank: White Belt 45 posts since
    Jul 8, 2008
    Currently Being Moderated
    4. Apr 13, 2009 6:50 AM (in response to hbokh)
    RE: Zenoss floods snmpd with too many requests per second
    Just to confirm that I'm having similar conditions too - more seriosly on some servers only, and not all the time, but pretty often.
    Some random samples:
    
    
    Mar 23 20:49:36 planas snmpd[5380]: Connection from UDP: [10.1.1.11]:33273
    Mar 23 20:49:46 planas last message repeated 1478 times
    
    Apr 13 13:24:10 dc2 snmpd[4743]: Connection from UDP: [10.1.1.11]:-16325
    Apr 13 13:24:11 dc2 last message repeated 133 times
    
    Apr 13 07:52:34 pl1600 snmpd[2618]: Connection from UDP: [10.1.1.18]:56886
    Apr 13 07:53:01 pl1600 last message repeated 1647 times
    
    


    Will be grateful for any advice.
  • Matt Ray Rank: Zen Master 2,484 posts since
    Apr 5, 2008
    Currently Being Moderated
    5. Apr 13, 2009 11:57 AM (in response to nikal)
    Zenoss floods snmpd with too many requests per second
    Could you take a look and see if the fix in this ticket works? http://dev.zenoss.org/trac/ticket/4096
    Better yet, get a tcpdump on the connections. Then it'll be clear
    exactly what Zenoss is asking for.. once we know that it'll be obvious
    what process is generating the requests.

    Thanks,
    Matt Ray
    Zenoss Community Manager
    community.zenoss.com
    mray@zenoss.com




    On Apr 13, 2009, at 5:50 AM, nikal wrote:

     

     

    Just to confirm that I'm having similar conditions too - more
    seriosly on some servers only, and not all the time, but pretty often.
    Some random samples:

    Code:


    Mar 23 20:49:36 planas snmpd[5380]: Connection from UDP: [10.1.1.11]:
    33273
    Mar 23 20:49:46 planas last message repeated 1478 times

    Apr 13 13:24:10 dc2 snmpd[4743]: Connection from UDP:
    [10.1.1.11]:-16325
    Apr 13 13:24:11 dc2 last message repeated 133 times

    Apr 13 07:52:34 pl1600 snmpd[2618]: Connection from UDP: [10.1.1.18]:
    56886
    Apr 13 07:53:01 pl1600 last message repeated 1647 times





    Will be grateful for any advice.







    _______________________________________________
    zenoss-users mailing list
    zenoss-users@zenoss.org
    http://lists.zenoss.org/mailman/listinfo/zenoss-users


    _______________________________________________
    zenoss-users mailing list
    zenoss-users@zenoss.org
    http://lists.zenoss.org/mailman/listinfo/zenoss-users
  • hbokh Rank: White Belt 16 posts since
    Jan 10, 2009
    Well... I started with a complete fresh and clean installation of Zenoss with the recent 2.4 beta "Blue Crab"...
    Only 2 host added so far, with no ZenPacks installed (yet) or extra checks added (e.g. processes).
    FYI on the previous version I had the MIB-pack, fping, Dell OptiPlex, HPMonitor installed. Maybe these matters.

    When I run an snmpwalk-command from within Zenoss to our Nagios-node, this is the output on that node in /var/log/daemon.log. This will trigger the "repeated":

    Apr 13 21:01:22 nagioslive snmpd[29205]: Connection from UDP: [192.168.0.110]:53315 
    Apr 13 21:01:22 nagioslive last message repeated 31 times


    This below however is the normal / new Zenoss-scan to that node and the "repeated" is not showing up now.

    Apr 13 21:04:33 nagioslive snmpd[29205]: Connection from UDP: [192.168.0.110]:40524 
    Apr 13 21:09:33 nagioslive snmpd[29205]: Connection from UDP: [192.168.0.110]:38142


    and it's accompanying tcpdump-output (slightly censored):

    # tcpdump -vv host nagioslive and port snmp
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
    21:04:33.902036 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 527) zenoss.40524 > nagioslive.snmp:  { SNMPv1 C=community { GetRequest(483) R=564372915  interfaces.ifTable.ifEntry.ifInOctets.1 [|snmp] } } 
    21:04:33.902545 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 600) nagioslive.snmp > zenoss.40524:  { SNMPv1 C=community { GetResponse(556) R=564372915  interfaces.ifTable.ifEntry.ifInOctets.1=866524717 [|snmp] } } 
    21:09:33.911916 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 527) zenoss.38142 > nagioslive.snmp:  { SNMPv1 C=community { GetRequest(483) R=564372917  interfaces.ifTable.ifEntry.ifInOctets.1 [|snmp] } } 
    21:09:33.912520 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 600) nagioslive.snmp > zenoss.38142:  { SNMPv1 C=community { GetResponse(556) R=564372917  interfaces.ifTable.ifEntry.ifInOctets.1=866569683 [|snmp] } } 
  • darkis Rank: White Belt 22 posts since
    Nov 12, 2007

    Did anyone get this resolved? I have this issue on two seperate installations.

  • gstanchev Rank: White Belt 8 posts since
    Jan 20, 2011

    same here Zenoss 3.0.1 a lot of repeating messages (aprox 1000)

  • Michel Lombart Rank: White Belt 7 posts since
    Mar 12, 2012

    Sorry, I've just saw this post after having poster mine. Same problem here : http://community.zenoss.org/thread/17554

  • kenhen93 Rank: White Belt 100 posts since
    Sep 27, 2011

    I am having this same issue on a couple instance of zenoss

     

    May 24 09:39:45 <server> snmpd[6881]: Connection from UDP: [64.12.227.196]:44796
        May 24 09:39:45
    <server> snmpd[6881]: Received SNMP packet(s) from UDP: [64.12.227.196]:44796
        May 24 09:39:45
    <server> snmpd[6881]: Connection from UDP: [64.12.227.196]:44796
        May 24 09:39:46
    <server> last message repeated 974 times

     

    I really need a way to fix this. I cannot have production serves under load from snmp traffic when they need those resources for other services.

     

    I actually had one server's /var filesystem fill up because messages file was logging so much snmpd traffic it, it got too big before it could roll over. I ended up having to change the snmpd setting to not log debug messages.

     

    I am not running zenmodeler as a daemon.

     

    Zenoss Core 3.2

  • Tyler Fisher Newbie 1 posts since
    Dec 30, 2011

    Was this ever resolved?  We are seeing the exact same issue with Zenoss Core 3.1.0.  We are not using the zenmodeler as a daemon either. Also, it seems to only occur on our collectors and not the master node. Any direction or help would be much appreciated.

     

    May 28 12:10:45 {server name} snmpd[26793]: Received SNMP packet(s) from UDP: [xxx.xxx.xxx.xxx]:55966

    May 28 12:10:45 {server name} snmpd[26793]: Connection from UDP: [xxx.xxx.xxx.xxx]:55966

    May 28 12:10:46 {server name} last message repeated 782 times

More Like This

  • Retrieving data ...