Skip navigation
3008 Views 3 Replies Latest reply: Apr 30, 2012 9:32 AM by Rob Coward RSS
tornado Rank: White Belt 28 posts since
Jan 20, 2010
Currently Being Moderated

Jun 17, 2011 11:56 AM

ZenPing on remote collector checks the devices he shouldn't

I use Zenoss Core 3.1.0 on CentOS 5.6 64bit

I've configured a remote collector using this ZenPack

It works pretty well, but I've faced an issue I can't solve.

 

Athough remote collector shall monitor just 4 devices, ZenPing checks 6.

 

>>> collector = dmd.Monitors.Performance._getOb('collector.company.com')
>>> len(collector.getPingDevices())
4
>>>

 

[zenoss@collector ~]$ zenping run
2011-06-17 10:20:42,091 INFO zen.ZenPing: started
2011-06-17 10:20:42,093 INFO zen.ZenPing: Connecting to master.company.com:8789
2011-06-17 10:20:42,379 INFO zen.ZenPing: Connected to ZenHub
2011-06-17 10:20:42,450 INFO zen.ZenPing: _configCycle fetching configuration
2011-06-17 10:20:42,450 INFO zen.ZenPing: fetching monitor properties
2011-06-17 10:20:42,531 INFO zen.ZenPing: completed "fetching monitoring properties" request in 0.08 seconds
2011-06-17 10:20:42,532 INFO zen.ZenPing: getting threshold classes
2011-06-17 10:20:42,601 INFO zen.ZenPing: completed "getting threshold classes" request in 0.07 seconds
2011-06-17 10:20:42,602 INFO zen.ZenPing: getting collector thresholds
2011-06-17 10:20:42,755 INFO zen.ZenPing: completed "getting collector thresholds" request in 0.15 seconds
2011-06-17 10:20:42,755 INFO zen.ZenPing: fetching default RRDCreateCommand
2011-06-17 10:20:42,824 INFO zen.ZenPing: getting ping tree
2011-06-17 10:20:43,443 INFO zen.ZenPing: completed "getting ping tree" request in 0.62 seconds
2011-06-17 10:20:43,481 INFO zen.ZenPing: Rescheduling configuration check in 21600 seconds
2011-06-17 10:20:46,482 WARNING zen.ZenPing: ip 123.123.123.123 is down
2011-06-17 10:20:46,483 INFO zen.ZenPing: Finished pinging 6 jobs in 3.00 seconds
2011-06-17 10:20:46,483 INFO zen.ZenPing: Daemon ZenPing shutting down

 

The problerm is, that IP 123.123.123.123 is not accessbile from the collector.

 

I've also checked several times, that device is monitord by the master.

 

>>> device=find('123.123.123.123')
>>> device.getPerformanceServerName()
'localhost'

 

Both master and collectors were restarted several times. reindex() commit() also was executed.

 

Any ideas what should I check and how to remove the device from the ZenPing list on the collector?

  • Rob Coward Newbie 2 posts since
    Apr 26, 2012

    Hi,

    Has anyone found the cause / solution to this problem ? I have a setup with 2 remote collectors and am experiencing the same problem where the remote collectors are trying to ping devices they dont have access to and subsequently marking them as down.

     

    Thanks,

    Rob

  • dpetzel Rank: Brown Belt 1,141 posts since
    Oct 17, 2010

    Can't say I've seen this one before. Rob, are you on 3.1.0 as well? My first reaction is that the devices are not properly assigned to the right collector. The original posters comments don't tell me much as they were manually running zenping, I'm not sure that proves its the collector doing the ping.

     

    If you have recently re-assigned the device from one collector to another, I wonder if "Pushing Configs" from the gear menu, or restarting zenoss on the collectors will get things back in sync.

     

    A little more information my yield some answers.

  • Rob Coward Newbie 2 posts since
    Apr 26, 2012

    Both the main zenoss install and the remote collectors are running the zenoss-3.2.1-1326.el5 rpm installation. We have a distributed network with an office network and WAN links to multiple datacenters. The main zenoss install is in the office, and I am trying to run remote collectors in each data center. The collector that is pinging the wrong devices is a replacement collector, and the addresses it is trying to ping are not even local to that data center... So far, it only has a handful of devices it should be collecting:

     

    >>> config = dmd.Monitors.Performance._getOb('zenoss1.redstation.eseye.net')

    >>> config.getPingDevices()

    [<Device at /zport/dmd/Devices/Server/Linux/Zenoss/devices/email1.redstation.eseye.net>, <Device at /zport/dmd/Devices/Server/Linux/devices/activemq01.eseye.net>, <Device at /zport/dmd/Devices/Server/SSH/Linux/devices/sw2.eseye.net>, <Device at /zport/dmd/Devices/Server/Linux/devices/activemq02.eseye.net>, <Device at /zport/dmd/Devices/Server/Linux/devices/batch01.eseye.net>]

    >>>

     

    However, when starting zenping, it seems to be pulling in a much larger list of devices to ping, many of which will never be accessible to the collector:

     

    [zenoss@zenoss1 ~]$ zenping run -v10

    2012-04-30 14:27:09,971 INFO zen.ZenPing: started

    2012-04-30 14:27:09,976 DEBUG zen.ZenPing: Starting PBDaemon initialization

    2012-04-30 14:27:09,976 INFO zen.ZenPing: Connecting to zenoss-01.eseye.net:8789

    2012-04-30 14:27:09,977 DEBUG zen.ZenPing: Logging in as admin

    2012-04-30 14:27:10,032 INFO zen.ZenPing: Connected to ZenHub

    2012-04-30 14:27:10,032 DEBUG zen.ZenPing: Setting up initial services: EventService, PingConfig

    2012-04-30 14:27:10,033 DEBUG zen.ZenPing: Chaining getInitialServices with d2

    2012-04-30 14:27:10,047 DEBUG zen.ZenPing: Loaded service EventService from zenhub

    2012-04-30 14:27:10,047 DEBUG zen.ZenPing: Loaded service PingConfig from zenhub

    2012-04-30 14:27:10,047 DEBUG zen.ZenPing: Queueing event {'severity': 0, 'component': 'zenping', 'agent': 'zenping', 'summary': 'started', 'manager': 'zenoss1.redstation.eseye.net', 'device': 'zenoss1.redstation.eseye.net', 'eventClass': '/App/Start', 'monitor': 'zenoss1.redstation.eseye.net'}

    2012-04-30 14:27:10,047 DEBUG zen.ZenPing: Total of 1 queued events

    2012-04-30 14:27:10,048 DEBUG zen.ZenPing: Calling connected.

    2012-04-30 14:27:10,048 DEBUG zen.ZenPing: Connected, getting config

    2012-04-30 14:27:10,048 INFO zen.ZenPing: _configCycle fetching configuration

    2012-04-30 14:27:10,048 INFO zen.ZenPing: fetching monitor properties

    2012-04-30 14:27:10,073 INFO zen.ZenPing: completed "fetching monitoring properties" request in 0.02 seconds

    2012-04-30 14:27:10,074 INFO zen.ZenPing: getting threshold classes

    2012-04-30 14:27:10,087 INFO zen.ZenPing: completed "getting threshold classes" request in 0.01 seconds

    2012-04-30 14:27:10,087 DEBUG zen.ZenPing: Loading classes ['Products.ZenModel.MinMaxThreshold', 'ZenPacks.community.deviceAdvDetail.thresholds.StatusThreshold']

    2012-04-30 14:27:10,089 INFO zen.ZenPing: getting collector thresholds

    2012-04-30 14:27:10,140 INFO zen.ZenPing: completed "getting collector thresholds" request in 0.05 seconds

    2012-04-30 14:27:10,140 INFO zen.ZenPing: fetching default RRDCreateCommand

    2012-04-30 14:27:10,153 DEBUG zen.thresholds: Updating threshold ('high event queue', ('zenoss1.redstation.eseye.net', ''))

    2012-04-30 14:27:10,154 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('zenoss1.redstation.eseye.net', ''))

    2012-04-30 14:27:10,154 DEBUG zen.thresholds: Updating threshold ('zenperfsnmp cycle time', ('zenoss1.redstation.eseye.net', ''))

    2012-04-30 14:27:10,154 DEBUG zen.thresholds: Updating threshold ('zenping cycle time', ('zenoss1.redstation.eseye.net', ''))

    2012-04-30 14:27:10,154 DEBUG zen.thresholds: Updating threshold ('zenprocess cycle time', ('zenoss1.redstation.eseye.net', ''))

    2012-04-30 14:27:10,154 INFO zen.ZenPing: getting ping tree

    2012-04-30 14:27:10,461 INFO zen.ZenPing: completed "getting ping tree" request in 0.31 seconds

    2012-04-30 14:27:10,462 INFO zen.ZenPing: Rescheduling configuration check in 21600 seconds

    2012-04-30 14:27:10,463 DEBUG zen.ZenPing: starting 192.168.232.18

    2012-04-30 14:27:10,463 DEBUG zen.Ping: send icmp to '192.168.232.18'

    2012-04-30 14:27:10,463 DEBUG zen.ZenPing: starting 192.168.112.3

    2012-04-30 14:27:10,463 DEBUG zen.Ping: send icmp to '192.168.112.3'

    2012-04-30 14:27:10,464 DEBUG zen.ZenPing: starting 192.168.226.1

    2012-04-30 14:27:10,464 DEBUG zen.Ping: send icmp to '192.168.226.1'

    2012-04-30 14:27:10,464 DEBUG zen.ZenPing: starting 192.168.111.253

    2012-04-30 14:27:10,464 DEBUG zen.Ping: send icmp to '192.168.111.253'

    2012-04-30 14:27:10,465 DEBUG zen.ZenPing: starting 192.168.112.4

    2012-04-30 14:27:10,465 DEBUG zen.Ping: send icmp to '192.168.112.4'

    2012-04-30 14:27:10,465 DEBUG zen.ZenPing: starting 192.168.111.10

    2012-04-30 14:27:10,465 DEBUG zen.Ping: send icmp to '192.168.111.10'

    2012-04-30 14:27:10,465 DEBUG zen.ZenPing: starting 192.168.226.3

    2012-04-30 14:27:10,466 DEBUG zen.Ping: send icmp to '192.168.226.3'

    2012-04-30 14:27:10,466 DEBUG zen.ZenPing: starting 192.168.111.9

    2012-04-30 14:27:10,466 DEBUG zen.Ping: send icmp to '192.168.111.9'

    2012-04-30 14:27:10,466 DEBUG zen.ZenPing: starting 192.168.107.1

    2012-04-30 14:27:10,467 DEBUG zen.Ping: send icmp to '192.168.107.1'

    2012-04-30 14:27:10,467 DEBUG zen.ZenPing: starting 169.254.1.2

    2012-04-30 14:27:10,467 DEBUG zen.Ping: send icmp to '169.254.1.2'

    2012-04-30 14:27:10,467 DEBUG zen.ZenPing: starting 192.168.111.254

    2012-04-30 14:27:10,467 DEBUG zen.Ping: send icmp to '192.168.111.254'

    2012-04-30 14:27:10,468 DEBUG zen.ZenPing: starting 192.168.254.24

    2012-04-30 14:27:10,468 DEBUG zen.Ping: send icmp to '192.168.254.24'

    2012-04-30 14:27:10,468 DEBUG zen.ZenPing: starting 169.254.1.3

    2012-04-30 14:27:10,469 DEBUG zen.Ping: send icmp to '169.254.1.3'

    2012-04-30 14:27:10,469 DEBUG zen.ZenPing: starting 192.168.111.3

    2012-04-30 14:27:10,469 DEBUG zen.Ping: send icmp to '192.168.111.3'

    2012-04-30 14:27:10,469 DEBUG zen.ZenPing: starting 192.168.233.11

    2012-04-30 14:27:10,469 DEBUG zen.Ping: send icmp to '192.168.233.11'

    2012-04-30 14:27:10,470 DEBUG zen.ZenPing: starting 192.168.233.12

    2012-04-30 14:27:10,470 DEBUG zen.Ping: send icmp to '192.168.233.12'

    2012-04-30 14:27:10,470 DEBUG zen.ZenPing: starting 192.168.232.6

    2012-04-30 14:27:10,470 DEBUG zen.Ping: send icmp to '192.168.232.6'

    2012-04-30 14:27:10,471 DEBUG zen.Ping: unexpected pkt 192.168.232.18 <ICMP packet 8 0>

    2012-04-30 14:27:10,472 DEBUG zen.Ping: echo reply pkt 192.168.232.18 <ICMP packet 0 0>

    2012-04-30 14:27:10,472 DEBUG zen.Ping: pj succeed for 192.168.232.18

    2012-04-30 14:27:10,472 DEBUG zen.ZenPing: Success 192.168.232.18

    2012-04-30 14:27:10,472 DEBUG zen.Ping: echo reply pkt 192.168.112.3 <ICMP packet 0 0>

    2012-04-30 14:27:10,472 DEBUG zen.Ping: pj succeed for 192.168.112.3

    2012-04-30 14:27:10,472 DEBUG zen.ZenPing: Success 192.168.112.3

    2012-04-30 14:27:10,473 DEBUG zen.Ping: echo reply pkt 192.168.111.253 <ICMP packet 0 0>

    2012-04-30 14:27:10,473 DEBUG zen.Ping: pj succeed for 192.168.111.253

    2012-04-30 14:27:10,473 DEBUG zen.ZenPing: Success 192.168.111.253

    2012-04-30 14:27:10,473 DEBUG zen.Ping: echo reply pkt 192.168.111.10 <ICMP packet 0 0>

    2012-04-30 14:27:10,473 DEBUG zen.Ping: pj succeed for 192.168.111.10

    2012-04-30 14:27:10,473 DEBUG zen.ZenPing: Success 192.168.111.10

    2012-04-30 14:27:10,474 DEBUG zen.Ping: pj fail for 192.168.107.1

    2012-04-30 14:27:10,474 DEBUG zen.ZenPing: Failed 192.168.107.1 166

    2012-04-30 14:27:10,474 WARNING zen.ZenPing: ip 192.168.107.1 is down

    2012-04-30 14:27:10,474 DEBUG zen.ZenPing: Queueing event {'severity': 5, 'component': '', 'agent': 'zenping', 'summary': 'ip 192.168.107.1 is down', 'manager': 'zenoss1.redstation.eseye.net', 'eventGroup': 'Ping', 'eventState': 0, 'device': '192.168.107.1', 'eventClass': '/Status/Ping', 'ipAddress': '192.168.107.1', 'monitor': 'zenoss1.redstation.eseye.net'}

    2012-04-30 14:27:10,474 DEBUG zen.ZenPing: Total of 1 queued events

    2012-04-30 14:27:10,475 DEBUG zen.Ping: echo reply pkt 192.168.226.3 <ICMP packet 0 0>

    2012-04-30 14:27:10,475 DEBUG zen.Ping: pj succeed for 192.168.226.3

    2012-04-30 14:27:10,475 DEBUG zen.ZenPing: Success 192.168.226.3

    2012-04-30 14:27:10,475 DEBUG zen.Ping: echo reply pkt 192.168.111.9 <ICMP packet 0 0>

    2012-04-30 14:27:10,475 DEBUG zen.Ping: pj succeed for 192.168.111.9

    2012-04-30 14:27:10,475 DEBUG zen.ZenPing: Success 192.168.111.9

    2012-04-30 14:27:10,476 DEBUG zen.Ping: pj fail for 192.168.254.24

    2012-04-30 14:27:10,476 DEBUG zen.ZenPing: Failed 192.168.254.24 70

    2012-04-30 14:27:10,476 WARNING zen.ZenPing: ip 192.168.254.24 is down

    2012-04-30 14:27:10,476 DEBUG zen.ZenPing: Queueing event {'severity': 5, 'component': '', 'agent': 'zenping', 'summary': 'ip 192.168.254.24 is down', 'manager': 'zenoss1.redstation.eseye.net', 'eventGroup': 'Ping', 'eventState': 0, 'device': 'lvs2.manx.eseye.net', 'eventClass': '/Status/Ping', 'ipAddress': '192.168.254.24', 'monitor': 'zenoss1.redstation.eseye.net'}

    2012-04-30 14:27:10,476 DEBUG zen.ZenPing: Total of 2 queued events

    2012-04-30 14:27:10,477 DEBUG zen.Ping: echo reply pkt 192.168.111.254 <ICMP packet 0 0>

    2012-04-30 14:27:10,477 DEBUG zen.Ping: pj succeed for 192.168.111.254

    2012-04-30 14:27:10,477 DEBUG zen.ZenPing: Success 192.168.111.254

    2012-04-30 14:27:10,477 DEBUG zen.Ping: echo reply pkt 192.168.112.4 <ICMP packet 0 0>

    2012-04-30 14:27:10,477 DEBUG zen.Ping: pj succeed for 192.168.112.4

    2012-04-30 14:27:10,477 DEBUG zen.ZenPing: Success 192.168.112.4

    2012-04-30 14:27:10,478 DEBUG zen.Ping: echo reply pkt 192.168.233.11 <ICMP packet 0 0>

    2012-04-30 14:27:10,478 DEBUG zen.Ping: pj succeed for 192.168.233.11

    2012-04-30 14:27:10,478 DEBUG zen.ZenPing: Success 192.168.233.11

    2012-04-30 14:27:10,478 DEBUG zen.Ping: echo reply pkt 192.168.226.1 <ICMP packet 0 0>

    2012-04-30 14:27:10,478 DEBUG zen.Ping: pj succeed for 192.168.226.1

    2012-04-30 14:27:10,478 DEBUG zen.ZenPing: Success 192.168.226.1

    2012-04-30 14:27:10,479 DEBUG zen.Ping: echo reply pkt 192.168.232.6 <ICMP packet 0 0>

    2012-04-30 14:27:10,479 DEBUG zen.Ping: pj succeed for 192.168.232.6

    2012-04-30 14:27:10,479 DEBUG zen.ZenPing: Success 192.168.232.6

    2012-04-30 14:27:10,479 DEBUG zen.Ping: echo reply pkt 192.168.233.12 <ICMP packet 0 0>

    2012-04-30 14:27:10,479 DEBUG zen.Ping: pj succeed for 192.168.233.12

    2012-04-30 14:27:10,479 DEBUG zen.ZenPing: Success 192.168.233.12

    2012-04-30 14:27:10,480 DEBUG zen.Ping: echo reply pkt 192.168.111.3 <ICMP packet 0 0>

    2012-04-30 14:27:10,480 DEBUG zen.Ping: pj succeed for 192.168.111.3

    2012-04-30 14:27:10,480 DEBUG zen.ZenPing: Success 192.168.111.3

    2012-04-30 14:27:11,968 DEBUG zen.Ping: send icmp to '169.254.1.2'

    2012-04-30 14:27:11,970 DEBUG zen.Ping: send icmp to '169.254.1.3'

    2012-04-30 14:27:13,469 DEBUG zen.Ping: pj timeout for 169.254.1.2

    2012-04-30 14:27:13,469 DEBUG zen.Ping: pj fail for 169.254.1.2

    2012-04-30 14:27:13,469 DEBUG zen.ZenPing: Failed 169.254.1.2 71

    2012-04-30 14:27:13,469 WARNING zen.ZenPing: ip 169.254.1.2 is down, failed at 192.168.107.1

    2012-04-30 14:27:13,469 DEBUG zen.ZenPing: Queueing event {'severity': 5, 'component': '', 'agent': 'zenping', 'summary': 'ip 169.254.1.2 is down, failed at 192.168.107.1', 'manager': 'zenoss1.redstation.eseye.net', 'eventGroup': 'Ping', 'eventState': 2, 'device': 'mnxrtr02.manx.eseye.net', 'eventClass': '/Status/Ping', 'ipAddress': '169.254.1.2', 'monitor': 'zenoss1.redstation.eseye.net'}

    2012-04-30 14:27:13,469 DEBUG zen.ZenPing: Total of 3 queued events

    2012-04-30 14:27:13,470 DEBUG zen.Ping: pj timeout for 169.254.1.3

    2012-04-30 14:27:13,471 DEBUG zen.Ping: pj fail for 169.254.1.3

    2012-04-30 14:27:13,471 DEBUG zen.ZenPing: Failed 169.254.1.3 70

    2012-04-30 14:27:13,471 WARNING zen.ZenPing: ip 169.254.1.3 is down, failed at lvs2.manx.eseye.net

    2012-04-30 14:27:13,471 DEBUG zen.ZenPing: Queueing event {'severity': 5, 'component': '', 'agent': 'zenping', 'summary': 'ip 169.254.1.3 is down, failed at lvs2.manx.eseye.net', 'manager': 'zenoss1.redstation.eseye.net', 'eventGroup': 'Ping', 'eventState': 2, 'device': 'mnxrtr03.manx.eseye.net', 'eventClass': '/Status/Ping', 'ipAddress': '169.254.1.3', 'monitor': 'zenoss1.redstation.eseye.net'}

    2012-04-30 14:27:13,471 DEBUG zen.ZenPing: Total of 4 queued events

    2012-04-30 14:27:13,471 INFO zen.ZenPing: Finished pinging 17 jobs in 3.01 seconds

    2012-04-30 14:27:13,472 DEBUG zen.ZenPing: Sent a 'stop' event

    2012-04-30 14:27:13,473 INFO zen.ZenPing: Daemon ZenPing shutting down

    2012-04-30 14:27:13,473 DEBUG zen.ZenPing: Removing service EventService

    2012-04-30 14:27:13,473 DEBUG zen.ZenPing: Removing service PingConfig

     

    Any thoughts ?

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points