Skip navigation
2486 Views 8 Replies Latest reply: Jul 25, 2012 3:11 PM by Ryan Matte RSS
Voorhees Rank: White Belt 24 posts since
Apr 26, 2012
Currently Being Moderated

Jul 24, 2012 2:23 PM

Windows SNMP Service Monitor - Error and Performance Issues

Hey Folks / Ryan - I installed the windows SNMP service monitor today and ran into an issue during the reindexing at the end of the zenpack install. Since then its been a fight with hi CPU utilization in Zenoss and failing heartbeats. It also caused most of my windows device names to no longer appear. Name was now showing up as the IP address and reports were broken.

 

Software Component Versions 

 

 

Zenoss Zenoss 3.2.1
OS Linux (x86_64) 2.6.32 (Linux zenoss 2.6.32-279.2.1.el6.x86_64 #1 SMP Fri Jul 20 01:55:29 UTC 2012 x86_64)
Zope Zope 2.12.1
Python Python 2.6.2
Database MySQL 5.0.45 (Ver 5.0.45)
RRD RRDtool 1.3.9
Twisted Twisted 8.1.0
NetSnmp NetSnmp 5.4.1
PyNetSnmp PyNetSnmp 0.29.13
WMI Wmi 1.3.13

 

Here's my exact steps:

 

wget http://community.zenoss.org/servlet/JiveServlet/download/12999-21-6444/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zip

 

         unzip ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zip

 

         zenpack --install unzip ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg

 

[zenoss@zenoss ~]$ wget http://community.zenoss.org/servlet/JiveServlet/download/12999-21-6444/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zip
--2012-07-24 10:15:12--  http://community.zenoss.org/servlet/JiveServlet/download/12999-21-6444/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zip
Resolving community.zenoss.org... 204.93.71.120
Connecting to community.zenoss.org|204.93.71.120|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 64228 (63K) [application/zip]
Saving to: âZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zipâ

 

100%[===================================================================================================================>] 64,228       211K/s   in 0.3s

 

2012-07-24 10:15:13 (211 KB/s) - âZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zipâ

 

[zenoss@zenoss ~]$ unzip ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zip
Archive:  ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg.zip
inflating: ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg

 

[zenoss@zenoss ~]$ zenpack --install ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg
2012-07-24 10:16:08,093 INFO zen.ZPLoader: Loading /local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/objects/objects.xml
2012-07-24 10:16:09,315 INFO zen.AddToPack: End loading objects
2012-07-24 10:16:09,315 INFO zen.AddToPack: Processing links
2012-07-24 10:16:09,639 INFO zen.AddToPack: Loaded 5 objects into the ZODB database
2012-07-24 10:16:09,662 INFO zen.HookReportLoader: loading reports from:/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/reports
2012-07-24 10:16:10,021 INFO zen.ZenWinSrvSNMP: Installing menus
2012-07-24 10:16:10,646 INFO zen.ZenWinSrvSNMP: Linking daemon into $ZENHOME/Products/ZenRRD
2012-07-24 10:16:10,705 INFO zen.ZenWinSrvSNMP: Linking daemon listener into $ZENHOME/Products/ZenHub/services
2012-07-24 10:16:10,745 INFO zen.ZenWinSrvSNMP: Making daemon executable
2012-07-24 10:16:10,758 INFO zen.ZenWinSrvSNMP: Setting zPythonClass in /Server/Windows to 'ZenPacks.Nova.WinServiceSNMP.WinServiceSNMPDevice'
2012-07-24 10:16:10,759 INFO zen.ZenWinSrvSNMP: Adding collector plugin 'community.snmp.WinServiceMap' to /Server/Windows
2012-07-24 10:16:10,759 INFO zen.ZenWinSrvSNMP: Starting ZenWinSrvSNMP daemon
starting...
2012-07-24 10:16:15,255 INFO zen.ZenWinSrvSNMP: Rebuilding device relations
2012-07-24 10:16:32,504 INFO zen.ImportRM: End loading objects
2012-07-24 10:16:32,504 INFO zen.ImportRM: Processing links
2012-07-24 10:16:33,192 INFO zen.ImportRM: Would have created 39 objects in the ZODB database
2012-07-24 10:16:42,210 INFO zen.ImportRM: End loading objects
2012-07-24 10:16:42,210 INFO zen.ImportRM: Processing links
2012-07-24 10:16:42,529 INFO zen.ImportRM: Would have created 52 objects in the ZODB database
2012-07-24 10:16:52,976 INFO zen.ImportRM: End loading objects
2012-07-24 10:16:52,976 INFO zen.ImportRM: Processing links
2012-07-24 10:16:53,181 INFO zen.ImportRM: Would have created 67 objects in the ZODB database
2012-07-24 10:17:03,478 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:03,478 INFO zen.ImportRM: Processing links
2012-07-24 10:17:03,591 INFO zen.ImportRM: Would have created 66 objects in the ZODB database
2012-07-24 10:17:11,208 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:11,208 INFO zen.ImportRM: Processing links
2012-07-24 10:17:11,559 INFO zen.ImportRM: Would have created 47 objects in the ZODB database
2012-07-24 10:17:18,916 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:18,916 INFO zen.ImportRM: Processing links
2012-07-24 10:17:18,984 INFO zen.ImportRM: Would have created 43 objects in the ZODB database
2012-07-24 10:17:25,780 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:25,780 INFO zen.ImportRM: Processing links
2012-07-24 10:17:26,061 INFO zen.ImportRM: Would have created 46 objects in the ZODB database
2012-07-24 10:17:32,802 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:32,803 INFO zen.ImportRM: Processing links
2012-07-24 10:17:32,896 INFO zen.ImportRM: Would have created 46 objects in the ZODB database
2012-07-24 10:17:39,855 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:39,855 INFO zen.ImportRM: Processing links
2012-07-24 10:17:39,937 INFO zen.ImportRM: Would have created 46 objects in the ZODB database
2012-07-24 10:17:50,271 INFO zen.ImportRM: End loading objects
2012-07-24 10:17:50,272 INFO zen.ImportRM: Processing links
2012-07-24 10:17:50,528 INFO zen.ImportRM: Would have created 67 objects in the ZODB database
2012-07-24 10:18:20,075 INFO zen.ImportRM: End loading objects
2012-07-24 10:18:20,075 INFO zen.ImportRM: Processing links
2012-07-24 10:18:20,570 INFO zen.ImportRM: Would have created 69 objects in the ZODB database
2012-07-24 10:18:37,945 INFO zen.ImportRM: End loading objects
2012-07-24 10:18:37,945 INFO zen.ImportRM: Processing links
2012-07-24 10:18:38,258 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 10:18:56,611 INFO zen.ImportRM: End loading objects
2012-07-24 10:18:56,611 INFO zen.ImportRM: Processing links
2012-07-24 10:18:57,405 INFO zen.ImportRM: Would have created 62 objects in the ZODB database
2012-07-24 10:19:19,420 INFO zen.ImportRM: End loading objects
2012-07-24 10:19:19,420 INFO zen.ImportRM: Processing links
2012-07-24 10:19:19,641 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 10:19:35,587 INFO zen.ImportRM: End loading objects
2012-07-24 10:19:35,587 INFO zen.ImportRM: Processing links
2012-07-24 10:19:35,751 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 10:19:53,170 INFO zen.ImportRM: End loading objects
2012-07-24 10:19:53,170 INFO zen.ImportRM: Processing links
2012-07-24 10:19:53,390 INFO zen.ImportRM: Would have created 59 objects in the ZODB database
2012-07-24 10:20:10,183 INFO zen.ImportRM: End loading objects
2012-07-24 10:20:10,183 INFO zen.ImportRM: Processing links
2012-07-24 10:20:10,452 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 10:20:27,069 INFO zen.ImportRM: End loading objects
2012-07-24 10:20:27,069 INFO zen.ImportRM: Processing links
2012-07-24 10:20:27,298 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 10:20:35,152 ERROR zen.ZenPackCmd: zenpack command failed
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenUtils/zenpack.py", line 406, in <module>
zp.run()
File "/usr/local/zenoss/zenoss/Products/ZenUtils/zenpack.py", line 134, in run
previousVersion= self.options.previousVersion)
File "/local/zenoss/zenoss/Products/ZenUtils/ZenPackCmd.py", line 249, in InstallEggAndZenPack
forceRunExternal=forceRunExternal)
File "/local/zenoss/zenoss/Products/ZenUtils/ZenPackCmd.py", line 424, in InstallDistAsZenPack
zenPack.install(dmd)
File "/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/init.py", line 58, in install
self.rebuildRelations(app.zport.dmd)
File "/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/init.py", line 136, in rebuildRelations
d.changeDeviceClass(d.getDeviceClassPath())
File "/local/zenoss/zenoss/Products/ZenModel/Device.py", line 405, in changeDeviceClass
self.deviceClass().moveDevices(deviceClassPath, (self.id,))
File "/local/zenoss/zenoss/Products/ZenModel/DeviceClass.py", line 282, in moveDevices
devImport(xmlfile)
File "/local/zenoss/zenoss/Products/ZenModel/DeviceClass.py", line 271, in devImport
im.loadObjectFromXML(xmlfile)
File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 458, in loadObjectFromXML
parser.parse(self.infile)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 109, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 216, in feed
self._parser.Parse(data, isFinal)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 312, in start_element
self._cont_handler.startElement(name, AttributesImpl(attrs))
File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 145, in startElement
obj = self.createObject(attrs)
File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 286, in createObject
obj = klass(id)
TypeError: init() takes exactly 3 arguments (2 given)

 

 

At this point my windows device names to no longer appear. Name was now showing up as the IP address. Reports were broken.
Tried to reinstall

 

INFO:zen.ZenPackCMD:Previous ZenPack exists with same name ZenPacks.Nova.WinServiceSNMP
INFO:zen.ZenWinSrvSNMP:Removing menus
INFO:zen.ZenWinSrvSNMP:Stopping ZenWinSrvSNMP daemon
stopping...
INFO:zen.ZenPackCMD:installing zenpack ZenPacks.Nova.WinServiceSNMP; launching process
2012-07-24 11:03:00,803 INFO zen.ZPLoader: Loading /usr/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/objects/objects.xml
2012-07-24 11:03:01,664 INFO zen.AddToPack: End loading objects
2012-07-24 11:03:01,664 INFO zen.AddToPack: Processing links
2012-07-24 11:03:01,664 INFO zen.AddToPack: Loaded 0 objects into the ZODB database
2012-07-24 11:03:01,686 INFO zen.HookReportLoader: loading reports from:/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/reports
2012-07-24 11:03:01,753 INFO zen.ZenWinSrvSNMP: Installing menus
2012-07-24 11:03:01,905 INFO zen.ZenWinSrvSNMP: Linking daemon into $ZENHOME/Products/ZenRRD
2012-07-24 11:03:01,928 INFO zen.ZenWinSrvSNMP: Linking daemon listener into $ZENHOME/Products/ZenHub/services
2012-07-24 11:03:01,954 INFO zen.ZenWinSrvSNMP: Making daemon executable
2012-07-24 11:03:01,967 INFO zen.ZenWinSrvSNMP: Adding collector plugin 'community.snmp.WinServiceMap' to /Server/Windows
2012-07-24 11:03:01,968 INFO zen.ZenWinSrvSNMP: Starting ZenWinSrvSNMP daemon
starting...
2012-07-24 11:03:06,741 INFO zen.ZenWinSrvSNMP: Rebuilding device relations
2012-07-24 11:04:31,569 INFO zen.ImportRM: End loading objects
2012-07-24 11:04:31,569 INFO zen.ImportRM: Processing links
2012-07-24 11:04:52,764 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 11:05:38,055 INFO zen.ImportRM: End loading objects
2012-07-24 11:05:38,056 INFO zen.ImportRM: Processing links
2012-07-24 11:05:38,132 INFO zen.ImportRM: Would have created 58 objects in the ZODB database
2012-07-24 11:06:25,593 INFO zen.ImportRM: End loading objects
2012-07-24 11:06:25,593 INFO zen.ImportRM: Processing links
2012-07-24 11:06:25,898 INFO zen.ImportRM: Would have created 64 objects in the ZODB database
2012-07-24 11:06:35,053 ERROR zen.ZenPackCmd: zenpack command failed
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenUtils/zenpack.py", line 406, in <module>
zp.run()
File "/usr/local/zenoss/zenoss/Products/ZenUtils/zenpack.py", line 134, in run
previousVersion= self.options.previousVersion)
File "/local/zenoss/zenoss/Products/ZenUtils/ZenPackCmd.py", line 249, in InstallEggAndZenPack
forceRunExternal=forceRunExternal)
File "/local/zenoss/zenoss/Products/ZenUtils/ZenPackCmd.py", line 424, in InstallDistAsZenPack
zenPack.install(dmd)
File "/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/init.py", l
self.rebuildRelations(app.zport.dmd)
File "/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.Nova.WinServiceSNMP-1.0-py2.6.egg/ZenPacks/Nova/WinServiceSNMP/init.py", l
d.changeDeviceClass(d.getDeviceClassPath())
File "/local/zenoss/zenoss/Products/ZenModel/Device.py", line 405, in changeDeviceClass
self.deviceClass().moveDevices(deviceClassPath, (self.id,))
File "/local/zenoss/zenoss/Products/ZenModel/DeviceClass.py", line 282, in moveDevices
devImport(xmlfile)
File "/local/zenoss/zenoss/Products/ZenModel/DeviceClass.py", line 271, in devImport
im.loadObjectFromXML(xmlfile)
File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 458, in loadObjectFromXML
parser.parse(self.infile)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 109, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 216, in feed
self._parser.Parse(data, isFinal)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 312, in start_element
self._cont_handler.startElement(name, AttributesImpl(attrs))
File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 145, in startElement
obj = self.createObject(attrs)
File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 286, in createObject
obj = klass(id)
TypeError: init() takes exactly 3 arguments (2 given)
ERROR:zen.ZenPackCmd:zenpack command failed
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenUtils/zenpack.py", line 406, in <module>
zp.run()
File "/usr/local/zenoss/zenoss/Products/ZenUtils/zenpack.py", line 134, in run
previousVersion= self.options.previousVersion)
File "/local/zenoss/zenoss/Products/ZenUtils/ZenPackCmd.py", line 249, in InstallEggAndZenPack
forceRunExternal=forceRunExternal)
File "/local/zenoss/zenoss/Products/ZenUtils/ZenPackCmd.py", line 415, in InstallDistAsZenPack
(p.returncode, err))
ZenPackException: Error installing the egg (255): None

 

 

I restarted Zenoss with init script. Came up still slow w/ no device names on the windoze boxen. Clicking infrastructure resulted in just a Loading Please Wait message that sat there for minutes before i cancelled it.

 

Decided to reindex zope.


cp $ZENHOME/var/Data.fs $ZENHOME/var/Data.fs.bak

reindex()

commit()

 

Seemed to fix the really slow load times but windows device names were lost. Meanwhile i noticed the WinServicesSNMP zenpack starting to pickup services on my windows boxes. So Zenpack is working as far as I can tell. I went ahead and renamed my devices to what they should have been and the reports work again. Load average on the box seems have settled down - Zenpack itself seems to be working.

 

Any thoughts on what might have happened or things I can check to make sure my indexing / zenpacks are ok.

 

 

Thanks

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009

    First off you need to restart Zenoss after installing the pack.  It sounds like you may not have done that based on your description.  Secondly, you can move the devices out of /Server/Windows and then back in to rebuild the device relations if the rebuild failed during install.  If you don't want to do that you can use the following piece of zendmd code to do the rebuild.

     

    Edit: nevermind, I see you restarted Zenoss.  Try the zendmd code below.

     

     

    for d in dmd.Devices.unrestrictedTraverse('Server/Windows').getSubDevicesGen():
        print 'Rebuilding relations for %s' % (d.getDeviceName())
        d.buildRelations()
        if d.__class__ != d.deviceClass().getPythonDeviceClass():
            d.changeDeviceClass(d.getDeviceClassPath())
  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009

    Sorry, I should have read your whole post before commenting.  I'm not sure what happened there, you're the first to report that type of behaviour.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009

    It sort of sounds like something was messed up in your zope database...

     

    File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 216, in feed

    self._parser.Parse(data, isFinal)

    File "/usr/local/zenoss/python/lib/python2.6/site-packages/_xmlplus/sax/expatreader.py", line 312, in start_element

    self._cont_handler.startElement(name, AttributesImpl(attrs))

    File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 145, in startElement

    obj = self.createObject(attrs)

    File "/local/zenoss/zenoss/Products/ZenRelations/ImportRM.py", line 286, in createObject

    obj = klass(id)

    TypeError: init() takes exactly 3 arguments (2 given)

     

    It's programmatically passing the arguments in, so something was blank that shouldn't have been and it threw it off.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009

    Based on the output I'd say your problem has to do with device 10.10.10.43.  Also, you're not renaming your devices correctly which partially explains why all your names reverted before.  All you're doing are giving your devices titles that show up in the place of the name.  To actually rename the devices so that device.id equals the name of the device you need to go to the device's page, click on the gear menu in the bottom left hand corner, and select "Rename Device".  Changing the "Device Name:" field on the device's overview page just sets a title.  It should be labeled Device Title, no idea why they didn't.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009

    I'm glad that you were able to sort it out fairly easily.

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points