Skip navigation
1 2 3 4 Previous Next 15793 Views 51 Replies Latest reply: Apr 10, 2012 2:20 AM by Stefan Reinke RSS
Stefan Reinke Rank: White Belt 72 posts since
Jan 27, 2012
Currently Being Moderated

Feb 21, 2012 7:56 AM

How many Devices can Zenoss Core Handle ?

Hi,

 

I am running Zenoss Core 3.2.1, and I was just wondering how many devices Zenoss can run without giving problems ? ATM I have about 3000 on, with graphing on them. About 60 of them are servers and 20 Routers, the rest are just ping devices with uptime graphing.

 

The reason I am asking is zenoss stopped graphing my devices after the devices reached 1000+ . I checked all the daemons are still running I will try a restart and check how it goes.

 

The device count will top 10000, so that is why I am asking the forums just to see what other people have.

 

Spesifications of the server are high end, but I will post if you guys thinks it is needed.

 

Regards,

Stefan

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    1. Feb 21, 2012 8:36 AM (in response to Stefan Reinke)
    Re: How many Devices can Zenoss Core Handle ?

    This can be a tricky question, but with 10,000 devices you're going to need distributed collectors. Core 4 is going to be much more scalable, but in 3.2.1, you're looking at about 5000 per collector on really high end hardware with fusionIO cards etc.

     

    However, your environment is quite different in that you've got really 80 devices or so running SNMP checks, and 900 running ping checks. Basic ping checks don't stress the system near as much - especially if they're not graphing anything (RRD IO is one biggie in performance). At this scale, have you considered either the enterprise product or a community consultant to help you get going?

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    4. Feb 21, 2012 8:55 AM (in response to Stefan Reinke)
    Re: How many Devices can Zenoss Core Handle ?

    Command datasources  for all your graphs? Those are very performance intensive and should be used judiciously. Are there a bunch of scripts hung / zencommand instances?

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    7. Feb 21, 2012 9:18 AM (in response to Stefan Reinke)
    Re: How many Devices can Zenoss Core Handle ?

    Basically, you can't really do that many scripts. If its the same script for all devices, you might want to look into coding a daemon. The event queue exceeded isn't good either, you're not able to process events fast enough. What events are you getting?

     

    Every time you run a command based datasource, you're actually spawing a shell to run the command, waiting for it to complete, and return. You might be close to "fork-bombing" your server if you're doing 10,000 shells + command every 60 seconds.

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    8. Feb 21, 2012 10:54 AM (in response to Stefan Reinke)
    Re: How many Devices can Zenoss Core Handle ?

    Stefan:

     

    To properly scale ZenOSS core to very large data centers you'll need to put a lot of hardware in to it. Collections should be limited to a set number of datapoints and be assigned their own dedicated zenhubs. A combination of disk IO and collector configuration handling are the most difficult hurdles. At larger scales enterprise ZenOSS pays off due to enhanced distributed collector support and improved zenhub<->daemon configuration handling. You'll want to use a configuration similar to ours here at Rackspace since we tend to have the largest instances. If you have specific questions, ask away.

     

    --Shane

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    9. Feb 21, 2012 10:58 AM (in response to Shane Scott)
    Re: How many Devices can Zenoss Core Handle ?

    Stefan:

     

    I just happened to notice you're scripting tests. Please don't do that. It's criminally inefficient. Try to use the specialized daemons for each test. If you have tests which require special activities otherwise unsupported, adapt an existing daemon to do the work. Only use zencommand for tests which aren't implemented on a massive scale. ZenOSS does come with some overhead, but the programmatic functionality pays off big. There is not another product with similar programatic functionality.

     

    --Shane

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    12. Feb 22, 2012 8:16 AM (in response to Stefan Reinke)
    Re: How many Devices can Zenoss Core Handle ?

    The issue is not Zenoss exactly. It's that your OS can only handle so many scripts running, and it needs to be run for each check. It's not a disk IO issue, it's a running processes issue. Making copies won't help. Distributed collectors could I guess - each different computer is running only so many devices. However, it's far more efficient to use a daemon. If you need help creating a daemon, you can ask questions on the forum, or there are community developers who are available for consulting engagements.

     

    You can also just ask for such a daemon to be created, but I don't think there's much support - you're the first or second person in years I've seen ask for such a thing.

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    14. Feb 22, 2012 12:29 PM (in response to Stefan Reinke)
    Re: How many Devices can Zenoss Core Handle ?

    Stefan:

     

    Zenping automatically collects availability data. Look under reports. As for uptime, it's being collected by snmp.

     

    --Shane

1 2 3 4 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points