Skip navigation
1 2 Previous Next 5455 Views 15 Replies Latest reply: Jan 31, 2012 11:50 AM by jmp242 RSS
tcaiazza Rank: White Belt 90 posts since
Mar 19, 2009
Currently Being Moderated

Jan 27, 2012 8:17 PM

Infrastructure Page slow to load

I'm running zenoss 3.2.1, and when I click the infrastructure page it take a very long time to load(10minutes+), some times it doesn't even load.

 

I found this old note in a thread somewhere, http://community.zenoss.org/docs/DOC-7798#d0e422,  about applying a patch for slow loading trees.  This was back in 3.0.1, so I'm assuming that this was fixed in 3.2.1.  Is that correct?

 

I don't see anything out of the ordinary on the server and restarting doesn't seem to help.  The logs look fine, granted I'm not sure if I'm looking at the right ones just what is under $ZENHOME/log.  Has anyone had issues like this?  Or can anyone point me to an area to focus on?

  • Stefan Reinke Rank: White Belt 72 posts since
    Jan 27, 2012
    Currently Being Moderated
    2. Jan 31, 2012 1:31 AM (in response to tcaiazza)
    Re: Infrastructure Page slow to load

    Hi,

     

    I am monitoring over 200 devices and my page takes about 5-15 secondes to load. Using Zenoss 3.2.1 and Chrome.

     

    What type of devices are you monitoring ?

  • Stefan Reinke Rank: White Belt 72 posts since
    Jan 27, 2012
    Currently Being Moderated
    4. Jan 31, 2012 8:23 AM (in response to tcaiazza)
    Re: Infrastructure Page slow to load

    Are all those devices pulling information (routes) etc. Or only some of them ? I have checked I only have about 30 devices pulling information and the rest only sends a icmp packet to check if the device is still alive.

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    5. Jan 31, 2012 8:44 AM (in response to Stefan Reinke)
    Re: Infrastructure Page slow to load

    On 3.2.1 with Opera 11.60, the initial ~40 or so of 167 devices show up fine for me. It does take a bit (30 seconds) to scroll farther as it loads more... Is your delay with the initial loading or with scrolling down the list?

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    7. Jan 31, 2012 10:05 AM (in response to tcaiazza)
    Re: Infrastructure Page slow to load

    tcaiazza:

     

    This can be many things, however, it's commonly substantiated by a browser JS engine issue. Can you run the follwing benchmark and let me know what your browser scores?

     

    http://v8.googlecode.com/svn/data/benchmarks/v5/run.html

     

    Best,

    --Shane (Hackman238)

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    8. Jan 31, 2012 10:07 AM (in response to tcaiazza)
    Re: Infrastructure Page slow to load

    You must have good eyes. I had to zoom out to 50% (where I can't read anything) to get more than 50 devices shown. It is indeed taking a while to display. My suggestion is "don't do that" for what it's worth. By default maximized web page window, I see 25 devices... Even up to 40 shows fast. It may well be a bug, but the work-around is easy.

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    10. Jan 31, 2012 10:20 AM (in response to tcaiazza)
    Re: Infrastructure Page slow to load

    tcaiazza:

     

    You might want to increase your zope object cache and zope cache (both in zope.conf) as well as ensure that mysql caching and threads are appropriate. The infrastructure page taxes the global catalog (in zope) and calls methods which query for event totals in mysql. If the page has to wait for either (due to IO wait or invalidation) the page will load very slowly. Maximizing caching helps ensure that this condition only occurs during invalidation events, which can be handled by increasing the invalidation entry in the zeo.conf.

     

    What are the current values? How large is your installation?

     

    --Shane

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    12. Jan 31, 2012 11:07 AM (in response to tcaiazza)
    Re: Infrastructure Page slow to load

    tcaiazza:

     

    Yeah that should definately be fine. I did try to replicate the problem, however, I couldn't duplciate the issue. My systems have massive amounts of resources, cache virtually everything and load balance across 40 zopes and is therefore not a good analog for this.

     

    It might be interesting to login to mysql and show processes while refreshing the page. I'm interested in the queury time for the page.

     

    --Shane

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    13. Jan 31, 2012 11:32 AM (in response to Shane Scott)
    Re: Infrastructure Page slow to load

    Zenoss graphing shows a MySQL  Spike when I tested this , FWIW...

     

    ZenossMySQL.png

     

    --

    James Pulver

    ZCA Member

    LEPP Computer Group

    Cornell University

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    14. Jan 31, 2012 11:39 AM (in response to jmp242)
    Re: Infrastructure Page slow to load

    James:

     

    Ouch! thats a serious spike.

     

    Watching here we don't see more than a tiny ripple. Here's my test config:

    [mysqld]

    socket = /opt/zends/data/zends.sock

    basedir = /opt/zends

    datadir = /opt/zends/data

    port = 13306

    user = zenoss

    innodb_file_per_table

    skip_external_locking

     

    key_buffer = 2048M                              # 25% of total memory on a machine that mainly runs MySQL is quite common

    innodb_buffer_pool_size = 22528M                # The larger you set this value, the less disk I/O is needed to access data in tables

    innodb_additional_mem_pool_size = 40M           # The more tables you have in your application, the more memory you need to allocate here

    innodb_log_file_size = 512M                     # from 1MB to 1/N-th of the size of the buffer pool, where N is the number of log files in the group

    innodb_log_buffer_size = 8M                     # allows large transactions to run without a need to write the log to disk before commit

    innodb_flush_log_at_trx_commit = 2              # complicated, see the docs. 1 is safest

    innodb_lock_wait_timeout = 50                   # timeout in seconds an InnoDB  transaction may wait for a row lock before giving up

    max_allowed_packet = 64M                        # should be as big as the largest BLOB you want to use

    table_cache = 24576                             # number of open tables for all threads

    sort_buffer_size = 256M                         # Increase this value for faster ORDER BY or GROUP BY operations

    read_buffer_size = 128M                         # If you do many sequential scans, you might want to increase this value

    read_rnd_buffer_size = 16M                      # change the session variable only from within those clients that need to run large queries

    myisam_sort_buffer_size = 1024M                 # size of the buffer that is allocated when sorting MyISAM indexes

    thread_cache_size = 512                         # can be increased to improve performance if you have a lot of new connections

    query_cache_size = 128M                         # amount of memory allocated for caching query results

    query_cache_limit = 3072M                       # Don't cache results that are larger than this number of bytes, leave high for zenoss

    thread_concurrency =24                          # Try number of CPU's*2 for thread_concurrency

    innodb_flush_method=O_DIRECT                   # according to a DBA here are Rackspace, this setting is difficult to apply correctly

     

    interactive_timeout = 600

    wait_timeout = 86400

    max_connections = 1500

    max_user_connections = 1500

    max_heap_table_size = 32768

     

    [client]

    socket = /opt/zends/data/zends.sock

    user = zenoss

     

    [mysql]

    prompt = "zends> "

    no-auto-rehash

     

    [mysqldump]

    quick

    max_allowed_packet = 64M

     

    [isamchk]

    key_buffer = 1024M

    sort_buffer_size = 256M

    read_buffer = 256M

    write_buffer = 256M

     

    [myisamchk]

    key_buffer = 1024M

    sort_buffer_size = 256M

    read_buffer = 256M

    write_buffer = 256M

1 2 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points