High CPU Load Rendering Graphs
list Vernon Everett
Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Galen Johnson
Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G=
▸
From: Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett <user-b3f8dacb72c8@xymon.invalid> Sent: Wednesday, April 23, 2014 10:17 PM To: Xymon mailinglist Subject: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon
▸
On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:
Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Galen Johnson
That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G=
▸
From: Vernon Everett <user-b3f8dacb72c8@xymon.invalid> Sent: Thursday, April 24, 2014 4:46 AM To: Galen Johnson Cc: Xymon mailinglist Subject: Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid<mailto:user-87f955643e3d@xymon.invalid>> wrote: Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= From: Xymon <xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>> on behalf of Vernon Everett <user-b3f8dacb72c8@xymon.invalid<mailto:user-b3f8dacb72c8@xymon.invalid>> Sent: Wednesday, April 23, 2014 10:17 PM To: Xymon mailinglist Subject: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
Hi Nope, no IPv6. Did a little constructive Googling on this error, and it's a fairly well known one. All the notes I can find tell me to add Listen 0.0.0.0:443 to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon
▸
On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:
That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= *From:* Vernon Everett <user-b3f8dacb72c8@xymon.invalid> *Sent:* Thursday, April 24, 2014 4:46 AM *To:* Galen Johnson *Cc:* Xymon mailinglist *Subject:* Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Jeremy Laidman
Is it quick to run the showgraph.cgi binary manually? Like so: SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly' /path/to/showgraph.cgi If that's slow, probably not an Apache problem. Are you running xymond_rrd with the "--no-cache" option? J
▸
On 29 April 2014 12:20, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:
Hi Nope, no IPv6. Did a little constructive Googling on this error, and it's a fairly well known one. All the notes I can find tell me to add Listen 0.0.0.0:443 to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= *From:* Vernon Everett <user-b3f8dacb72c8@xymon.invalid> *Sent:* Thursday, April 24, 2014 4:46 AM *To:* Galen Johnson *Cc:* Xymon mailinglist *Subject:* Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
Hi Jeremy
Thanks for the tips.
Manually, it runs in under a second.
And not using the "--no-cache" option?
--- snip ---
# "rrdstatus" updates RRD files with information that arrives as "status"
messages.
[rrdstatus]
ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log
xymond_rrd --rrddir=$XYMONVAR/rrd
# "rrddata" updates RRD files with information that arrives as "data"
messages.
[rrddata]
ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log
xymond_rrd --rrddir=$XYMONVAR/rrd
--- snip ---
Something I did notice, that may be relevant.
As I mentioned earlier, viewing the trends column spawns a host of
showgraph.cgi processes.
I ran a truss on one of them, and I see this group of errors repeated often.
--- snip ---
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
munmap(0xFE380000, 399) = 0
xstat(2,
"/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
0x0803F2D0) = 0
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F130) = 0
mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000
close(7) = 0
--- snip ---
This looks like it's trying to find locale related stuff.
However, I can find no reference to any zh_CN locales anywhere on my system.
Any ideas?
Regards
Vernon
▸
On 29 April 2014 10:46, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:
Is it quick to run the showgraph.cgi binary manually? Like so: SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly' /path/to/showgraph.cgi If that's slow, probably not an Apache problem. Are you running xymond_rrd with the "--no-cache" option? J On 29 April 2014 12:20, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:Hi Nope, no IPv6. Did a little constructive Googling on this error, and it's a fairly well known one. All the notes I can find tell me to add Listen 0.0.0.0:443 to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= *From:* Vernon Everett <user-b3f8dacb72c8@xymon.invalid> *Sent:* Thursday, April 24, 2014 4:46 AM *To:* Galen Johnson *Cc:* Xymon mailinglist *Subject:* Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
Just had a more careful look at the truss output.
It's trying to open every font or locale in existence.
Is this normal?
--- snip ---
open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Italic.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Italic.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
munmap(0xFE0A0000, 15723) = 0
xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm",
0x0803F2C0) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F120) = 0
mmap(0x00000000, 15742, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000
close(7) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Roman.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Roman.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/rsrc", O_RDONLY)
Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
munmap(0xFE0A0000, 15742) = 0
xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", 0x0803F2C0)
= 0
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F120) = 0
mmap(0x00000000, 17286, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000
close(7) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/._Rockwell-Bold.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Rockwell-Bold.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/rsrc", O_RDONLY)
Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
--- snip ---
▸
On 29 April 2014 12:07, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:
Hi Jeremy Thanks for the tips. Manually, it runs in under a second. And not using the "--no-cache" option? --- snip --- # "rrdstatus" updates RRD files with information that arrives as "status" messages. [rrdstatus] ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg NEEDS xymond CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd # "rrddata" updates RRD files with information that arrives as "data" messages. [rrddata] ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg NEEDS xymond CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log xymond_rrd --rrddir=$XYMONVAR/rrd --- snip --- Something I did notice, that may be relevant. As I mentioned earlier, viewing the trends column spawns a host of showgraph.cgi processes. I ran a truss on one of them, and I see this group of errors repeated often. --- snip --- open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc", O_RDONLY) Err#20 ENOTDIR open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT munmap(0xFE380000, 399) = 0 xstat(2, "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", 0x0803F2D0) = 0 open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", O_RDONLY) = 7 fcntl(7, F_SETFD, 0x00000001) = 0 fxstat(2, 7, 0x0803F130) = 0 mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000 close(7) = 0 --- snip --- This looks like it's trying to find locale related stuff. However, I can find no reference to any zh_CN locales anywhere on my system. Any ideas? Regards Vernon On 29 April 2014 10:46, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:Is it quick to run the showgraph.cgi binary manually? Like so: SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly' /path/to/showgraph.cgi If that's slow, probably not an Apache problem. Are you running xymond_rrd with the "--no-cache" option? J On 29 April 2014 12:20, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:Hi Nope, no IPv6. Did a little constructive Googling on this error, and it's a fairly well known one. All the notes I can find tell me to add Listen 0.0.0.0:443 to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= *From:* Vernon Everett <user-b3f8dacb72c8@xymon.invalid> *Sent:* Thursday, April 24, 2014 4:46 AM *To:* Galen Johnson *Cc:* Xymon mailinglist *Subject:* Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Jeremy Laidman
I've seem this type of output before from truss/strace, but I think it was for all locales. Are you sure this wasn't just a sample of lookups for all locales? I wonder if this is to do with librrd trying to write text onto the graph. My librrd has some font paths hard-coded, but I can imagine some builds might go looking elsewhere for fonts, or might be falling back to some setting inside Apache. But if this is the problem, then I would expect running it manually would be slow. Try running manually through truss and see if you get the same output. Also, perhaps try temporarily renaming the whole "locale" directory to "locale.off" and see if the processes complete faster. If they do, then it looks like it's related to this font location behaviour. Cheers Jeremy
▸
On 29 April 2014 14:07, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:
Hi Jeremy Thanks for the tips. Manually, it runs in under a second. And not using the "--no-cache" option? --- snip --- # "rrdstatus" updates RRD files with information that arrives as "status" messages. [rrdstatus] ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg NEEDS xymond CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd # "rrddata" updates RRD files with information that arrives as "data" messages. [rrddata] ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg NEEDS xymond CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log xymond_rrd --rrddir=$XYMONVAR/rrd --- snip --- Something I did notice, that may be relevant. As I mentioned earlier, viewing the trends column spawns a host of showgraph.cgi processes. I ran a truss on one of them, and I see this group of errors repeated often. --- snip --- open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc", O_RDONLY) Err#20 ENOTDIR open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT munmap(0xFE380000, 399) = 0 xstat(2, "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", 0x0803F2D0) = 0 open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", O_RDONLY) = 7 fcntl(7, F_SETFD, 0x00000001) = 0 fxstat(2, 7, 0x0803F130) = 0 mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000 close(7) = 0 --- snip --- This looks like it's trying to find locale related stuff. However, I can find no reference to any zh_CN locales anywhere on my system. Any ideas? Regards Vernon On 29 April 2014 10:46, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:Is it quick to run the showgraph.cgi binary manually? Like so: SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly' /path/to/showgraph.cgi If that's slow, probably not an Apache problem. Are you running xymond_rrd with the "--no-cache" option? J On 29 April 2014 12:20, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:Hi Nope, no IPv6. Did a little constructive Googling on this error, and it's a fairly well known one. All the notes I can find tell me to add Listen 0.0.0.0:443 to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= *From:* Vernon Everett <user-b3f8dacb72c8@xymon.invalid> *Sent:* Thursday, April 24, 2014 4:46 AM *To:* Galen Johnson *Cc:* Xymon mailinglist *Subject:* Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Ryan Novosielski
I have this problem too with Solaris. Or I should say I went from not having this problem to having this problem when I did a Solaris upgrade (from one update to another). I rolled back the upgrade and the problem went away. But I never identified what patch (presumably a patch) caused the problem. I'd love to fix this and move forward though.
▸
From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, April 23, 2014 10:17 PM To: Xymon mailinglist <xymon at xymon.com> Subject: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Ryan Novosielski
I believe I tried this and it didn't help. I did consult the list at that time, so Vernon, you might want to check the archives to see what I already tried.
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Tuesday, April 29, 2014 12:24 AM
To: Vernon Everett <user-b3f8dacb72c8@xymon.invalid>
Cc: Xymon mailinglist <xymon at xymon.com>
Subject: Re: [Xymon] High CPU Load Rendering Graphs
I've seem this type of output before from truss/strace, but I think it was for all locales. Are you sure this wasn't just a sample of lookups for all locales?
I wonder if this is to do with librrd trying to write text onto the graph. My librrd has some font paths hard-coded, but I can imagine some builds might go looking elsewhere for fonts, or might be falling back to some setting inside Apache.
But if this is the problem, then I would expect running it manually would be slow. Try running manually through truss and see if you get the same output.
Also, perhaps try temporarily renaming the whole "locale" directory to "locale.off" and see if the processes complete faster. If they do, then it looks like it's related to this font location behaviour.
Cheers
Jeremy
On 29 April 2014 14:07, Vernon Everett <user-b3f8dacb72c8@xymon.invalid<mailto:user-b3f8dacb72c8@xymon.invalid>> wrote:
Hi Jeremy
Thanks for the tips.
Manually, it runs in under a second.
And not using the "--no-cache" option?
--- snip ---
# "rrdstatus" updates RRD files with information that arrives as "status" messages.
[rrdstatus]
ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd
# "rrddata" updates RRD files with information that arrives as "data" messages.
[rrddata]
ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log xymond_rrd --rrddir=$XYMONVAR/rrd
--- snip ---
Something I did notice, that may be relevant.
As I mentioned earlier, viewing the trends column spawns a host of showgraph.cgi processes.
I ran a truss on one of them, and I see this group of errors repeated often.
--- snip ---
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc", O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT
munmap(0xFE380000, 399) = 0
xstat(2, "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", 0x0803F2D0) = 0
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F130) = 0
mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000
close(7) = 0
--- snip ---
This looks like it's trying to find locale related stuff.
However, I can find no reference to any zh_CN locales anywhere on my system.
Any ideas?
Regards
Vernon
On 29 April 2014 10:46, Jeremy Laidman <user-71895fb2e44c@xymon.invalid<mailto:user-71895fb2e44c@xymon.invalid>> wrote:
Is it quick to run the showgraph.cgi binary manually? Like so:
SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com<http://hostname.example.com>&service=la&graph=hourly'; /path/to/showgraph.cgi
▸
If that's slow, probably not an Apache problem.
Are you running xymond_rrd with the "--no-cache" option?
J
On 29 April 2014 12:20, Vernon Everett <user-b3f8dacb72c8@xymon.invalid<mailto:user-b3f8dacb72c8@xymon.invalid>> wrote:
Hi
Nope, no IPv6.
Did a little constructive Googling on this error, and it's a fairly well known one.
All the notes I can find tell me to addListen 0.0.0.0:443<http://0.0.0.0:443>;
▸
to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid<mailto:user-87f955643e3d@xymon.invalid>> wrote: That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= From: Vernon Everett <user-b3f8dacb72c8@xymon.invalid<mailto:user-b3f8dacb72c8@xymon.invalid>> Sent: Thursday, April 24, 2014 4:46 AM To: Galen Johnson Cc: Xymon mailinglist Subject: Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid<mailto:user-87f955643e3d@xymon.invalid>> wrote: Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= From: Xymon <xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>> on behalf of Vernon Everett <user-b3f8dacb72c8@xymon.invalid<mailto:user-b3f8dacb72c8@xymon.invalid>> Sent: Wednesday, April 23, 2014 10:17 PM To: Xymon mailinglist Subject: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
"Curiouser and curiouser!" said Alice.
Running truss from the command line, still runs pretty quick, and gives me
similar errors, and appears to be opening font files, but from a different
location, and not trying quite as many different files.
From command line, it's using fonts on /var/opt/csw/...
Normally, it's searching /usr/openwin/lib/...
--- snip ---
xstat(2, "/usr/openwin/lib/locale/ar/X11/fonts", 0x0803D4E0) = 0
open("/var/opt/csw/cache/fontconfig/17d92556e6d5835205e09e630e4ebbba-le32d4.cache-3",
O_RDONLY) Err#2 ENOENT
open("/usr/local/xymon/.fontconfig/17d92556e6d5835205e09e630e4ebbba-le32d4.cache-3",
O_RDONLY) = 5
fxstat(2, 5, 0x0803D5B0) = 0
read(5, "04FC02FC03\0\0\0C0\0\0\0".., 192) = 192
close(5) = 0
xstat(2, "/usr/openwin/lib/locale/en_AU.UTF-8/X11/fonts", 0x0803D4E0) = 0
open("/var/opt/csw/cache/fontconfig/dc5030b131e61fddf5c2a81c27184e48-le32d4.cache-3",
O_RDONLY) Err#2 ENOENT
open("/usr/local/xymon/.fontconfig/dc5030b131e61fddf5c2a81c27184e48-le32d4.cache-3",
O_RDONLY) = 5
fxstat(2, 5, 0x0803D5B0) = 0
read(5, "04FC02FC03\0\0\0A0\0\0\0".., 160) = 160
close(5) = 0
xstat(2, "/usr/openwin/lib/locale/en_NZ.UTF-8/X11/fonts", 0x0803D4E0) = 0
open("/var/opt/csw/cache/fontconfig/085c56eac2786bc78910232323657f58-le32d4.cache-3",
O_RDONLY) Err#2 ENOENT
open("/usr/local/xymon/.fontconfig/085c56eac2786bc78910232323657f58-le32d4.cache-3",
O_RDONLY) = 5
fxstat(2, 5, 0x0803D5B0) = 0
read(5, "04FC02FC03\0\0\0A0\0\0\0".., 160) = 160
close(5) = 0
xstat(2, "/usr/openwin/lib/locale/en_US.UTF-8/X11/fonts", 0x0803D4E0) = 0
open("/var/opt/csw/cache/fontconfig/62bec7de5a4b10f67c3506b74fa341f8-le32d4.cache-3",
O_RDONLY) Err#2 ENOENT
open("/usr/local/xymon/.fontconfig/62bec7de5a4b10f67c3506b74fa341f8-le32d4.cache-3",
O_RDONLY) = 5
fxstat(2, 5, 0x0803D5B0) = 0
read(5, "04FC02FC03\0\0\0A0\0\0\0".., 160) = 160
close(5) = 0
xstat(2, "/usr/openwin/lib/locale/euro_fonts/X11/fonts", 0x0803D4E0) = 0
open("/var/opt/csw/cache/fontconfig/4eac2ae93f2253367493efcbe9322e7f-le32d4.cache-3",
O_RDONLY) Err#2 ENOENT
open("/usr/local/xymon/.fontconfig/4eac2ae93f2253367493efcbe9322e7f-le32d4.cache-3",
O_RDONLY) = 5
fxstat(2, 5, 0x0803D5B0) = 0
read(5, "04FC02FC03\0\0\0A0\0\0\0".., 160) = 160
close(5) = 0
--- snip ---
Something else interesting, it's also looking at
/usr/local/xymon/.fontconfig/...
Doing a strings on one of these files,
# strings
/usr/local/xymon/.fontconfig/fe2855a3721a02bc50804ac04520849b-le32d4.cache-3
| grep "/usr/openwin"
J/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/angsa.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/angsab.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/angsai.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/angsaz.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/browa.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/browab.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/browai.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/browaz.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/cordia.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/cordiab.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/cordiai.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/cordiaz.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/lucida.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/lucidab.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/lucidai.ttf
/usr/openwin/lib/locale/th_TH/X11/fonts/TrueType/lucidaz.ttf
The /usr/local/xymon/.fontconfig files are date-stamped today.
So what's different?
One thing for sure, all the fonts listed in the
/usr/local/xymon/.fontconfig files do exist.
This gives no output.
# for i in *
do
strings $i | grep "^/usr/openwin" | while read a
do
[ -s $a ] || echo $a
done
done
And reversing the logic, lists plenty. Piping it to wc -l gives 1599.
strings * | wc -l
Also returns 1599.
It looks to me like Xymon keeps a cache of the fonts that really exist.
Can anybody look at the code for me and confirm this is the case?
When running from command line, it's using the Xymon cache, and only
attempting to open fonts that really exist.
When running normally, it's trying to open all sorts of fonts that don't
exist.
So I need to figure out where it's getting those non-existent fonts from.
It also puzzles me why it first tries to open a font cache file in
/var/opt/csw/cache/fontconfig/
And then tries to open the exact same file in the
/usr/local/xymon/.fontconfig directory.
▸
Regards
Vernon
On 29 April 2014 12:24, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:
I've seem this type of output before from truss/strace, but I think it was for all locales. Are you sure this wasn't just a sample of lookups for all locales? I wonder if this is to do with librrd trying to write text onto the graph. My librrd has some font paths hard-coded, but I can imagine some builds might go looking elsewhere for fonts, or might be falling back to some setting inside Apache. But if this is the problem, then I would expect running it manually would be slow. Try running manually through truss and see if you get the same output. Also, perhaps try temporarily renaming the whole "locale" directory to "locale.off" and see if the processes complete faster. If they do, then it looks like it's related to this font location behaviour. Cheers Jeremy On 29 April 2014 14:07, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:Hi Jeremy Thanks for the tips. Manually, it runs in under a second. And not using the "--no-cache" option? --- snip --- # "rrdstatus" updates RRD files with information that arrives as "status" messages. [rrdstatus] ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg NEEDS xymond CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd # "rrddata" updates RRD files with information that arrives as "data" messages. [rrddata] ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg NEEDS xymond CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log xymond_rrd --rrddir=$XYMONVAR/rrd --- snip --- Something I did notice, that may be relevant. As I mentioned earlier, viewing the trends column spawns a host of showgraph.cgi processes. I ran a truss on one of them, and I see this group of errors repeated often. --- snip --- open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc", O_RDONLY) Err#20 ENOTDIR open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT munmap(0xFE380000, 399) = 0 xstat(2, "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", 0x0803F2D0) = 0 open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", O_RDONLY) = 7 fcntl(7, F_SETFD, 0x00000001) = 0 fxstat(2, 7, 0x0803F130) = 0 mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000 close(7) = 0 --- snip --- This looks like it's trying to find locale related stuff. However, I can find no reference to any zh_CN locales anywhere on my system. Any ideas? Regards Vernon On 29 April 2014 10:46, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:Is it quick to run the showgraph.cgi binary manually? Like so: SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly' /path/to/showgraph.cgi If that's slow, probably not an Apache problem. Are you running xymond_rrd with the "--no-cache" option? J On 29 April 2014 12:20, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:Hi Nope, no IPv6. Did a little constructive Googling on this error, and it's a fairly well known one. All the notes I can find tell me to add Listen 0.0.0.0:443 to my httpd.conf file, and the error will go away. But that would be too easy. Not working in my world. And I still get the error. :-/ In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart. But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering. Lots of junk in the error.log file I can fix with an appropriate entry in logadm. Taking forever to draw my graphs is a pain. Has anybody experienced problems with the rrd graphs taking long to render? Regards Vernon On 25 April 2014 03:02, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it). =G= *From:* Vernon Everett <user-b3f8dacb72c8@xymon.invalid> *Sent:* Thursday, April 24, 2014 4:46 AM *To:* Galen Johnson *Cc:* Xymon mailinglist *Subject:* Re: [Xymon] High CPU Load Rendering Graphs Yes, and no. Have just enabled the status page in the web config, and it appears to have got rid of the one error message. (Not sure how I missed the config change at initial install time) But, I still get the other error [warn] (128)Network is unreachable: connect to listener on [::]:443 And my graphs still take way too long to render, and send my CPU utilisation through the roof. Public holiday here tomorrow, so only back at this client on Tuesday. Thanks Vernon On 24 April 2014 10:27, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs? =G= *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett < user-b3f8dacb72c8@xymon.invalid> *Sent:* Wednesday, April 23, 2014 10:17 PM *To:* Xymon mailinglist *Subject:* [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Ryan Novosielski
It appears as if, for me, upgrading from Solaris 10u10 to Solaris 10u11 caused this problem. Incidentally, I'm on SPARC. But I know it's just about the same issue as when I did a truss, it was the same type of stuff that was going on (searches through locale and font directories). I think it took nearly a minute to generate graphs.
▸
From: Novosielski, Ryan [mailto:user-6e4f7a3bb37f@xymon.invalid] Sent: Tuesday, April 29, 2014 12:59 AM To: 'user-b3f8dacb72c8@xymon.invalid' <user-b3f8dacb72c8@xymon.invalid>; 'xymon at xymon.com' <xymon at xymon.com> Subject: Re: [Xymon] High CPU Load Rendering Graphs I have this problem too with Solaris. Or I should say I went from not having this problem to having this problem when I did a Solaris upgrade (from one update to another). I rolled back the upgrade and the problem went away. But I never identified what patch (presumably a patch) caused the problem. I'd love to fix this and move forward though. From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, April 23, 2014 10:17 PM To: Xymon mailinglist <xymon at xymon.com> Subject: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
Hi Ryan Not sure it's related to the Solaris release or a specific patch. The server is an old one, running S10u5, and was last patched in 2011.
▸
Regards
Vernon
On 29 April 2014 13:34, Novosielski, Ryan <user-6e4f7a3bb37f@xymon.invalid> wrote:
It appears as if, for me, upgrading from Solaris 10u10 to Solaris 10u11 caused this problem. Incidentally, I'm on SPARC. But I know it's just about the same issue as when I did a truss, it was the same type of stuff that was going on (searches through locale and font directories). I think it took nearly a minute to generate graphs. *From*: Novosielski, Ryan [mailto:user-6e4f7a3bb37f@xymon.invalid] *Sent*: Tuesday, April 29, 2014 12:59 AM *To*: 'user-b3f8dacb72c8@xymon.invalid' <user-b3f8dacb72c8@xymon.invalid>; ' xymon at xymon.com' <xymon at xymon.com> *Subject*: Re: [Xymon] High CPU Load Rendering Graphs I have this problem too with Solaris. Or I should say I went from not having this problem to having this problem when I did a Solaris upgrade (from one update to another). I rolled back the upgrade and the problem went away. But I never identified what patch (presumably a patch) caused the problem. I'd love to fix this and move forward though. *From*: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] *Sent*: Wednesday, April 23, 2014 10:17 PM *To*: Xymon mailinglist <xymon at xymon.com> *Subject*: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Vernon Everett
Hi Ryan Trying to narrow this one down a little, based on what was same and different on your system. We can exclude - Architecture. I have x86 and you had Sparc. - Solaris build version. I have s10u5 and you had s10u11. But it worked on S10u10. Makes me wonder what Xymon dependencies may have been upgrade between u10 and u11. Things I am checking. Can you check what versions you have please? - I am using perl 5.8.4. What are you using? - I was using Apache2 2.2.22, and upgrade to 2.2.26 and we still have this issue. - CGI version 3.33 - RRDtool 1.4.7 Thanks Vernon
▸
On 29 April 2014 13:48, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:
Hi Ryan Not sure it's related to the Solaris release or a specific patch. The server is an old one, running S10u5, and was last patched in 2011. Regards Vernon On 29 April 2014 13:34, Novosielski, Ryan <user-6e4f7a3bb37f@xymon.invalid> wrote:It appears as if, for me, upgrading from Solaris 10u10 to Solaris 10u11 caused this problem. Incidentally, I'm on SPARC. But I know it's just about the same issue as when I did a truss, it was the same type of stuff that was going on (searches through locale and font directories). I think it took nearly a minute to generate graphs. *From*: Novosielski, Ryan [mailto:user-6e4f7a3bb37f@xymon.invalid] *Sent*: Tuesday, April 29, 2014 12:59 AM *To*: 'user-b3f8dacb72c8@xymon.invalid' <user-b3f8dacb72c8@xymon.invalid>; ' xymon at xymon.com' <xymon at xymon.com> *Subject*: Re: [Xymon] High CPU Load Rendering Graphs I have this problem too with Solaris. Or I should say I went from not having this problem to having this problem when I did a Solaris upgrade (from one update to another). I rolled back the upgrade and the problem went away. But I never identified what patch (presumably a patch) caused the problem. I'd love to fix this and move forward though. *From*: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] *Sent*: Wednesday, April 23, 2014 10:17 PM *To*: Xymon mailinglist <xymon at xymon.com> *Subject*: [Xymon] High CPU Load Rendering Graphs Hi all My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view. It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5.. Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1 Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems. So I am not really thinking the issue is with the hardware. It's been slow since it was installed. If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times. Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user. Top under normal conditions. CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap Top rendering the trends column. CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Also getting this error (128)Network is unreachable: connect to listener on [::]:443 in my Apache error.log file, repeated every second while rendering the graphs. And from time to time, I get this one. File does not exist: /opt/csw/apache2/share/htdocs/server-status Anybody seen anything like this? Perhaps know of somewhere I can look for more info? I have looked at this http://lists.xymon.com/archive/2014-January/038780.html But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference. Other info that may be important.... bash-3.00# ./httpd -v Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 bash-3.00# ./httpd -V Server version: Apache/2.2.22 (Unix) Server built: Jun 1 2012 05:09:20 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.6, APR-Util 1.3.12 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/opt/csw/apache2" -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/mime.types" -D SERVER_CONFIG_FILE="etc/httpd.conf" Thanks Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton