performance issue with multiple pagesets 🔗 link

Hi Henrik,

i loaded hobbit 4.0.5RC1 on my standby server (which isn't as fast as my
production bbgen3.5 server..)

i'm having lots of iowait, (it isn't swapping),
i suspect it's bbgen
i have 8 pagesets, and my hobbit/www directory is over 20M...
and generating all that every minute (sshhh.. i know..)

i'm using most of my pagesets only for the bb2.html, and i am not using the
bb.html page at all.

here's what is looks for my unix pageset :

[root at bigbrother-stdby www]# ls -lh unix/
total 2.0M
-rw-r--r--    1 bb-dev   alert         94K Jul 20 16:22 bb2.html
-rw-r--r--    1 bb-dev   alert        1.8M Jul 20 16:22 bb.html
-rw-r--r--    1 bb-dev   alert         519 Jul 20 16:22 bbnk.html
lrwxrwxrwx    1 bb-dev   alert           7 Jul 20 16:07 index.html -> bb.html
[root at bigbrother-stdby www]# 


--> Is there a way to disable the generating of the bb.html page for pagesets ?


--
Olivier Beau

On Wed, Jul 20, 2005 at 04:54:56PM +0200, Olivier Beau wrote:

It would be interesting to see the output from your "bbgen" column
if you run bbgen with the "--report" option.

I haven't seen bbgen do very much I/O. Sure, you're generating
8x2 MB of data per minute, but most disk today will easily deliver
10 mbit/sec throughput. And it's not random access, just sequential
writing of a single file.

I'd be more inclined to think it's your RRD file updates that are
responsible for the disk I/O you see. Try disabling the [rrddata]
and [rrdstatus] tasks in hobbitlaunch.cfg and see if the iowait drops.

--> Is there a way to disable the generating of the bb.html page for pagesets ?

No.


Regards,
Henrik

i've disabled [rrdstatus] and [rrddata].
i still have lots of I/O, practicaly no change.

i'm moved my www directory on another disk, and now most of the iowait is on the
other disk.
i then moved www in ramdisk, and all the iowait disapeared !


here is a paste of bbgen report, and then a paste of the unix bbgen report
(witch are similar whether www is on disk or ram):

bbgen for Hobbit version 4.0.5RC1

Statistics:
 Hosts               :  1617
 Status messages     :  8359
 Purple messages     :     8
 Pages               :   390


TIME SPENT
Event                                            Starttime          Duration
Startup                                  1121874341.021044                 -
Load links done                          1121874341.021332          0.000288 Load bbhosts done                        1121874341.147357          0.126025 ACK removal done                         1121874341.147481          0.000124 Load STATE done                          1121874341.540020          0.392539 Color calculation done                   1121874341.542579          0.002559 Hobbit pagegen start                     1121874341.542610          0.000031 Hobbit pagegen done                      1121874344.373265          2.830655 BB2 generation done                      1121874344.397772          0.024507 BBNK generation done                     1121874344.399964          0.002192 Summary transmission done                1121874344.399968          0.000004 Run completed                            1121874344.399969          0.000001 TIME TOTAL                                                          3.378925 


bbgen for Hobbit version 4.0.5RC1

Statistics:
 Hosts               :   589
 Status messages     :  8359
 Purple messages     :     0
 Pages               :     1


TIME SPENT
Event                                            Starttime          Duration
Startup                                  1121874421.091144                 -
Load links done                          1121874421.091429          0.000285 Load bbhosts done                        1121874421.192460          0.101031 ACK removal done                         1121874421.192561          0.000101 Load STATE done                          1121874421.497978          0.305417 Color calculation done                   1121874421.498891          0.000913 Hobbit pagegen start                     1121874421.498925          0.000034 Hobbit pagegen done                      1121874421.586783          0.087858 BB2 generation done                      1121874421.606831          0.020048 BBNK generation done                     1121874421.607756          0.000925 Run completed                            1121874421.607758          0.000002 TIME TOTAL                                                          0.516614 


Selon Henrik Stoerner <user-ce4a2c883f75@xymon.invalid>:

--

Olivier Beau

On Wed, Jul 20, 2005 at 06:14:04PM +0200, Olivier Beau wrote:

OK, so you just have a lot of disk I/O for those pages.

Just checking - the [bbstatus] module in hobbitlaunch.cfg is disabled ?
It should be, or you *will* have lots of disk I/O.

 Hosts               :  1617
 Status messages     :  8359
 Pages               :   390

OK, so your main view if split on quite a few pages...

Hobbit pagegen done                      1121874344.373265          2.830655 
TIME TOTAL                                                          3.378925 

and it takes 3.4 seconds to generate the main view with those 390 pages.
Not a lot. The other pageset takes even less:

TIME TOTAL                                                          0.516614 

0.5 secs total, and about 0.1 seconds to generate that page.

So building all of your pagesets should not take more than 5-10 seconds
max. Your system should be idle at least 80% of the time.

Your system is probably doing a great job of buffering the disk writes,
but I really can't see what the big problem is.

What OS are you running ?


Regards,
Henrik

Selon Henrik Stoerner <user-ce4a2c883f75@xymon.invalid>:

Yes, bbstatus is disabled,
the i/o is clearly on the www directory; i didn't touch the data directory
during my tests...

well, all that i/o load hangs the server for a couple second, 
(pretty anoying while doing vi..), and i say a couple of 
"Whoops ! bb failed to send message - timeout" during the I/O load

that's why i wondered about disabling the generation of bb.html on some pagesets..

for now i think i'll have to stick in a better raid card (the default raid card
on those server are pretty crummy...)

would it be possible to have those bb*.html page generated dynamicly ?

What OS are you running ?

i'm running redhat 3.0 on a compaq server with 2 CPU (1,1Ghz)


Regards,

Olivier

On Wed, Jul 20, 2005 at 07:28:44PM +0200, Olivier Beau wrote:

[snip] 

Hmmm - it shouldn't load your system that bad, it sounds like there's
some driver problem. Or a really crappy disk controller/driver. I 
don't know what Linux kernel you're running, but I know there's been
a lot of work going on lately with different algorithms for I/O
scheduling and ensuring a "fair" ressource sharing for processes.

If writing ~20 MB of data in one go can cause your system to stall, 
it does sound like a problem. If you can reproduce it with a simple 
"dd if=/dev/zero of=dummyfile size=4k count=5000" then I'd be inclined
to ask your Red Hat support contact if they have any ideas what to do.

that's why i wondered about disabling the generation of bb.html on some pagesets..

OK, the attached patch will let you do that - add "--no-pages" to the 
bbgen command line.

would it be possible to have those bb*.html page generated dynamicly ?

It's something I've thought about, yes - but haven't gotten around to
implement yet. The individual status pages are generated on-the-fly
so it would be a nice "clean" implementation to have everything be
dynamically generated. Plus you'd always have an up-to-date view.

I guess I won't run out of things to do for quite some time :-)


Regards,
Henrik

-------------- next part --------------
--- bbdisplay/bbgen.c	2005/07/14 08:12:25	1.214
+++ bbdisplay/bbgen.c	2005/07/20 16:25:26
@@ -101,6 +101,7 @@
 	int		embedded = 0;
 	int		hobbitddump = 0;
 	char		*envarea = NULL;
+	int		do_normal_pages = 1;
 
 	/* Setup standard header+footer (might be modified by option pageset) */
 	select_headers_and_footers("bb");
@@ -337,6 +338,9 @@
 		else if (strcmp(argv[i], "--no-acklog") == 0) {
 			bb2acklog = 0;
 		}
+		else if (strcmp(argv[i], "--no-pages") == 0) {
+			do_normal_pages = 0;
+		}
 
 		else if (argnmatch(argv[i], "--noprop=")) {
 			char *lp = strchr(argv[i], '=');
@@ -434,6 +438,7 @@
 			printf("    --eventignore=test[,test]   : Columns to ignore in bb2 event-log display\n");
 			printf("    --no-eventlog               : Do not generate the bb2 eventlog display\n");
 			printf("    --no-acklog                 : Do not generate the bb2 ack-log display\n");
+			printf("    --no-pages                  : Generate only the bb2 and bbnk pages\n");
 			printf("    --docurl=documentation-URL  : Hostnames link to a general (dynamic) web page for docs\n");
 			printf("    --no-doc-window             : Open doc-links in same window\n");
 			printf("    --htmlextension=.EXT        : Sets filename extension for generated file (default: .html\n");
@@ -598,7 +603,7 @@
 	if (reportstart && csvfile) {
 		csv_availability(csvfile, csvdelim);
 	}
-	else {
+	else if (do_normal_pages) {
 		do_page_with_subs(pagehead, dispsums);
 	}
 	add_timestamp("Hobbit pagegen done");

Selon Henrik Stoerner <user-ce4a2c883f75@xymon.invalid>:

i've did some more investigations...
and yes, the disk controller is REALLLLY crummy on those servers (compaq dl 380
g2); i'm having better performance with my old desktop (1Ghz and a 40G IDE drive !)
i really have to tick in a better scsi board...


Thank you for the patch henrik :))

Yes, that would be great : having an real-time view, and saving ressource for
hobbit server for large sites :)


--
Olivier Beau