Trace number 3728876

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pwbo 2.02MSAT (TO) 1800.18 900.328

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB06/roussel/
normalized-pigeon-cardinality-31-30--soft-0-100-0.wbo
MD5SUM168e81c26eea0b6e6155adb2f3f4d195
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark0.037993
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables930
Total number of constraints61
Number of soft constraints61
Number of constraints which are clauses31
Number of constraints which are cardinality constraints (but not clauses)30
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint30
Maximum length of a constraint31
Top cost 3358
Min constraint cost 1
Max constraint cost 97
Sum of constraints costs 3357
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 32
Number of bits of the biggest sum of numbers6
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Quality of the solution as a function of time


Download the above graph as a PDF file / as an EPS file

Solver Data 

0.00/0.00	c ==============================[ Solver Version ]===============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  pwbo:                     2.02 beta - May, 2012                            |
0.00/0.00	c |                                                                             |
0.00/0.00	c Parsing wbo file format. File HOME/instance-3728876-1338673990.wbo.
0.00/0.00	c Instance file HOME/instance-3728876-1338673990.wbo
0.00/0.00	c File size is 15679 bytes.
0.00/0.00	c Highest Coefficient sum: 32
0.00/0.00	c ============================[ Problem Statistics ]=============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Parsing time:          0.00         s                                      |
0.00/0.00	c |  Number Variables:      991                                                 |
0.00/0.00	c |  Number Clauses:        31                                                  |
0.00/0.00	c |  Number Cardinality:    0                                                   |
0.00/0.00	c |  Number PB Constraints: 30                                                  |
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Problem Type:          Optimization                                        |
0.00/0.00	c |  Objective Function:    PB Constraint                                       |
0.00/0.00	c |                                                                             |
0.00/0.00	c ===============================================================================
1800.18/900.31	o 91
1800.18/900.31	o 39
1800.18/900.31	o 1
1800.18/900.31	c ===============================================================================
1800.18/900.31	c Wall Clock Time: 900.31 s
1800.18/900.31	c Best Solution: 1
1800.18/900.31	s SATISFIABLE
1800.18/900.31	v -x1 -x2 -x3 -x4 -x5 -x6 -x7 -x8 -x9 -x10 -x11 -x12 -x13 -x14 -x15 -x16 -x17 -x18 -x19 -x20 -x21 -x22 -x23 -x24 -x25 x26 -x27 -x28 -x29 -x30 -x31 -x32 -x33 -x34 -x35 -x36 -x37 -x38 -x39 -x40 -x41 -x42 -x43 -x44 -x45 -x46 -x47 -x48 -x49 -x50 -x51 -x52 -x53 -x54 x55 -x56 -x57 -x58 -x59 -x60 -x61 -x62 -x63 -x64 -x65 -x66 -x67 -x68 -x69 -x70 -x71 -x72 -x73 -x74 -x75 -x76 -x77 -x78 -x79 -x80 -x81 -x82 -x83 -x84 -x85 -x86 -x87 x88 -x89 -x90 -x91 -x92 -x93 -x94 -x95 -x96 -x97 -x98 -x99 -x100 -x101 -x102 -x103 -x104 -x105 -x106 -x107 -x108 -x109 -x110 -x111 -x112 -x113 -x114 -x115 -x116 x117 -x118 -x119 -x120 -x121 -x122 -x123 -x124 -x125 -x126 -x127 -x128 -x129 -x130 -x131 -x132 -x133 -x134 -x135 -x136 -x137 -x138 -x139 -x140 -x141 -x142 -x143 -x144 -x145 -x146 -x147 -x148 x149 -x150 -x151 -x152 -x153 -x154 -x155 -x156 -x157 -x158 -x159 -x160 -x161 -x162 -x163 -x164 -x165 -x166 -x167 -x168 -x169 -x170 -x171 -x172 -x173 -x174 -x175 -x176 -x177 -x178 -x179 x180 -x181 -x182 -x183 -x184 -x185 -x186 -x187 -x188 -x189 -x190 -x191 -x192 -x193 -x194 -x195 -x196 -x197 -x198 -x199 -x200 -x201 -x202 -x203 -x204 -x205 -x206 -x207 -x208 -x209 -x210 -x211 -x212 -x213 -x214 -x215 -x216 -x217 -x218 -x219 -x220 -x221 -x222 -x223 -x224 -x225 -x226 -x227 -x228 -x229 -x230 -x231 -x232 -x233 x234 -x235 -x236 -x237 -x238 -x239 -x240 -x241 -x242 -x243 -x244 -x245 -x246 -x247 -x248 -x249 -x250 -x251 -x252 -x253 -x254 -x255 -x256 -x257 -x258 -x259 -x260 -x261 -x262 x263 -x264 -x265 -x266 -x267 -x268 -x269 -x270 -x271 -x272 -x273 -x274 -x275 -x276 -x277 -x278 -x279 -x280 -x281 -x282 -x283 -x284 -x285 -x286 -x287 -x288 -x289 -x290 -x291 x292 -x293 -x294 -x295 -x296 -x297 -x298 -x299 -x300 -x301 -x302 -x303 -x304 -x305 -x306 -x307 -x308 -x309 -x310 -x311 -x312 -x313 -x314 -x315 -x316 -x317 -x318 -x319 -x320 x321 -x322 -x323 -x324 -x325 -x326 -x327 -x328 -x329 -x330 -x331 -x332 -x333 -x334 -x335 -x336 -x337 -x338 -x339 -x340 -x341 -x342 -x343 -x344 -x345 -x346 -x347 -x348 -x349 x350 -x351 -x352 -x353 -x354 -x355 -x356 -x357 -x358 -x359 -x360 -x361 -x362 -x363 -x364 -x365 -x366 -x367 -x368 -x369 -x370 -x371 -x372 -x373 -x374 -x375 -x376 -x377 -x378 x379 -x380 -x381 -x382 -x383 -x384 -x385 -x386 -x387 -x388 -x389 -x390 -x391 -x392 -x393 -x394 -x395 -x396 -x397 -x398 -x399 -x400 -x401 -x402 -x403 -x404 -x405 -x406 -x407 x408 -x409 -x410 -x411 -x412 -x413 -x414 -x415 -x416 -x417 -x418 -x419 -x420 -x421 -x422 -x423 -x424 -x425 -x426 -x427 -x428 -x429 -x430 -x431 -x432 -x433 -x434 -x435 -x436 x437 -x438 -x439 -x440 -x441 -x442 -x443 -x444 -x445 -x446 -x447 -x448 -x449 -x450 -x451 -x452 -x453 -x454 -x455 -x456 -x457 -x458 -x459 -x460 -x461 -x462 -x463 -x464 -x465 x466 -x467 -x468 -x469 -x470 -x471 -x472 -x473 -x474 -x475 -x476 -x477 -x478 -x479 -x480 -x481 -x482 -x483 -x484 -x485 -x486 -x487 -x488 -x489 -x490 -x491 -x492 -x493 -x494 x495 -x496 -x497 -x498 -x499 -x500 -x501 -x502 -x503 -x504 -x505 -x506 -x507 -x508 -x509 -x510 -x511 -x512 -x513 -x514 -x515 -x516 -x517 -x518 -x519 -x520 -x521 -x522 -x523 x524 -x525 -x526 -x527 -x528 -x529 -x530 -x531 -x532 -x533 -x534 -x535 -x536 -x537 -x538 -x539 -x540 -x541 -x542 -x543 -x544 -x545 -x546 -x547 -x548 -x549 -x550 -x551 -x552 x553 -x554 -x555 -x556 -x557 -x558 -x559 -x560 -x561 -x562 -x563 -x564 -x565 -x566 -x567 -x568 -x569 -x570 -x571 -x572 -x573 -x574 -x575 -x576 -x577 -x578 -x579 -x580 -x581 x582 -x583 -x584 -x585 -x586 -x587 -x588 -x589 -x590 -x591 -x592 -x593 -x594 -x595 -x596 -x597 -x598 -x599 -x600 -x601 -x602 -x603 -x604 -x605 -x606 -x607 -x608 -x609 -x610 x611 -x612 -x613 -x614 -x615 -x616 -x617 -x618 -x619 -x620 -x621 -x622 -x623 -x624 -x625 -x626 -x627 -x628 -x629 -x630 -x631 -x632 -x633 -x634 -x635 -x636 -x637 -x638 -x639 x640 -x641 -x642 -x643 -x644 -x645 -x646 -x647 -x648 -x649 -x650 -x651 -x652 -x653 -x654 -x655 -x656 -x657 -x658 -x659 -x660 -x661 -x662 -x663 -x664 -x665 -x666 -x667 -x668 x669 -x670 -x671 -x672 -x673 -x674 -x675 -x676 -x677 -x678 -x679 -x680 -x681 -x682 -x683 -x684 -x685 -x686 -x687 -x688 -x689 -x690 -x691 -x692 -x693 -x694 -x695 -x696 -x697 x698 -x699 -x700 -x701 -x702 -x703 -x704 -x705 -x706 -x707 -x708 -x709 -x710 -x711 -x712 -x713 -x714 -x715 -x716 -x717 -x718 -x719 -x720 -x721 -x722 -x723 -x724 -x725 -x726 x727 -x728 -x729 -x730 -x731 -x732 -x733 -x734 -x735 -x736 -x737 -x738 -x739 -x740 -x741 -x742 -x743 -x744 -x745 -x746 -x747 -x748 -x749 -x750 -x751 -x752 -x753 -x754 -x755 x756 -x757 -x758 -x759 -x760 -x761 -x762 -x763 -x764 -x765 -x766 -x767 -x768 -x769 -x770 -x771 -x772 -x773 -x774 -x775 -x776 -x777 -x778 -x779 -x780 -x781 -x782 -x783 -x784 x785 -x786 -x787 -x788 -x789 -x790 -x791 -x792 -x793 -x794 -x795 -x796 -x797 -x798 -x799 -x800 -x801 -x802 -x803 -x804 -x805 -x806 -x807 -x808 -x809 -x810 -x811 -x812 -x813 x814 -x815 -x816 -x817 -x818 -x819 -x820 -x821 -x822 -x823 -x824 -x825 -x826 -x827 -x828 -x829 -x830 -x831 -x832 -x833 -x834 -x835 -x836 -x837 -x838 -x839 -x840 -x841 -x842 x843 -x844 -x845 -x846 -x847 -x848 -x849 -x850 -x851 -x852 -x853 -x854 -x855 -x856 -x857 -x858 -x859 -x860 -x861 -x862 -x863 -x864 -x865 -x866 -x867 -x868 -x869 -x870 -x871 x872 -x873 -x874 -x875 -x876 -x877 -x878 -x879 -x880 -x881 -x882 -x883 -x884 -x885 -x886 -x887 -x888 -x889 -x890 -x891 -x892 -x893 -x894 -x895 -x896 -x897 -x898 -x899 -x900 x901 -x902 -x903 -x904 -x905 -x906 -x907 -x908 -x909 -x910 -x911 -x912 -x913 -x914 -x915 -x916 -x917 -x918 -x919 -x920 -x921 -x922 -x923 -x924 -x925 -x926 -x927 -x928 -x929 -x930

Verifier Data 

OK	1

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3728876-1338673990/watcher-3728876-1338673990 -o /tmp/evaluation-result-3728876-1338673990/solver-3728876-1338673990 -C 1800 -W 1900 -M 15500 pwbo2.02 -file-format=wbo -verbosity=0 HOME/instance-3728876-1338673990.wbo 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.99 3.98 3.92 4/179 21359
/proc/meminfo: memFree=27967796/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=162744 CPUtime=0 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 1094 0 0 0 0 0 0 0 20 0 3 0 333523491 166649856 1042 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40686 1042 196 433 0 40250 0
[pid=21359/tid=21360] ppid=21357 vsize=162744 CPUtime=0 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 125 0 0 0 0 0 0 0 20 0 3 0 333523491 166649856 1042 33554432000 4194304 5964478 140734488433664 139950613064392 5136420 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=162744 CPUtime=0 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 99 0 0 0 0 0 0 0 20 0 3 0 333523491 166649856 1042 33554432000 4194304 5964478 140734488433664 139950602574072 4382008 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0

[startup+0.089669 s]
/proc/loadavg: 3.99 3.98 3.92 4/179 21359
/proc/meminfo: memFree=27967796/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=162744 CPUtime=0.17 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 1622 0 0 0 17 0 0 0 20 0 3 0 333523491 166649856 1570 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40686 1570 198 433 0 40250 0
[pid=21359/tid=21360] ppid=21357 vsize=162744 CPUtime=0.07 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 412 0 0 0 7 0 0 0 20 0 3 0 333523491 166649856 1571 33554432000 4194304 5964478 140734488433664 139950613063720 5136420 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=162744 CPUtime=0.07 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 341 0 0 0 7 0 0 0 20 0 3 0 333523491 166649856 1571 33554432000 4194304 5964478 140734488433664 139950602574184 4382029 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 162744

[startup+0.100329 s]
/proc/loadavg: 3.99 3.98 3.92 4/179 21359
/proc/meminfo: memFree=27967796/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=162744 CPUtime=0.19 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 1679 0 0 0 19 0 0 0 20 0 3 0 333523491 166649856 1627 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40686 1628 198 433 0 40250 0
[pid=21359/tid=21360] ppid=21357 vsize=162744 CPUtime=0.08 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 442 0 0 0 8 0 0 0 20 0 3 0 333523491 166649856 1628 33554432000 4194304 5964478 140734488433664 139950613063720 5136490 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=162744 CPUtime=0.08 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 368 0 0 0 8 0 0 0 20 0 3 0 333523491 166649856 1628 33554432000 4194304 5964478 140734488433664 139950602574184 4382029 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 162744

[startup+0.300259 s]
/proc/loadavg: 3.99 3.98 3.92 4/179 21359
/proc/meminfo: memFree=27967796/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=162744 CPUtime=0.58 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 2683 0 0 0 58 0 0 0 20 0 3 0 333523491 166649856 2631 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40686 2631 198 433 0 40250 0
[pid=21359/tid=21360] ppid=21357 vsize=162744 CPUtime=0.28 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 973 0 0 0 28 0 0 0 20 0 3 0 333523491 166649856 2631 33554432000 4194304 5964478 140734488433664 139950613063672 5136420 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=162744 CPUtime=0.28 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 840 0 0 0 28 0 0 0 20 0 3 0 333523491 166649856 2631 33554432000 4194304 5964478 140734488433664 139950602573944 4387996 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 0.58
Current children cumulated vsize (KiB) 162744

[startup+0.700302 s]
/proc/loadavg: 3.99 3.98 3.92 4/179 21359
/proc/meminfo: memFree=27967796/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=162744 CPUtime=1.39 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 4451 0 0 0 138 1 0 0 20 0 3 0 333523491 166649856 4399 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40686 4399 198 433 0 40250 0
[pid=21359/tid=21360] ppid=21357 vsize=162744 CPUtime=0.68 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 1942 0 0 0 68 0 0 0 20 0 3 0 333523491 166649856 4399 33554432000 4194304 5964478 140734488433664 139950613063720 5136420 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=162744 CPUtime=0.68 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 1639 0 0 0 68 0 0 0 20 0 3 0 333523491 166649856 4399 33554432000 4194304 5964478 140734488433664 139950602574136 5136420 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1.39
Current children cumulated vsize (KiB) 162744

[startup+1.5003 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21362
/proc/meminfo: memFree=27944064/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=162888 CPUtime=2.99 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 7372 0 0 0 297 2 0 0 20 0 3 0 333523491 166797312 7320 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40722 7320 198 433 0 40286 0
[pid=21359/tid=21360] ppid=21357 vsize=162888 CPUtime=1.48 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 3565 0 0 0 147 1 0 0 20 0 3 0 333523491 166797312 7320 33554432000 4194304 5964478 140734488433664 139950613063608 4400087 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=162888 CPUtime=1.48 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 2937 0 0 0 148 0 0 0 20 0 3 0 333523491 166797312 7320 33554432000 4194304 5964478 140734488433664 139950602574136 5136420 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 2.99
Current children cumulated vsize (KiB) 162888

[startup+3.1003 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21362
/proc/meminfo: memFree=27932532/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=163164 CPUtime=6.18 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 9092 0 0 0 616 2 0 0 20 0 3 0 333523491 167079936 9040 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40791 9040 198 433 0 40355 0
[pid=21359/tid=21360] ppid=21357 vsize=163164 CPUtime=3.07 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 4455 0 0 0 306 1 0 0 20 0 3 0 333523491 167079936 9040 33554432000 4194304 5964478 140734488433664 139950613063720 5136420 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=163164 CPUtime=3.07 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 3767 0 0 0 307 0 0 0 20 0 3 0 333523491 167079936 9040 33554432000 4194304 5964478 140734488433664 139950602574136 4400087 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 6.18
Current children cumulated vsize (KiB) 163164

[startup+6.30031 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21362
/proc/meminfo: memFree=27912568/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=163308 CPUtime=12.58 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 12414 0 0 0 1254 4 0 0 20 0 3 0 333523491 167227392 12362 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40827 12362 198 433 0 40391 0
[pid=21359/tid=21360] ppid=21357 vsize=163308 CPUtime=6.26 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 6249 0 0 0 624 2 0 0 20 0 3 0 333523491 167227392 12362 33554432000 4194304 5964478 140734488433664 139950613063672 4400097 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=163308 CPUtime=6.26 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 5295 0 0 0 624 2 0 0 20 0 3 0 333523491 167227392 12362 33554432000 4194304 5964478 140734488433664 139950602574136 4400109 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 12.58
Current children cumulated vsize (KiB) 163308

[startup+12.7003 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21362
/proc/meminfo: memFree=27872504/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=163308 CPUtime=25.37 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 19432 0 0 0 2527 10 0 0 20 0 3 0 333523491 167227392 19380 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 40827 19380 198 433 0 40391 0
[pid=21359/tid=21360] ppid=21357 vsize=163308 CPUtime=12.65 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 8780 0 0 0 1260 5 0 0 20 0 3 0 333523491 167227392 19380 33554432000 4194304 5964478 140734488433664 139950613063672 4400091 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=163308 CPUtime=12.65 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 9782 0 0 0 1261 4 0 0 20 0 3 0 333523491 167227392 19380 33554432000 4194304 5964478 140734488433664 139950602573944 4400087 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 25.37
Current children cumulated vsize (KiB) 163308

[startup+25.5003 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21362
/proc/meminfo: memFree=27801832/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=229324 CPUtime=50.97 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 31335 0 0 0 5075 22 0 0 20 0 3 0 333523491 234827776 31283 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 57331 31283 198 433 0 56895 0
[pid=21359/tid=21360] ppid=21357 vsize=229324 CPUtime=25.41 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 17284 0 0 0 2525 16 0 0 20 0 3 0 333523491 234827776 31283 33554432000 4194304 5964478 140734488433664 139950613063672 4399952 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=229324 CPUtime=25.41 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 13181 0 0 0 2536 5 0 0 20 0 3 0 333523491 234827776 31283 33554432000 4194304 5964478 140734488433664 139950602574136 4400091 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 50.97
Current children cumulated vsize (KiB) 229324

[startup+51.1004 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21362
/proc/meminfo: memFree=27729120/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=295736 CPUtime=102.16 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 44305 0 0 0 10183 33 0 0 20 0 3 0 333523491 302833664 44253 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 73934 44253 198 433 0 73498 0
[pid=21359/tid=21360] ppid=21357 vsize=295736 CPUtime=50.95 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 24513 0 0 0 5069 26 0 0 20 0 3 0 333523491 302833664 44253 33554432000 4194304 5964478 140734488433664 139950613063672 4400087 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=295736 CPUtime=50.95 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 18922 0 0 0 5088 7 0 0 20 0 3 0 333523491 302833664 44253 33554432000 4194304 5964478 140734488433664 139950602574184 4400109 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 102.16
Current children cumulated vsize (KiB) 295736

[startup+102.307 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21362
/proc/meminfo: memFree=27640636/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=361792 CPUtime=204.56 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 63665 0 0 0 20405 51 0 0 20 0 3 0 333523491 370475008 63613 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 90448 63613 198 433 0 90012 0
[pid=21359/tid=21360] ppid=21357 vsize=361792 CPUtime=102.02 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 35600 0 0 0 10163 39 0 0 20 0 3 0 333523491 370475008 63613 33554432000 4194304 5964478 140734488433664 139950613063672 4400115 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=361792 CPUtime=102.03 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 27195 0 0 0 10190 13 0 0 20 0 3 0 333523491 370475008 63613 33554432000 4194304 5964478 140734488433664 139950602574136 4400115 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 204.56
Current children cumulated vsize (KiB) 361792

[startup+162.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21362
/proc/meminfo: memFree=27518100/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=427568 CPUtime=324.52 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 82861 0 0 0 32382 70 0 0 20 0 3 0 333523491 437829632 82809 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 106892 82809 198 433 0 106456 0
[pid=21359/tid=21360] ppid=21357 vsize=427568 CPUtime=161.86 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 42874 0 0 0 16135 51 0 0 20 0 3 0 333523491 437829632 82809 33554432000 4194304 5964478 140734488433664 139950613063672 4400103 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=427568 CPUtime=161.87 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 39117 0 0 0 16164 23 0 0 20 0 3 0 333523491 437829632 82809 33554432000 4194304 5964478 140734488433664 139950602574136 5136420 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 324.52
Current children cumulated vsize (KiB) 427568

[startup+222.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21362
/proc/meminfo: memFree=27451916/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=427568 CPUtime=444.5 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 87082 0 0 0 44376 74 0 0 20 0 3 0 333523491 437829632 87030 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 106892 87030 198 433 0 106456 0
[pid=21359/tid=21360] ppid=21357 vsize=427568 CPUtime=221.7 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 44724 0 0 0 22116 54 0 0 20 0 3 0 333523491 437829632 87030 33554432000 4194304 5964478 140734488433664 139950613063672 4400087 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=427568 CPUtime=221.71 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 41488 0 0 0 22145 26 0 0 20 0 3 0 333523491 437829632 87030 33554432000 4194304 5964478 140734488433664 139950602574136 4221120 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 444.5
Current children cumulated vsize (KiB) 427568

################
# More data... #
################

[pid=21359/tid=21361] ppid=21357 vsize=494328 CPUtime=341.4 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 41505 0 0 0 34112 28 0 0 20 0 3 0 333523491 506191872 93394 33554432000 4194304 5964478 140734488433664 139950602574136 4400078 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 684.46
Current children cumulated vsize (KiB) 494328

[startup+402.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21363
/proc/meminfo: memFree=27346796/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=496188 CPUtime=804.42 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 98655 0 0 0 80353 89 0 0 20 0 3 0 333523491 508096512 98791 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 124047 98791 198 433 0 123611 0
[pid=21359/tid=21360] ppid=21357 vsize=496188 CPUtime=401.22 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 56089 0 0 0 40051 71 0 0 20 0 3 0 333523491 508096512 98791 33554432000 4194304 5964478 140734488433664 139950613063720 4388783 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=496188 CPUtime=401.23 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 41696 0 0 0 40094 29 0 0 20 0 3 0 333523491 508096512 98791 33554432000 4194304 5964478 140734488433664 139950602574168 4400087 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 804.42
Current children cumulated vsize (KiB) 496188

[startup+462.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21366
/proc/meminfo: memFree=27327496/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=497484 CPUtime=924.37 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 103736 0 0 0 92346 91 0 0 20 0 3 0 333523491 509423616 103928 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 124371 103928 198 433 0 123935 0
[pid=21359/tid=21360] ppid=21357 vsize=497484 CPUtime=461.03 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 57052 0 0 0 46031 72 0 0 20 0 3 0 333523491 509423616 103928 33554432000 4194304 5964478 140734488433664 139950613063480 4400115 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=497484 CPUtime=461.07 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 45814 0 0 0 46075 32 0 0 20 0 3 0 333523491 509423616 103928 33554432000 4194304 5964478 140734488433664 139950602573944 4400087 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 924.37
Current children cumulated vsize (KiB) 497484

[startup+522.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/182 21430
/proc/meminfo: memFree=27916580/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=501120 CPUtime=1044.32 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 110031 0 0 0 104335 97 0 0 20 0 3 0 333523491 513146880 110171 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 125280 110171 198 433 0 124844 0
[pid=21359/tid=21360] ppid=21357 vsize=501120 CPUtime=520.83 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 61810 0 0 0 52006 77 0 0 20 0 3 0 333523491 513146880 110171 33554432000 4194304 5964478 140734488433664 139950613063672 4220672 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=501120 CPUtime=520.92 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 47351 0 0 0 52059 33 0 0 20 0 3 0 333523491 513146880 110171 33554432000 4194304 5964478 140734488433664 139950602573960 4221120 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1044.32
Current children cumulated vsize (KiB) 501120

[startup+582.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21430
/proc/meminfo: memFree=27699508/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=504004 CPUtime=1164.26 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 114697 0 0 0 116324 102 0 0 20 0 3 0 333523491 516100096 115314 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 126001 115314 198 433 0 125565 0
[pid=21359/tid=21360] ppid=21357 vsize=504004 CPUtime=580.67 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 66328 0 0 0 57983 84 0 0 20 0 3 0 333523491 516100096 115314 33554432000 4194304 5964478 140734488433664 139950613063672 4399920 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=504004 CPUtime=580.74 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 47499 0 0 0 58040 34 0 0 20 0 3 0 333523491 516100096 115314 33554432000 4194304 5964478 140734488433664 139950602574312 4382045 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1164.26
Current children cumulated vsize (KiB) 504004

[startup+642.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21431
/proc/meminfo: memFree=27528948/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=573744 CPUtime=1284.26 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 128392 0 0 0 128306 120 0 0 20 0 3 0 333523491 587513856 129756 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 143436 129756 198 433 0 143000 0
[pid=21359/tid=21360] ppid=21357 vsize=573744 CPUtime=640.52 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 79960 0 0 0 63949 103 0 0 20 0 3 0 333523491 587513856 129757 33554432000 4194304 5964478 140734488433664 139950613063672 5136420 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=573744 CPUtime=640.6 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 47563 0 0 0 64025 35 0 0 20 0 3 0 333523491 587513856 129757 33554432000 4194304 5964478 140734488433664 139950602573896 4400103 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1284.26
Current children cumulated vsize (KiB) 573744

[startup+702.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21431
/proc/meminfo: memFree=27416632/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=639824 CPUtime=1404.23 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 147680 0 0 0 140282 141 0 0 20 0 3 0 333523491 655179776 149044 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 159956 149044 198 433 0 159520 0
[pid=21359/tid=21360] ppid=21357 vsize=639824 CPUtime=700.36 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 99155 0 0 0 69912 124 0 0 20 0 3 0 333523491 655179776 149044 33554432000 4194304 5964478 140734488433664 139950613063672 4221120 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=639824 CPUtime=700.44 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 47655 0 0 0 70008 36 0 0 20 0 3 0 333523491 655179776 149044 33554432000 4194304 5964478 140734488433664 139950602573896 4400076 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1404.23
Current children cumulated vsize (KiB) 639824

[startup+762.307 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21431
/proc/meminfo: memFree=27321056/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=705932 CPUtime=1524.23 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 151777 0 0 0 152276 147 0 0 20 0 3 0 333523491 722874368 153141 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 176483 153141 198 433 0 176047 0
[pid=21359/tid=21360] ppid=21357 vsize=705932 CPUtime=760.21 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 102174 0 0 0 75893 128 0 0 20 0 3 0 333523491 722874368 153141 33554432000 4194304 5964478 140734488433664 139950613063672 4220672 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=705932 CPUtime=760.29 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 48733 0 0 0 75989 40 0 0 20 0 3 0 333523491 722874368 153141 33554432000 4194304 5964478 140734488433664 139950602574136 4400067 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1524.23
Current children cumulated vsize (KiB) 705932

[startup+822.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21431
/proc/meminfo: memFree=27250168/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=705932 CPUtime=1644.19 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 151805 0 0 0 164272 147 0 0 20 0 3 0 333523491 722874368 152955 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 176483 152955 198 433 0 176047 0
[pid=21359/tid=21360] ppid=21357 vsize=705932 CPUtime=820.04 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 102181 0 0 0 81875 129 0 0 20 0 3 0 333523491 722874368 152955 33554432000 4194304 5964478 140734488433664 139950613063672 4400089 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=705932 CPUtime=820.13 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 48754 0 0 0 81972 41 0 0 20 0 3 0 333523491 722874368 152955 33554432000 4194304 5964478 140734488433664 139950602574136 4379311 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1644.19
Current children cumulated vsize (KiB) 705932

[startup+882.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/181 21431
/proc/meminfo: memFree=27168852/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=771656 CPUtime=1764.17 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 155550 0 0 0 176263 154 0 0 20 0 3 0 333523491 790175744 156700 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 192914 156700 198 433 0 192478 0
[pid=21359/tid=21360] ppid=21357 vsize=771656 CPUtime=879.89 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 102189 0 0 0 87859 130 0 0 20 0 3 0 333523491 790175744 156700 33554432000 4194304 5964478 140734488433664 139950613063672 4400083 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=771656 CPUtime=879.98 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 52491 0 0 0 87949 49 0 0 20 0 3 0 333523491 790175744 156700 33554432000 4194304 5964478 140734488433664 139950602574136 4399920 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1764.17
Current children cumulated vsize (KiB) 771656



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+900.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/180 21431
/proc/meminfo: memFree=27156012/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=771656 CPUtime=1800.18 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 157284 0 0 0 179858 160 0 0 20 0 3 0 333523491 790175744 158487 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 192914 158487 198 433 0 192478 0
[pid=21359/tid=21360] ppid=21357 vsize=771656 CPUtime=897.85 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 102192 0 0 0 89655 130 0 0 20 0 3 0 333523491 790175744 158487 33554432000 4194304 5964478 140734488433664 139950613063672 4221120 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=771656 CPUtime=897.94 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 54222 0 0 0 89739 55 0 0 20 0 3 0 333523491 790175744 158487 33554432000 4194304 5964478 140734488433664 139950602574136 4399920 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1800.18
Current children cumulated vsize (KiB) 771656

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+900.3 s]
/proc/loadavg: 3.99 3.98 3.92 5/180 21431
/proc/meminfo: memFree=27156012/32873648 swapFree=7388/7388
[pid=21359] ppid=21357 vsize=771656 CPUtime=1800.18 cores=1,3,5,7
/proc/21359/stat : 21359 (pwbo2.02) S 21357 21359 19426 0 -1 4202496 157284 0 0 0 179858 160 0 0 20 0 3 0 333523491 790175744 158487 33554432000 4194304 5964478 140734488433664 140734488433184 4546685 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/21359/statm: 192914 158487 198 433 0 192478 0
[pid=21359/tid=21360] ppid=21357 vsize=771656 CPUtime=897.85 cores=1,3,5,7
/proc/21359/task/21360/stat : 21360 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 102192 0 0 0 89655 130 0 0 20 0 3 0 333523491 790175744 158487 33554432000 4194304 5964478 140734488433664 139950613063672 4221120 0 0 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=21359/tid=21361] ppid=21357 vsize=771656 CPUtime=897.94 cores=1,3,5,7
/proc/21359/task/21361/stat : 21361 (pwbo2.02) R 21357 21359 19426 0 -1 4202560 54222 0 0 0 89739 55 0 0 20 0 3 0 333523491 790175744 158487 33554432000 4194304 5964478 140734488433664 139950602574136 4399920 0 0 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1800.18
Current children cumulated vsize (KiB) 771656

Child status: 0

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21359 and gives
#  childrusage.ru_utime.tv_sec=1793
#  childrusage.ru_utime.tv_usec=950278
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=890712
# CPU time returned by wait4() is 1795.84
# while last known CPU time is 1800.18
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 900.328
CPU time (s): 1800.18
CPU user time (s): 1798.58
CPU system time (s): 1.6
CPU usage (%): 199.947
Max. virtual memory (cumulated for all children) (KiB): 771656

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1793.95
system time used= 1.89071
maximum resident set size= 633952
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 157285
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 498
involuntary context switches= 1838

runsolver used 1.76973 second user time and 5.2752 second system time

The end

Launcher Data 

Begin job on node148 at 2012-06-02 23:53:10
IDJOB=3728876
IDBENCH=79172
IDSOLVER=2316
FILE ID=node148/3728876-1338673990
RUNJOBID= node148-1338669340-19566
PBS_JOBID= 14636523
Free space on /tmp= 71328 MiB

SOLVER NAME= pwbo 2.02
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB06/roussel/normalized-pigeon-cardinality-31-30--soft-0-100-0.wbo
COMMAND LINE= pwbo2.02 -file-format=wbo -verbosity=0 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3728876-1338673990/watcher-3728876-1338673990 -o /tmp/evaluation-result-3728876-1338673990/solver-3728876-1338673990 -C 1800 -W 1900 -M 15500  pwbo2.02 -file-format=wbo -verbosity=0 HOME/instance-3728876-1338673990.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 168e81c26eea0b6e6155adb2f3f4d195
RANDOM SEED=302014339

node148.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.94
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.284
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        27968424 kB
Buffers:          581892 kB
Cached:          2842180 kB
SwapCached:          628 kB
Active:          1958208 kB
Inactive:        2044036 kB
Active(anon):     567528 kB
Inactive(anon):    10672 kB
Active(file):    1390680 kB
Inactive(file):  2033364 kB
Unevictable:        7388 kB
Mlocked:            7388 kB
SwapTotal:      67108856 kB
SwapFree:       67107564 kB
Dirty:              5964 kB
Writeback:             0 kB
AnonPages:        582020 kB
Mapped:            11592 kB
Shmem:                36 kB
Slab:             766272 kB
SReclaimable:     209588 kB
SUnreclaim:       556684 kB
KernelStack:        1480 kB
PageTables:         4724 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     169752 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:     67584 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 71320 MiB
End job on node148 at 2012-06-03 00:08:12