Trace number 3395026

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40001 5010.92

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m47-q841.5-n45-p15-s16393788.cnf
MD5SUM683c9f75b69575e6451b8563d12f176b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.196969
Satisfiable
(Un)Satisfiability was proved
Number of variables1981
Number of clauses333838
Sum of the clauses size669567
Maximum clause length47
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2333793
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3395026-1306110748/watcher-3395026-1306110748 -o /tmp/evaluation-result-3395026-1306110748/solver-3395026-1306110748 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3395026-1306110748.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 7.99 7.98 1/427 574
/proc/meminfo: memFree=9261132/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=8728 CPUtime=0 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 215 0 0 0 0 0 0 0 24 0 1 0 722674441 8937472 146 33554432000 4194304 4417452 140734241532896 18446744073709551615 274664808192 0 0 4096 0 18446744071562362840 0 0 17 3 0 0 0
/proc/574/statm: 2182 146 122 55 0 68 0

[startup+0.074544 s]
/proc/loadavg: 8.04 7.99 7.98 1/427 574
/proc/meminfo: memFree=9261132/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=18356 CPUtime=0.06 cores=0-7
/proc/574/stat : 574 (plingeling) R 572 574 411 0 -1 4202496 2801 1250 0 0 6 0 0 0 24 0 1 0 722674441 18796544 2612 33554432000 4194304 4417452 140734241532896 18446744073709551615 4231377 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/574/statm: 4589 2612 148 55 0 2475 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 18356

[startup+0.100537 s]
/proc/loadavg: 8.04 7.99 7.98 1/427 574
/proc/meminfo: memFree=9261132/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=26548 CPUtime=0.08 cores=0-7
/proc/574/stat : 574 (plingeling) R 572 574 411 0 -1 4202496 4849 1250 0 0 8 0 0 0 24 0 1 0 722674441 27185152 4660 33554432000 4194304 4417452 140734241532896 18446744073709551615 4231277 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/574/statm: 6637 4660 148 55 0 4523 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26548

[startup+0.300516 s]
/proc/loadavg: 8.04 7.99 7.98 1/427 574
/proc/meminfo: memFree=9261132/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=71604 CPUtime=0.29 cores=0-7
/proc/574/stat : 574 (plingeling) R 572 574 411 0 -1 4202496 16113 1250 0 0 27 2 0 0 25 0 1 0 722674441 73322496 15924 33554432000 4194304 4417452 140734241532896 18446744073709551615 274664866218 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/574/statm: 17901 15924 148 55 0 15787 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 71604

[startup+0.700401 s]
/proc/loadavg: 8.04 7.99 7.98 1/427 574
/proc/meminfo: memFree=9261132/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=241040 CPUtime=2.46 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 23371 1250 0 0 242 4 0 0 25 0 9 0 722674441 246824960 21456 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 60260 21456 176 55 0 58146 0
Current children cumulated CPU time (s) 2.46
Current children cumulated vsize (KiB) 241040

[startup+1.50126 s]
/proc/loadavg: 8.04 7.99 7.98 9/436 588
/proc/meminfo: memFree=9174280/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=259756 CPUtime=8.83 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 41360 1250 0 0 877 6 0 0 25 0 9 0 722674441 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 64939 26334 180 55 0 62825 0
[pid=574/tid=581] ppid=572 vsize=259756 CPUtime=1.09 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 2588 1250 0 0 109 0 0 0 25 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4216603 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=259756 CPUtime=1.09 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 3129 1250 0 0 109 0 0 0 18 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4235981 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=583] ppid=572 vsize=259756 CPUtime=0.97 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 404 1250 0 0 97 0 0 0 25 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4258379 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=584] ppid=572 vsize=259756 CPUtime=1.06 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 7284 1250 0 0 105 1 0 0 25 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4250807 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=259756 CPUtime=0.94 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2523 1250 0 0 94 0 0 0 25 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=259756 CPUtime=1.08 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 1702 1250 0 0 108 0 0 0 25 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249151 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=259756 CPUtime=1.08 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 3361 1250 0 0 108 0 0 0 25 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249526 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=574/tid=588] ppid=572 vsize=259756 CPUtime=1.06 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 3215 1250 0 0 106 0 0 0 18 0 9 0 722674482 265990144 26334 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 8.83
Current children cumulated vsize (KiB) 259756

[startup+3.10098 s]
/proc/loadavg: 8.04 7.99 7.98 9/436 588
/proc/meminfo: memFree=9154360/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=277740 CPUtime=21.62 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 48808 1250 0 0 2155 7 0 0 25 0 9 0 722674441 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 69435 30408 180 55 0 67321 0
[pid=574/tid=581] ppid=572 vsize=277740 CPUtime=2.69 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 2588 1250 0 0 269 0 0 0 25 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4236311 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=277740 CPUtime=2.69 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 3129 1250 0 0 269 0 0 0 21 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4211469 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=583] ppid=572 vsize=277740 CPUtime=2.57 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 404 1250 0 0 257 0 0 0 25 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4258472 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=584] ppid=572 vsize=277740 CPUtime=2.65 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 14732 1250 0 0 263 2 0 0 25 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=277740 CPUtime=2.54 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2523 1250 0 0 254 0 0 0 25 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4258627 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=277740 CPUtime=2.68 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 1702 1250 0 0 268 0 0 0 25 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=277740 CPUtime=2.68 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 3361 1250 0 0 268 0 0 0 25 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4377419 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=574/tid=588] ppid=572 vsize=277740 CPUtime=2.66 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 3215 1250 0 0 266 0 0 0 22 0 9 0 722674482 284405760 30408 33554432000 4194304 4417452 140734241532896 18446744073709551615 4248895 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 21.62
Current children cumulated vsize (KiB) 277740

[startup+6.30043 s]
/proc/loadavg: 8.04 7.99 7.98 9/436 588
/proc/meminfo: memFree=9111140/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=309188 CPUtime=46.29 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 66472 1250 0 0 4617 12 0 0 25 0 9 0 722674441 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 77297 38692 180 55 0 75183 0
[pid=574/tid=581] ppid=572 vsize=309188 CPUtime=5.89 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 2588 1250 0 0 589 0 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249991 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=309188 CPUtime=5.89 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 3129 1250 0 0 589 0 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4236294 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=583] ppid=572 vsize=309188 CPUtime=5.77 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 681 1250 0 0 577 0 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364996 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=584] ppid=572 vsize=309188 CPUtime=5.72 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 31829 1250 0 0 568 4 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4251133 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=309188 CPUtime=5.74 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2523 1250 0 0 574 0 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4236380 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=309188 CPUtime=5.88 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 1702 1250 0 0 588 0 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=309188 CPUtime=5.87 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 3651 1250 0 0 587 0 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=574/tid=588] ppid=572 vsize=309188 CPUtime=5.08 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 3215 1250 0 0 505 3 0 0 25 0 9 0 722674482 316608512 38692 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 46.29
Current children cumulated vsize (KiB) 309188

[startup+12.7013 s]
/proc/loadavg: 8.03 7.99 7.98 9/436 588
/proc/meminfo: memFree=9077708/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=344004 CPUtime=97.4 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 86478 1250 0 0 9722 18 0 0 25 0 9 0 722674441 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 86001 47422 180 55 0 83887 0
[pid=574/tid=581] ppid=572 vsize=344004 CPUtime=12.29 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 3601 1250 0 0 1229 0 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249597 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=344004 CPUtime=12.29 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 3129 1250 0 0 1229 0 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=583] ppid=572 vsize=344004 CPUtime=12.17 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 1264 1250 0 0 1217 0 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4366102 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=584] ppid=572 vsize=344004 CPUtime=12.03 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 48853 1250 0 0 1196 7 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4320854 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=344004 CPUtime=12.14 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 3536 1250 0 0 1214 0 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4236017 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=344004 CPUtime=12.28 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 2075 1250 0 0 1228 0 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=344004 CPUtime=12.27 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 3651 1250 0 0 1227 0 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364693 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=574/tid=588] ppid=572 vsize=344004 CPUtime=11.47 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 3215 1250 0 0 1142 5 0 0 25 0 9 0 722674482 352260096 47422 33554432000 4194304 4417452 140734241532896 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 97.4
Current children cumulated vsize (KiB) 344004

[startup+25.5011 s]
/proc/loadavg: 8.02 7.99 7.98 9/436 589
/proc/meminfo: memFree=8939420/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=473688 CPUtime=199.57 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 153319 1250 0 0 19927 30 0 0 25 0 9 0 722674441 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 118422 80238 180 55 0 116308 0
[pid=574/tid=581] ppid=572 vsize=473688 CPUtime=25.08 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 11122 1250 0 0 2507 1 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=473688 CPUtime=25.08 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 5936 1250 0 0 2508 0 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4365572 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=583] ppid=572 vsize=473688 CPUtime=24.97 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 7480 1250 0 0 2496 1 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=584] ppid=572 vsize=473688 CPUtime=24.66 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 86889 1250 0 0 2453 13 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=473688 CPUtime=24.93 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 7270 1250 0 0 2492 1 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4237371 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=473688 CPUtime=25.07 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 4855 1250 0 0 2507 0 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4356867 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=473688 CPUtime=25.06 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 7553 1250 0 0 2506 0 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4251081 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=574/tid=588] ppid=572 vsize=473688 CPUtime=24.25 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 5060 1250 0 0 2418 7 0 0 25 0 9 0 722674482 485056512 80238 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364597 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 199.57
Current children cumulated vsize (KiB) 473688

[startup+51.1007 s]
/proc/loadavg: 8.08 8.01 7.99 9/436 589
/proc/meminfo: memFree=8768792/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=673400 CPUtime=403.97 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 240374 1250 0 0 40352 45 0 0 25 0 9 0 722674441 689561600 129686 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 168350 129686 180 55 0 166236 0
[pid=574/tid=581] ppid=572 vsize=673400 CPUtime=50.68 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 14268 1250 0 0 5066 2 0 0 25 0 9 0 722674482 689561600 129686 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=673400 CPUtime=50.68 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 5963 1250 0 0 5067 1 0 0 25 0 9 0 722674482 689561600 129686 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=583] ppid=572 vsize=673400 CPUtime=50.56 cores=0-7

################
# More data... #
################

/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 1649573 1250 0 0 471543 396 0 0 25 0 9 0 722674482 11988680704 2888226 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37700.7
Current children cumulated vsize (KiB) 11707696

[startup+4782.3 s]
/proc/loadavg: 8.09 8.08 8.03 9/436 737
/proc/meminfo: memFree=226208/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=11885688 CPUtime=38179.6 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 14804011 1250 0 0 3814719 3245 0 0 25 0 9 0 722674441 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 2971422 2932724 180 55 0 2969308 0
[pid=574/tid=581] ppid=572 vsize=11885688 CPUtime=4781.6 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 1907178 1250 0 0 477681 479 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=11885688 CPUtime=4781.7 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 2210672 1250 0 0 477757 413 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=583] ppid=572 vsize=11885688 CPUtime=4781.51 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 1402356 1250 0 0 477889 262 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=584] ppid=572 vsize=11885688 CPUtime=4712.81 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 1726024 1250 0 0 470854 427 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=11885688 CPUtime=4779.83 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2032303 1250 0 0 477497 486 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=11885688 CPUtime=4781.11 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 2079316 1250 0 0 477748 363 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=11885688 CPUtime=4781.25 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 1779435 1250 0 0 477712 413 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=574/tid=588] ppid=572 vsize=11885688 CPUtime=4779.36 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 1649573 1250 0 0 477540 396 0 0 25 0 9 0 722674482 12170944512 2932724 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38179.6
Current children cumulated vsize (KiB) 11885688

[startup+4842.3 s]
/proc/loadavg: 8.06 8.07 8.02 9/436 739
/proc/meminfo: memFree=146968/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=11965188 CPUtime=38658.7 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 14856654 1250 0 0 3862621 3252 0 0 25 0 9 0 722674441 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 2991297 2952599 180 55 0 2989183 0
[pid=574/tid=581] ppid=572 vsize=11965188 CPUtime=4841.6 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 1907178 1250 0 0 483681 479 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=11965188 CPUtime=4841.71 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 2246931 1250 0 0 483753 418 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=583] ppid=572 vsize=11965188 CPUtime=4841.51 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 1402356 1250 0 0 483889 262 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=584] ppid=572 vsize=11965188 CPUtime=4771.93 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 1726024 1250 0 0 476766 427 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=585] ppid=572 vsize=11965188 CPUtime=4839.81 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2032303 1250 0 0 483495 486 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=11965188 CPUtime=4841.11 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 2095700 1250 0 0 483746 365 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=11965188 CPUtime=4841.22 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 1779435 1250 0 0 483709 413 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=574/tid=588] ppid=572 vsize=11965188 CPUtime=4839.36 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 1649573 1250 0 0 483540 396 0 0 25 0 9 0 722674482 12252352512 2952599 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38658.7
Current children cumulated vsize (KiB) 11965188

[startup+4902.3 s]
/proc/loadavg: 8.02 8.06 8.02 9/436 741
/proc/meminfo: memFree=228900/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=12379016 CPUtime=39137.6 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 15009263 1250 0 0 3910482 3280 0 0 25 0 9 0 722674441 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 3094754 3056056 180 55 0 3092640 0
[pid=574/tid=581] ppid=572 vsize=12379016 CPUtime=4901.53 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 1907178 1250 0 0 489674 479 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4263080 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=12379016 CPUtime=4901.21 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 2263315 1250 0 0 489701 420 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=583] ppid=572 vsize=12379016 CPUtime=4901.51 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 1435124 1250 0 0 489885 266 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=584] ppid=572 vsize=12379016 CPUtime=4831.44 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 1763945 1250 0 0 482711 433 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=585] ppid=572 vsize=12379016 CPUtime=4899.81 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2065071 1250 0 0 489484 497 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=12379016 CPUtime=4901.11 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 2095700 1250 0 0 489746 365 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=12379016 CPUtime=4901.2 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 1779435 1250 0 0 489706 414 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364861 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=574/tid=588] ppid=572 vsize=12379016 CPUtime=4899.35 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 1682341 1250 0 0 489534 401 0 0 25 0 9 0 722674482 12676112384 3056056 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39137.6
Current children cumulated vsize (KiB) 12379016

[startup+4962.3 s]
/proc/loadavg: 8.12 8.07 8.02 9/436 742
/proc/meminfo: memFree=223428/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=12709320 CPUtime=39616.3 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 15278726 1250 0 0 3958276 3352 0 0 25 0 9 0 722674441 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 3177330 3138632 180 55 0 3175216 0
[pid=574/tid=581] ppid=572 vsize=12709320 CPUtime=4961.4 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 1907178 1250 0 0 495661 479 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=582] ppid=572 vsize=12709320 CPUtime=4960.33 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 2312199 1250 0 0 495600 433 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=583] ppid=572 vsize=12709320 CPUtime=4961.33 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 1435124 1250 0 0 495867 266 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=584] ppid=572 vsize=12709320 CPUtime=4891.32 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 1763945 1250 0 0 488699 433 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=585] ppid=572 vsize=12709320 CPUtime=4959.8 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2065071 1250 0 0 495483 497 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=586] ppid=572 vsize=12709320 CPUtime=4961.11 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 2267127 1250 0 0 495701 410 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=12709320 CPUtime=4961.17 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 1812203 1250 0 0 495692 425 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=574/tid=588] ppid=572 vsize=12709320 CPUtime=4959.35 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 1698725 1250 0 0 495532 403 0 0 25 0 9 0 722674482 13014343680 3138632 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39616.3
Current children cumulated vsize (KiB) 12709320



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.4 s]
/proc/loadavg: 8.05 8.06 8.02 9/436 743
/proc/meminfo: memFree=292380/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=12727576 CPUtime=40000.3 cores=0-7
/proc/574/stat : 574 (plingeling) S 572 574 411 0 -1 4202496 15348826 1250 0 0 3996655 3377 0 0 25 0 9 0 722674441 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 3181894 3143196 180 55 0 3179780 0
[pid=574/tid=581] ppid=572 vsize=12727576 CPUtime=5009.47 cores=0-7
/proc/574/task/581/stat : 581 (plingeling) R 572 574 411 0 -1 4202560 1907178 1250 0 0 500468 479 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=574/tid=582] ppid=572 vsize=12727576 CPUtime=5008.01 cores=0-7
/proc/574/task/582/stat : 582 (plingeling) R 572 574 411 0 -1 4202560 2312199 1250 0 0 500368 433 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=574/tid=583] ppid=572 vsize=12727576 CPUtime=5009.09 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4202560 1435124 1250 0 0 500643 266 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=574/tid=584] ppid=572 vsize=12727576 CPUtime=4939.46 cores=0-7
/proc/574/task/584/stat : 584 (plingeling) R 572 574 411 0 -1 4202560 1784893 1250 0 0 493506 440 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=574/tid=585] ppid=572 vsize=12727576 CPUtime=5007.91 cores=0-7
/proc/574/task/585/stat : 585 (plingeling) R 572 574 411 0 -1 4202560 2065071 1250 0 0 500294 497 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=574/tid=586] ppid=572 vsize=12727576 CPUtime=5009.24 cores=0-7
/proc/574/task/586/stat : 586 (plingeling) R 572 574 411 0 -1 4202560 2299895 1250 0 0 500500 424 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=574/tid=587] ppid=572 vsize=12727576 CPUtime=5009.29 cores=0-7
/proc/574/task/587/stat : 587 (plingeling) R 572 574 411 0 -1 4202560 1812203 1250 0 0 500503 426 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364794 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=574/tid=588] ppid=572 vsize=12727576 CPUtime=5007.47 cores=0-7
/proc/574/task/588/stat : 588 (plingeling) R 572 574 411 0 -1 4202560 1715109 1250 0 0 500341 406 0 0 25 0 9 0 722674482 13033037824 3143196 33554432000 4194304 4417452 140734241532896 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 12727576

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010.6 s]
/proc/loadavg: 8.05 8.06 8.02 9/436 743
/proc/meminfo: memFree=292380/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=0 CPUtime=40000.6 cores=0-7
/proc/574/stat : 574 (plingeling) Z 572 574 411 0 -1 4203548 15348826 1250 0 0 3996669 3393 0 0 18 0 2 0 722674441 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 0 0 0 0 0 0 0
[pid=574/tid=583] ppid=572 vsize=0 CPUtime=5009.27 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4203588 1435124 1250 0 0 500644 283 0 0 25 0 2 0 722674482 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 0

[startup+5010.8 s]
/proc/loadavg: 8.05 8.06 8.02 9/436 743
/proc/meminfo: memFree=292380/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=0 CPUtime=40000.8 cores=0-7
/proc/574/stat : 574 (plingeling) Z 572 574 411 0 -1 4203548 15348826 1250 0 0 3996669 3413 0 0 18 0 2 0 722674441 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 0 0 0 0 0 0 0
[pid=574/tid=583] ppid=572 vsize=0 CPUtime=5009.47 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4203588 1435124 1250 0 0 500644 303 0 0 25 0 2 0 722674482 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 0

[startup+5010.9 s]
/proc/loadavg: 8.05 8.06 8.02 9/436 743
/proc/meminfo: memFree=292380/32951132 swapFree=67093840/67111528
[pid=574] ppid=572 vsize=0 CPUtime=40000.9 cores=0-7
/proc/574/stat : 574 (plingeling) Z 572 574 411 0 -1 4203548 15348826 1250 0 0 3996669 3423 0 0 18 0 2 0 722674441 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/574/statm: 0 0 0 0 0 0 0
[pid=574/tid=583] ppid=572 vsize=0 CPUtime=5009.57 cores=0-7
/proc/574/task/583/stat : 583 (plingeling) R 572 574 411 0 -1 4203588 1435124 1250 0 0 500644 313 0 0 25 0 2 0 722674482 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5010.92
CPU time (s): 40001
CPU user time (s): 39966.7
CPU system time (s): 34.2548
CPU usage (%): 798.276
Max. virtual memory (cumulated for all children) (KiB): 12924184

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39966.7
system time used= 34.2548
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 15350076
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 470
involuntary context switches= 193545

runsolver used 20.4409 second user time and 52.0691 second system time

The end

Launcher Data 

Begin job on node131 at 2011-05-23 02:32:28
IDJOB=3395026
IDBENCH=88221
IDSOLVER=1851
FILE ID=node131/3395026-1306110748
RUNJOBID= node131-1306110506-425
PBS_JOBID= 13388048
Free space on /tmp= 72976 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m47-q841.5-n45-p15-s16393788.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3395026-1306110748/watcher-3395026-1306110748 -o /tmp/evaluation-result-3395026-1306110748/solver-3395026-1306110748 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3395026-1306110748.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 683c9f75b69575e6451b8563d12f176b
RANDOM SEED=163096874

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:       9261536 kB
Buffers:        336032 kB
Cached:        8934180 kB
SwapCached:       1504 kB
Active:       17672988 kB
Inactive:      5709776 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:       9261536 kB
SwapTotal:    67111528 kB
SwapFree:     67093840 kB
Dirty:            4296 kB
Writeback:           0 kB
AnonPages:    14111764 kB
Mapped:          37896 kB
Slab:           184480 kB
PageTables:      55968 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 95322552 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72976 MiB
End job on node131 at 2011-05-23 03:56:00