Trace number 3394084

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.6 5012.21

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v1820c171155gyes3.cnf
MD5SUMcd19d9eb9be97e183fa1a1d47d679d01
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1820
Number of clauses171155
Sum of the clauses size344000
Maximum clause length28
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2171090
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 565

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394084-1306086488/watcher-3394084-1306086488 -o /tmp/evaluation-result-3394084-1306086488/solver-3394084-1306086488 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394084-1306086488.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.49 7.92 7.97 1/543 22004
/proc/meminfo: memFree=13579360/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=12136 CPUtime=0.01 cores=0-7
/proc/22004/stat : 22004 (plingeling) R 22002 22004 21978 0 -1 4202496 1111 1252 0 0 1 0 0 0 25 0 1 0 720249202 12427264 1041 33554432000 4194304 4417452 140733985555024 18446744073709551615 253640859392 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22004/statm: 3034 1041 148 55 0 920 0

[startup+0.0364461 s]
/proc/loadavg: 7.49 7.92 7.97 1/543 22004
/proc/meminfo: memFree=13579360/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=14444 CPUtime=0.02 cores=0-7
/proc/22004/stat : 22004 (plingeling) R 22002 22004 21978 0 -1 4202496 1688 1252 0 0 2 0 0 0 25 0 1 0 720249202 14790656 1618 33554432000 4194304 4417452 140733985555024 18446744073709551615 253640859392 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22004/statm: 3611 1618 148 55 0 1497 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 14444

[startup+0.100411 s]
/proc/loadavg: 7.49 7.92 7.97 1/543 22004
/proc/meminfo: memFree=13579360/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=26732 CPUtime=0.08 cores=0-7
/proc/22004/stat : 22004 (plingeling) R 22002 22004 21978 0 -1 4202496 4760 1252 0 0 8 0 0 0 25 0 1 0 720249202 27373568 4690 33554432000 4194304 4417452 140733985555024 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22004/statm: 6683 4690 148 55 0 4569 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26732

[startup+0.319332 s]
/proc/loadavg: 7.49 7.92 7.97 1/543 22004
/proc/meminfo: memFree=13579360/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=192892 CPUtime=1.02 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 9769 1252 0 0 101 1 0 0 25 0 9 0 720249202 197521408 9451 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 48223 9451 176 55 0 46109 0
Current children cumulated CPU time (s) 1.02
Current children cumulated vsize (KiB) 192892

[startup+0.700259 s]
/proc/loadavg: 7.49 7.92 7.97 1/543 22004
/proc/meminfo: memFree=13579360/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=217088 CPUtime=4.06 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 23254 1252 0 0 402 4 0 0 25 0 9 0 720249202 222298112 15614 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 54272 15614 180 55 0 52158 0
Current children cumulated CPU time (s) 4.06
Current children cumulated vsize (KiB) 217088

[startup+1.50111 s]
/proc/loadavg: 7.49 7.92 7.97 9/552 22018
/proc/meminfo: memFree=13515312/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=220148 CPUtime=10.45 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 29168 1252 0 0 1040 5 0 0 25 0 9 0 720249202 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 55037 16379 180 55 0 52923 0
[pid=22004/tid=22011] ppid=22002 vsize=220148 CPUtime=1.28 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 1817 1252 0 0 128 0 0 0 19 0 9 0 720249223 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4258525 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=220148 CPUtime=1.28 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 2334 1252 0 0 128 0 0 0 25 0 9 0 720249223 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4258635 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=220148 CPUtime=1.27 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 1545 1252 0 0 127 0 0 0 25 0 9 0 720249223 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=220148 CPUtime=1.28 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 9337 1252 0 0 127 1 0 0 25 0 9 0 720249223 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=220148 CPUtime=1.27 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 507 1252 0 0 127 0 0 0 25 0 9 0 720249223 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=220148 CPUtime=1.25 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 1837 1252 0 0 125 0 0 0 19 0 9 0 720249223 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=220148 CPUtime=1.28 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 1986 1252 0 0 128 0 0 0 25 0 9 0 720249224 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4376531 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=220148 CPUtime=1.28 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 934 1252 0 0 128 0 0 0 25 0 9 0 720249224 225431552 16379 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 10.45
Current children cumulated vsize (KiB) 220148

[startup+3.10082 s]
/proc/loadavg: 7.49 7.92 7.97 9/552 22018
/proc/meminfo: memFree=13499812/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=246164 CPUtime=23.22 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 41141 1252 0 0 2315 7 0 0 25 0 9 0 720249202 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 61541 22971 180 55 0 59427 0
[pid=22004/tid=22011] ppid=22002 vsize=246164 CPUtime=2.88 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 2784 1252 0 0 288 0 0 0 24 0 9 0 720249223 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249512 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=246164 CPUtime=2.87 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 3301 1252 0 0 287 0 0 0 25 0 9 0 720249223 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=246164 CPUtime=2.87 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 1553 1252 0 0 287 0 0 0 25 0 9 0 720249223 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=246164 CPUtime=2.88 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 16539 1252 0 0 285 3 0 0 25 0 9 0 720249223 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=246164 CPUtime=2.87 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 1450 1252 0 0 287 0 0 0 25 0 9 0 720249223 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=246164 CPUtime=2.83 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 2780 1252 0 0 283 0 0 0 23 0 9 0 720249223 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=246164 CPUtime=2.87 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 1986 1252 0 0 287 0 0 0 25 0 9 0 720249224 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4264926 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=246164 CPUtime=2.87 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 1877 1252 0 0 287 0 0 0 25 0 9 0 720249224 252071936 22971 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 23.22
Current children cumulated vsize (KiB) 246164

[startup+6.30124 s]
/proc/loadavg: 7.53 7.92 7.97 9/552 22018
/proc/meminfo: memFree=13485804/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=262464 CPUtime=48.76 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 55540 1252 0 0 4867 9 0 0 25 0 9 0 720249202 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 65616 27014 180 55 0 63502 0
[pid=22004/tid=22011] ppid=22002 vsize=262464 CPUtime=6.08 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 2784 1252 0 0 608 0 0 0 25 0 9 0 720249223 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4250013 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=262464 CPUtime=6.07 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 3301 1252 0 0 607 0 0 0 25 0 9 0 720249223 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249583 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=262464 CPUtime=6.07 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 1912 1252 0 0 607 0 0 0 25 0 9 0 720249223 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4277780 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=262464 CPUtime=6.09 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 30579 1252 0 0 603 6 0 0 25 0 9 0 720249223 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4358218 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=262464 CPUtime=6.07 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 1450 1252 0 0 607 0 0 0 25 0 9 0 720249223 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249193 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=262464 CPUtime=5.97 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 2780 1252 0 0 597 0 0 0 25 0 9 0 720249223 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4258379 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=262464 CPUtime=6.08 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 1986 1252 0 0 608 0 0 0 25 0 9 0 720249224 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4275934 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=262464 CPUtime=6.07 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 1877 1252 0 0 607 0 0 0 25 0 9 0 720249224 268763136 27014 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 48.76
Current children cumulated vsize (KiB) 262464

[startup+12.7011 s]
/proc/loadavg: 7.57 7.92 7.97 9/552 22018
/proc/meminfo: memFree=13430052/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=308440 CPUtime=99.84 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 89304 1252 0 0 9968 16 0 0 25 0 9 0 720249202 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 77110 38508 180 55 0 74996 0
[pid=22004/tid=22011] ppid=22002 vsize=308440 CPUtime=12.48 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 3727 1252 0 0 1248 0 0 0 25 0 9 0 720249223 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4336297 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=308440 CPUtime=12.47 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 4244 1252 0 0 1247 0 0 0 25 0 9 0 720249223 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=308440 CPUtime=12.47 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 3813 1252 0 0 1247 0 0 0 25 0 9 0 720249223 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4259612 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=308440 CPUtime=12.48 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 56082 1252 0 0 1237 11 0 0 25 0 9 0 720249223 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4357566 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=308440 CPUtime=12.46 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 2393 1252 0 0 1246 0 0 0 25 0 9 0 720249223 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=308440 CPUtime=12.25 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 3723 1252 0 0 1225 0 0 0 25 0 9 0 720249223 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4236244 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=308440 CPUtime=12.47 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 3631 1252 0 0 1247 0 0 0 25 0 9 0 720249224 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=308440 CPUtime=12.47 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 2820 1252 0 0 1247 0 0 0 25 0 9 0 720249224 315842560 38508 33554432000 4194304 4417452 140733985555024 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 99.84
Current children cumulated vsize (KiB) 308440

[startup+25.5007 s]
/proc/loadavg: 7.66 7.93 7.97 9/552 22019
/proc/meminfo: memFree=13362900/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=370440 CPUtime=201.99 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 129506 1252 0 0 20176 23 0 0 25 0 9 0 720249202 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 92610 54229 180 55 0 90496 0
[pid=22004/tid=22011] ppid=22002 vsize=370440 CPUtime=25.28 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 4241 1252 0 0 2528 0 0 0 25 0 9 0 720249223 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4359075 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=370440 CPUtime=25.27 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 4705 1252 0 0 2527 0 0 0 25 0 9 0 720249223 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=370440 CPUtime=25.27 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 8230 1252 0 0 2526 1 0 0 25 0 9 0 720249223 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4236559 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=370440 CPUtime=25.28 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 84385 1252 0 0 2512 16 0 0 25 0 9 0 720249223 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4357146 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=370440 CPUtime=25.26 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 2393 1252 0 0 2526 0 0 0 25 0 9 0 720249223 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=370440 CPUtime=24.81 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 5014 1252 0 0 2481 0 0 0 25 0 9 0 720249223 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4232559 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=370440 CPUtime=25.27 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 7723 1252 0 0 2526 1 0 0 25 0 9 0 720249224 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4356971 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=370440 CPUtime=25.27 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 3944 1252 0 0 2527 0 0 0 25 0 9 0 720249224 379330560 54229 33554432000 4194304 4417452 140733985555024 18446744073709551615 4250513 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 201.99
Current children cumulated vsize (KiB) 370440

[startup+51.1011 s]
/proc/loadavg: 7.78 7.93 7.97 9/552 22019
/proc/meminfo: memFree=13382528/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=358220 CPUtime=406.32 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 180909 1252 0 0 40600 32 0 0 25 0 9 0 720249202 366817280 50937 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 89555 50937 180 55 0 87441 0
[pid=22004/tid=22011] ppid=22002 vsize=358220 CPUtime=50.88 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 5080 1252 0 0 5088 0 0 0 25 0 9 0 720249223 366817280 50937 33554432000 4194304 4417452 140733985555024 18446744073709551615 4258176 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=358220 CPUtime=50.88 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 6525 1252 0 0 5087 1 0 0 25 0 9 0 720249223 366817280 50937 33554432000 4194304 4417452 140733985555024 18446744073709551615 4248966 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=358220 CPUtime=50.87 cores=0-7

################
# More data... #
################


[startup+4722.32 s]
/proc/loadavg: 8.15 8.12 8.03 9/552 22168
/proc/meminfo: memFree=7570020/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=6155080 CPUtime=37688.1 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 5787438 1252 0 0 3767736 1078 0 0 25 0 9 0 720249202 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 1538770 1500152 180 55 0 1536656 0
[pid=22004/tid=22011] ppid=22002 vsize=6155080 CPUtime=4708.02 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 566921 1252 0 0 470698 104 0 0 25 0 9 0 720249223 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=6155080 CPUtime=4722.02 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 708237 1252 0 0 472070 132 0 0 25 0 9 0 720249223 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=6155080 CPUtime=4719.65 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 543149 1252 0 0 471874 91 0 0 25 0 9 0 720249223 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=6155080 CPUtime=4722.22 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 1289997 1252 0 0 471933 289 0 0 25 0 9 0 720249223 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=6155080 CPUtime=4720.51 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 569526 1252 0 0 471949 102 0 0 25 0 9 0 720249223 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=6155080 CPUtime=4668.74 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 819127 1252 0 0 466738 136 0 0 25 0 9 0 720249223 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4343771 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=6155080 CPUtime=4720.79 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 810555 1252 0 0 471937 142 0 0 25 0 9 0 720249224 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=6155080 CPUtime=4705.88 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 471055 1252 0 0 470512 76 0 0 25 0 9 0 720249224 6302801920 1500152 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364809 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37688.1
Current children cumulated vsize (KiB) 6155080

[startup+4782.3 s]
/proc/loadavg: 8.13 8.12 8.03 9/552 22169
/proc/meminfo: memFree=7357228/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=6367284 CPUtime=38166.9 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 5848650 1252 0 0 3815594 1094 0 0 25 0 9 0 720249202 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 1591821 1553172 180 55 0 1589707 0
[pid=22004/tid=22011] ppid=22002 vsize=6367284 CPUtime=4768.01 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 566921 1252 0 0 476697 104 0 0 25 0 9 0 720249223 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=6367284 CPUtime=4782 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 708237 1252 0 0 478068 132 0 0 25 0 9 0 720249223 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=6367284 CPUtime=4779.64 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 547803 1252 0 0 477872 92 0 0 25 0 9 0 720249223 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=6367284 CPUtime=4782.21 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 1339149 1252 0 0 477917 304 0 0 25 0 9 0 720249223 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364675 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=6367284 CPUtime=4780.49 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 569526 1252 0 0 477947 102 0 0 25 0 9 0 720249223 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4354332 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=6367284 CPUtime=4727.63 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 819127 1252 0 0 472627 136 0 0 25 0 9 0 720249223 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=6367284 CPUtime=4780.75 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 810555 1252 0 0 477933 142 0 0 25 0 9 0 720249224 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=6367284 CPUtime=4765.87 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 478461 1252 0 0 476510 77 0 0 25 0 9 0 720249224 6520098816 1553172 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38166.9
Current children cumulated vsize (KiB) 6367284

[startup+4842.3 s]
/proc/loadavg: 8.11 8.11 8.03 9/552 22171
/proc/meminfo: memFree=7831640/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=5894712 CPUtime=38645.8 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 6020255 1252 0 0 3863449 1129 0 0 25 0 9 0 720249202 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 1473678 1435056 180 55 0 1471564 0
[pid=22004/tid=22011] ppid=22002 vsize=5894712 CPUtime=4828.01 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 575113 1252 0 0 482694 107 0 0 25 0 9 0 720249223 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4263109 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=5894712 CPUtime=4842 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 805413 1252 0 0 484047 153 0 0 25 0 9 0 720249223 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=5894712 CPUtime=4839.64 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 552475 1252 0 0 483872 92 0 0 25 0 9 0 720249223 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=5894712 CPUtime=4842.21 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 1339149 1252 0 0 483917 304 0 0 25 0 9 0 720249223 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=5894712 CPUtime=4840.5 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 593452 1252 0 0 483944 106 0 0 25 0 9 0 720249223 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4357753 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=5894712 CPUtime=4786.53 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 831971 1252 0 0 478513 140 0 0 25 0 9 0 720249223 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=5894712 CPUtime=4840.74 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 810555 1252 0 0 483932 142 0 0 25 0 9 0 720249224 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=5894712 CPUtime=4825.87 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 503256 1252 0 0 482507 80 0 0 25 0 9 0 720249224 6036185088 1435056 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38645.8
Current children cumulated vsize (KiB) 5894712

[startup+4902.3 s]
/proc/loadavg: 8.04 8.09 8.03 9/552 22173
/proc/meminfo: memFree=7760832/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=5964920 CPUtime=39124.7 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 6037807 1252 0 0 3911336 1131 0 0 25 0 9 0 720249202 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 1491230 1452608 180 55 0 1489116 0
[pid=22004/tid=22011] ppid=22002 vsize=5964920 CPUtime=4888.01 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 575113 1252 0 0 488694 107 0 0 25 0 9 0 720249223 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=5964920 CPUtime=4902 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 805413 1252 0 0 490047 153 0 0 25 0 9 0 720249223 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=5964920 CPUtime=4899.64 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 561819 1252 0 0 489871 93 0 0 25 0 9 0 720249223 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=5964920 CPUtime=4902.21 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 1339149 1252 0 0 489917 304 0 0 25 0 9 0 720249223 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=5964920 CPUtime=4900.5 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 598012 1252 0 0 489944 106 0 0 25 0 9 0 720249223 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=5964920 CPUtime=4845.45 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 831971 1252 0 0 484405 140 0 0 25 0 9 0 720249223 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=5964920 CPUtime=4900.72 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 810555 1252 0 0 489930 142 0 0 25 0 9 0 720249224 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=5964920 CPUtime=4885.86 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 506904 1252 0 0 488506 80 0 0 25 0 9 0 720249224 6108078080 1452608 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39124.7
Current children cumulated vsize (KiB) 5964920

[startup+4962.3 s]
/proc/loadavg: 8.01 8.07 8.02 9/552 22174
/proc/meminfo: memFree=8074420/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=5676152 CPUtime=39603.6 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 6224900 1252 0 0 3959185 1171 0 0 25 0 9 0 720249202 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 1419038 1380416 180 55 0 1416924 0
[pid=22004/tid=22011] ppid=22002 vsize=5676152 CPUtime=4948.01 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 589601 1252 0 0 494690 111 0 0 25 0 9 0 720249223 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364741 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=5676152 CPUtime=4962 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 805413 1252 0 0 496047 153 0 0 25 0 9 0 720249223 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=5676152 CPUtime=4959.64 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 561819 1252 0 0 495871 93 0 0 25 0 9 0 720249223 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=5676152 CPUtime=4962.21 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 1339149 1252 0 0 495917 304 0 0 25 0 9 0 720249223 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=5676152 CPUtime=4960.5 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 604092 1252 0 0 495943 107 0 0 25 0 9 0 720249223 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=5676152 CPUtime=4904.33 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 878119 1252 0 0 490285 148 0 0 25 0 9 0 720249223 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=5676152 CPUtime=4960.71 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 930932 1252 0 0 495901 170 0 0 25 0 9 0 720249224 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4263247 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=5676152 CPUtime=4945.87 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 506904 1252 0 0 494507 80 0 0 25 0 9 0 720249224 5812379648 1380416 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39603.6
Current children cumulated vsize (KiB) 5676152



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5012 s]
/proc/loadavg: 8.08 8.07 8.02 9/552 22176
/proc/meminfo: memFree=7932308/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=5793772 CPUtime=40000.4 cores=0-7
/proc/22004/stat : 22004 (plingeling) S 22002 22004 21978 0 -1 4202496 6268206 1252 0 0 3998862 1175 0 0 25 0 9 0 720249202 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 1448443 1409821 180 55 0 1446329 0
[pid=22004/tid=22011] ppid=22002 vsize=5793772 CPUtime=4997.73 cores=0-7
/proc/22004/task/22011/stat : 22011 (plingeling) R 22002 22004 21978 0 -1 4202560 589601 1252 0 0 499662 111 0 0 25 0 9 0 720249223 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22004/tid=22012] ppid=22002 vsize=5793772 CPUtime=5011.72 cores=0-7
/proc/22004/task/22012/stat : 22012 (plingeling) R 22002 22004 21978 0 -1 4202560 805413 1252 0 0 501019 153 0 0 25 0 9 0 720249223 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22004/tid=22013] ppid=22002 vsize=5793772 CPUtime=5009.36 cores=0-7
/proc/22004/task/22013/stat : 22013 (plingeling) R 22002 22004 21978 0 -1 4202560 561819 1252 0 0 500843 93 0 0 25 0 9 0 720249223 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22004/tid=22014] ppid=22002 vsize=5793772 CPUtime=5011.93 cores=0-7
/proc/22004/task/22014/stat : 22014 (plingeling) R 22002 22004 21978 0 -1 4202560 1339149 1252 0 0 500889 304 0 0 25 0 9 0 720249223 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=22004/tid=22015] ppid=22002 vsize=5793772 CPUtime=5010.22 cores=0-7
/proc/22004/task/22015/stat : 22015 (plingeling) R 22002 22004 21978 0 -1 4202560 642534 1252 0 0 500912 110 0 0 25 0 9 0 720249223 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4354515 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=5793772 CPUtime=4953.12 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4202560 878119 1252 0 0 495164 148 0 0 25 0 9 0 720249223 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22004/tid=22017] ppid=22002 vsize=5793772 CPUtime=5010.42 cores=0-7
/proc/22004/task/22017/stat : 22017 (plingeling) R 22002 22004 21978 0 -1 4202560 930932 1252 0 0 500872 170 0 0 25 0 9 0 720249224 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22004/tid=22018] ppid=22002 vsize=5793772 CPUtime=4995.59 cores=0-7
/proc/22004/task/22018/stat : 22018 (plingeling) R 22002 22004 21978 0 -1 4202560 511768 1252 0 0 499478 81 0 0 25 0 9 0 720249224 5932822528 1409821 33554432000 4194304 4417452 140733985555024 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 5793772

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5012.2 s]
/proc/loadavg: 8.08 8.07 8.02 9/552 22176
/proc/meminfo: memFree=7932308/32951124 swapFree=56449260/67111528
[pid=22004] ppid=22002 vsize=0 CPUtime=40000.6 cores=0-7
/proc/22004/stat : 22004 (plingeling) Z 22002 22004 21978 0 -1 4203548 6268206 1252 0 0 3998867 1193 0 0 18 0 2 0 720249202 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/22004/statm: 0 0 0 0 0 0 0
[pid=22004/tid=22016] ppid=22002 vsize=0 CPUtime=4953.29 cores=0-7
/proc/22004/task/22016/stat : 22016 (plingeling) R 22002 22004 21978 0 -1 4203588 878119 1252 0 0 495164 165 0 0 25 0 2 0 720249223 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5012.21
CPU time (s): 40000.6
CPU user time (s): 39988.7
CPU system time (s): 11.9432
CPU usage (%): 798.063
Max. virtual memory (cumulated for all children) (KiB): 6502268

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39988.7
system time used= 11.9432
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6269458
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 802
involuntary context switches= 143953

runsolver used 25.4711 second user time and 66.3059 second system time

The end

Launcher Data 

Begin job on node106 at 2011-05-22 19:48:08
IDJOB=3394084
IDBENCH=71409
IDSOLVER=1851
FILE ID=node106/3394084-1306086488
RUNJOBID= node106-1306086488-21992
PBS_JOBID= 13387922
Free space on /tmp= 73260 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v1820c171155gyes3.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394084-1306086488/watcher-3394084-1306086488 -o /tmp/evaluation-result-3394084-1306086488/solver-3394084-1306086488 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394084-1306086488.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= cd19d9eb9be97e183fa1a1d47d679d01
RANDOM SEED=692982096

node106.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      13579772 kB
Buffers:        341600 kB
Cached:       10941548 kB
SwapCached:      27520 kB
Active:        8014616 kB
Inactive:     11047052 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      13579772 kB
SwapTotal:    67111528 kB
SwapFree:     56449260 kB
Dirty:            2348 kB
Writeback:           0 kB
AnonPages:     7759772 kB
Mapped:          25448 kB
Slab:           156836 kB
PageTables:      78780 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 167519444 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73260 MiB
End job on node106 at 2011-05-22 21:11:42