Trace number 3371151

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.26 1202.97

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v2400c266431gyes3.cnf
MD5SUMcef3d3066fec910e94448f2404c09548
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3371151-1305420421/watcher-3371151-1305420421 -o /tmp/evaluation-result-3371151-1305420421/solver-3371151-1305420421 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3371151-1305420421.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.12 8.06 7.44 5/283 20669
/proc/meminfo: memFree=23849452/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=9164 CPUtime=0 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) R 20667 20669 19869 0 -1 4202496 384 1256 0 0 0 0 0 0 22 0 1 0 634457433 9383936 313 33554432000 4194304 4417452 140734270784224 18446744073709551615 4262059 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20669/statm: 2291 313 147 55 0 177 0

[startup+0.0480041 s]
/proc/loadavg: 8.12 8.06 7.44 5/283 20669
/proc/meminfo: memFree=23849452/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=18412 CPUtime=0.03 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) R 20667 20669 19869 0 -1 4202496 2665 1256 0 0 3 0 0 0 22 0 1 0 634457433 18853888 2594 33554432000 4194304 4417452 140734270784224 18446744073709551615 4259612 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20669/statm: 4603 2594 148 55 0 2489 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 18412

[startup+0.100983 s]
/proc/loadavg: 8.12 8.06 7.44 5/283 20669
/proc/meminfo: memFree=23849452/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=26604 CPUtime=0.08 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) R 20667 20669 19869 0 -1 4202496 4713 1256 0 0 8 0 0 0 22 0 1 0 634457433 27242496 4642 33554432000 4194304 4417452 140734270784224 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20669/statm: 6651 4642 148 55 0 4537 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26604

[startup+0.300913 s]
/proc/loadavg: 8.12 8.06 7.44 5/283 20669
/proc/meminfo: memFree=23849452/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=151056 CPUtime=0.65 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 9912 1256 0 0 64 1 0 0 23 0 5 0 634457433 154681344 9226 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 37764 9226 174 55 0 35650 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 151056

[startup+0.700842 s]
/proc/loadavg: 8.12 8.06 7.44 5/283 20669
/proc/meminfo: memFree=23849452/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=169252 CPUtime=2.24 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 18393 1256 0 0 222 2 0 0 23 0 5 0 634457433 173314048 13749 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 42313 13749 180 55 0 40199 0
Current children cumulated CPU time (s) 2.24
Current children cumulated vsize (KiB) 169252

[startup+1.5007 s]
/proc/loadavg: 8.11 8.06 7.45 9/288 20679
/proc/meminfo: memFree=23794108/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=176508 CPUtime=5.44 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 22836 1256 0 0 541 3 0 0 23 0 5 0 634457433 180744192 15563 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 44127 15563 180 55 0 42013 0
[pid=20669/tid=20676] ppid=20667 vsize=176508 CPUtime=1.31 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 2104 1256 0 0 131 0 0 0 25 0 5 0 634457451 180744192 15563 33554432000 4194304 4417452 140734270784224 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=176508 CPUtime=1.31 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 2254 1256 0 0 131 0 0 0 25 0 5 0 634457451 180744192 15563 33554432000 4194304 4417452 140734270784224 18446744073709551615 4258392 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=176508 CPUtime=1.31 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 2618 1256 0 0 131 0 0 0 25 0 5 0 634457451 180744192 15563 33554432000 4194304 4417452 140734270784224 18446744073709551615 4236008 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=176508 CPUtime=1.29 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 7039 1256 0 0 129 0 0 0 25 0 5 0 634457451 180744192 15563 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5.44
Current children cumulated vsize (KiB) 176508

[startup+3.10042 s]
/proc/loadavg: 8.11 8.06 7.45 9/288 20679
/proc/meminfo: memFree=23773056/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=196144 CPUtime=11.83 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 33364 1256 0 0 1179 4 0 0 23 0 5 0 634457433 200851456 20472 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 49036 20472 180 55 0 46922 0
[pid=20669/tid=20676] ppid=20667 vsize=196144 CPUtime=2.91 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 2104 1256 0 0 291 0 0 0 25 0 5 0 634457451 200851456 20472 33554432000 4194304 4417452 140734270784224 18446744073709551615 4236012 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=196144 CPUtime=2.91 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 3492 1256 0 0 291 0 0 0 25 0 5 0 634457451 200851456 20472 33554432000 4194304 4417452 140734270784224 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=196144 CPUtime=2.91 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 2618 1256 0 0 291 0 0 0 25 0 5 0 634457451 200851456 20472 33554432000 4194304 4417452 140734270784224 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=196144 CPUtime=2.89 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 16329 1256 0 0 287 2 0 0 25 0 5 0 634457451 200851456 20472 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.83
Current children cumulated vsize (KiB) 196144

[startup+6.30085 s]
/proc/loadavg: 8.10 8.06 7.45 10/288 20679
/proc/meminfo: memFree=23766840/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=222320 CPUtime=24.6 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 54398 1256 0 0 2452 8 0 0 23 0 5 0 634457433 227655680 27016 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 55580 27016 180 55 0 53466 0
[pid=20669/tid=20676] ppid=20667 vsize=222320 CPUtime=6.11 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 2104 1256 0 0 611 0 0 0 25 0 5 0 634457451 227655680 27016 33554432000 4194304 4417452 140734270784224 18446744073709551615 4250006 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=222320 CPUtime=6.1 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 3492 1256 0 0 610 0 0 0 25 0 5 0 634457451 227655680 27016 33554432000 4194304 4417452 140734270784224 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=222320 CPUtime=6.11 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 2618 1256 0 0 611 0 0 0 25 0 5 0 634457451 227655680 27016 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=222320 CPUtime=6.06 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 37363 1256 0 0 601 5 0 0 25 0 5 0 634457451 227655680 27016 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.6
Current children cumulated vsize (KiB) 222320

[startup+12.7007 s]
/proc/loadavg: 8.10 8.05 7.45 9/288 20680
/proc/meminfo: memFree=23711640/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=212184 CPUtime=50.13 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 90392 1256 0 0 5000 13 0 0 23 0 5 0 634457433 217276416 24514 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 53046 24514 180 55 0 50932 0
[pid=20669/tid=20676] ppid=20667 vsize=212184 CPUtime=12.51 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 4580 1256 0 0 1251 0 0 0 25 0 5 0 634457451 217276416 24514 33554432000 4194304 4417452 140734270784224 18446744073709551615 4336209 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=212184 CPUtime=12.5 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 4730 1256 0 0 1250 0 0 0 25 0 5 0 634457451 217276416 24514 33554432000 4194304 4417452 140734270784224 18446744073709551615 4236224 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=212184 CPUtime=12.51 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 2618 1256 0 0 1251 0 0 0 25 0 5 0 634457451 217276416 24514 33554432000 4194304 4417452 140734270784224 18446744073709551615 4343760 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=212184 CPUtime=12.4 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 69643 1256 0 0 1230 10 0 0 25 0 5 0 634457451 217276416 24514 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.13
Current children cumulated vsize (KiB) 212184

[startup+25.5004 s]
/proc/loadavg: 8.08 8.05 7.46 9/288 20680
/proc/meminfo: memFree=23680892/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=285568 CPUtime=101.21 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 145499 1256 0 0 10100 21 0 0 23 0 5 0 634457433 292421632 42828 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 71392 42828 180 55 0 69278 0
[pid=20669/tid=20676] ppid=20667 vsize=285568 CPUtime=25.31 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 5070 1256 0 0 2531 0 0 0 25 0 5 0 634457451 292421632 42828 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=285568 CPUtime=25.3 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 4847 1256 0 0 2530 0 0 0 25 0 5 0 634457451 292421632 42828 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=285568 CPUtime=25.31 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 3002 1256 0 0 2531 0 0 0 25 0 5 0 634457451 292421632 42828 33554432000 4194304 4417452 140734270784224 18446744073709551615 4249597 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=285568 CPUtime=25.08 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 123759 1256 0 0 2490 18 0 0 25 0 5 0 634457451 292421632 42828 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.21
Current children cumulated vsize (KiB) 285568

[startup+51.1009 s]
/proc/loadavg: 8.18 8.08 7.48 9/288 20681
/proc/meminfo: memFree=23398856/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=384704 CPUtime=203.39 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 241460 1256 0 0 20304 35 0 0 23 0 5 0 634457433 393936896 67612 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 96176 67612 180 55 0 94062 0
[pid=20669/tid=20676] ppid=20667 vsize=384704 CPUtime=50.91 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 6434 1256 0 0 5091 0 0 0 25 0 5 0 634457451 393936896 67612 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=384704 CPUtime=50.9 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 5202 1256 0 0 5090 0 0 0 25 0 5 0 634457451 393936896 67612 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=384704 CPUtime=50.91 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 5521 1256 0 0 5091 0 0 0 25 0 5 0 634457451 393936896 67612 33554432000 4194304 4417452 140734270784224 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=384704 CPUtime=50.45 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 215482 1256 0 0 5014 31 0 0 25 0 5 0 634457451 393936896 67612 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 203.39
Current children cumulated vsize (KiB) 384704

[startup+102.311 s]
/proc/loadavg: 8.20 8.09 7.52 9/288 20682
/proc/meminfo: memFree=23131996/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=597944 CPUtime=407.76 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 366063 1256 0 0 40720 56 0 0 23 0 5 0 634457433 612294656 120922 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 149486 120922 180 55 0 147372 0
[pid=20669/tid=20676] ppid=20667 vsize=597944 CPUtime=102.13 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 8068 1256 0 0 10212 1 0 0 25 0 5 0 634457451 612294656 120922 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364657 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=597944 CPUtime=102.1 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 6829 1256 0 0 10210 0 0 0 25 0 5 0 634457451 612294656 120922 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=597944 CPUtime=102.12 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 7438 1256 0 0 10212 0 0 0 25 0 5 0 634457451 612294656 120922 33554432000 4194304 4417452 140734270784224 18446744073709551615 4336180 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=597944 CPUtime=101.21 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 334907 1256 0 0 10069 52 0 0 25 0 5 0 634457451 612294656 120922 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.76
Current children cumulated vsize (KiB) 597944

[startup+162.301 s]
/proc/loadavg: 8.20 8.11 7.56 9/288 20684
/proc/meminfo: memFree=22859884/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=631692 CPUtime=647.19 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 453781 1256 0 0 64651 68 0 0 23 0 5 0 634457433 646852608 129359 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 157923 129359 180 55 0 155809 0
[pid=20669/tid=20676] ppid=20667 vsize=631692 CPUtime=162.11 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 11314 1256 0 0 16210 1 0 0 25 0 5 0 634457451 646852608 129359 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=631692 CPUtime=162.09 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 10885 1256 0 0 16208 1 0 0 25 0 5 0 634457451 646852608 129359 33554432000 4194304 4417452 140734270784224 18446744073709551615 4250623 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=631692 CPUtime=162.11 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 9577 1256 0 0 16211 0 0 0 25 0 5 0 634457451 646852608 129359 33554432000 4194304 4417452 140734270784224 18446744073709551615 4344176 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=631692 CPUtime=160.66 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 413184 1256 0 0 16004 62 0 0 25 0 5 0 634457451 646852608 129359 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 647.19
Current children cumulated vsize (KiB) 631692


################
# More data... #
################

[pid=20669/tid=20677] ppid=20667 vsize=1679248 CPUtime=762.02 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 29324 1256 0 0 76199 3 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1679248 CPUtime=762.14 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 30026 1256 0 0 76211 3 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1679248 CPUtime=755.32 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1028475 1256 0 0 75371 161 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3041.82
Current children cumulated vsize (KiB) 1679248

[startup+822.301 s]
/proc/loadavg: 8.08 8.11 7.82 9/288 20738
/proc/meminfo: memFree=21906064/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1679248 CPUtime=3281.23 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1137138 1256 0 0 327949 174 0 0 23 0 5 0 634457433 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 419812 391248 180 55 0 417698 0
[pid=20669/tid=20676] ppid=20667 vsize=1679248 CPUtime=822.13 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 82208 5 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4250432 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1679248 CPUtime=822 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 29324 1256 0 0 82197 3 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1679248 CPUtime=822.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 30026 1256 0 0 82210 3 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1679248 CPUtime=814.77 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1028475 1256 0 0 81316 161 0 0 25 0 5 0 634457451 1719549952 391248 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3281.23
Current children cumulated vsize (KiB) 1679248

[startup+882.3 s]
/proc/loadavg: 8.07 8.10 7.83 9/288 20749
/proc/meminfo: memFree=21433712/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1798936 CPUtime=3520.67 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1167060 1256 0 0 351887 180 0 0 23 0 5 0 634457433 1842110464 421170 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 449734 421170 180 55 0 447620 0
[pid=20669/tid=20676] ppid=20667 vsize=1798936 CPUtime=882.13 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 88208 5 0 0 25 0 5 0 634457451 1842110464 421170 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1798936 CPUtime=881.99 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 30776 1256 0 0 88196 3 0 0 25 0 5 0 634457451 1842110464 421170 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1798936 CPUtime=882.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 88210 3 0 0 25 0 5 0 634457451 1842110464 421170 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1798936 CPUtime=874.22 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1053945 1256 0 0 87256 166 0 0 25 0 5 0 634457451 1842110464 421170 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3520.67
Current children cumulated vsize (KiB) 1798936

[startup+942.301 s]
/proc/loadavg: 8.02 8.08 7.84 9/288 20750
/proc/meminfo: memFree=21110448/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=2121052 CPUtime=3760.13 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1247589 1256 0 0 375818 195 0 0 23 0 5 0 634457433 2171957248 501699 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 530263 501699 180 55 0 528149 0
[pid=20669/tid=20676] ppid=20667 vsize=2121052 CPUtime=942.13 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 94208 5 0 0 25 0 5 0 634457451 2171957248 501699 33554432000 4194304 4417452 140734270784224 18446744073709551615 4357263 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=2121052 CPUtime=941.98 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 31465 1256 0 0 94195 3 0 0 25 0 5 0 634457451 2171957248 501699 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=2121052 CPUtime=942.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 94210 3 0 0 25 0 5 0 634457451 2171957248 501699 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=2121052 CPUtime=933.68 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1133785 1256 0 0 93186 182 0 0 25 0 5 0 634457451 2171957248 501699 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3760.13
Current children cumulated vsize (KiB) 2121052

[startup+1002.3 s]
/proc/loadavg: 8.13 8.09 7.86 9/288 20752
/proc/meminfo: memFree=22813588/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1268900 CPUtime=3999.57 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1363617 1256 0 0 399746 211 0 0 23 0 5 0 634457433 1299353600 288847 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 317225 288847 180 55 0 315111 0
[pid=20669/tid=20676] ppid=20667 vsize=1268900 CPUtime=1002.14 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 100209 5 0 0 25 0 5 0 634457451 1299353600 288847 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1268900 CPUtime=1001.97 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 31651 1256 0 0 100194 3 0 0 25 0 5 0 634457451 1299353600 288847 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1268900 CPUtime=1002.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 100210 3 0 0 25 0 5 0 634457451 1299353600 288847 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1268900 CPUtime=993.13 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1249627 1256 0 0 99115 198 0 0 25 0 5 0 634457451 1299353600 288847 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3999.57
Current children cumulated vsize (KiB) 1268900

[startup+1062.3 s]
/proc/loadavg: 8.16 8.11 7.87 9/288 20754
/proc/meminfo: memFree=22789028/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1268900 CPUtime=4239.04 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1381260 1256 0 0 423690 214 0 0 23 0 5 0 634457433 1299353600 293323 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 317225 293323 180 55 0 315111 0
[pid=20669/tid=20676] ppid=20667 vsize=1268900 CPUtime=1062.14 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 106209 5 0 0 25 0 5 0 634457451 1299353600 293323 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366149 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1268900 CPUtime=1061.97 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 49294 1256 0 0 106192 5 0 0 25 0 5 0 634457451 1299353600 293323 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1268900 CPUtime=1062.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 106210 3 0 0 25 0 5 0 634457451 1299353600 293323 33554432000 4194304 4417452 140734270784224 18446744073709551615 4365150 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1268900 CPUtime=1052.6 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1249627 1256 0 0 105062 198 0 0 25 0 5 0 634457451 1299353600 293323 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4239.04
Current children cumulated vsize (KiB) 1268900

[startup+1122.3 s]
/proc/loadavg: 8.16 8.11 7.89 9/288 20755
/proc/meminfo: memFree=22269340/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1370240 CPUtime=4478.48 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1437309 1256 0 0 447628 220 0 0 23 0 5 0 634457433 1403125760 317901 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 342560 317901 180 55 0 340446 0
[pid=20669/tid=20676] ppid=20667 vsize=1370240 CPUtime=1122.14 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 112209 5 0 0 25 0 5 0 634457451 1403125760 317901 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1370240 CPUtime=1121.96 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 80008 1256 0 0 112188 8 0 0 25 0 5 0 634457451 1403125760 317901 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1370240 CPUtime=1122.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 112210 3 0 0 25 0 5 0 634457451 1403125760 317901 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1370240 CPUtime=1112.05 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1274962 1256 0 0 111004 201 0 0 25 0 5 0 634457451 1403125760 317901 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4478.48
Current children cumulated vsize (KiB) 1370240

[startup+1182.3 s]
/proc/loadavg: 8.16 8.12 7.91 9/288 20757
/proc/meminfo: memFree=22012164/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1626076 CPUtime=4717.94 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1519082 1256 0 0 471566 228 0 0 23 0 5 0 634457433 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 406519 377955 180 55 0 404405 0
[pid=20669/tid=20676] ppid=20667 vsize=1626076 CPUtime=1182.14 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 118209 5 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4366661 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1626076 CPUtime=1181.95 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 97822 1256 0 0 118185 10 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364809 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1626076 CPUtime=1182.13 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 118210 3 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1626076 CPUtime=1171.51 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1338921 1256 0 0 116944 207 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4717.94
Current children cumulated vsize (KiB) 1626076



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.9 s]
/proc/loadavg: 8.19 8.13 7.91 9/288 20757
/proc/meminfo: memFree=22012160/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1626076 CPUtime=4800.17 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1519082 1256 0 0 479789 228 0 0 23 0 5 0 634457433 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 406519 377955 180 55 0 404405 0
[pid=20669/tid=20676] ppid=20667 vsize=1626076 CPUtime=1202.75 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 120270 5 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1626076 CPUtime=1202.55 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 97822 1256 0 0 120245 10 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1626076 CPUtime=1202.74 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 120271 3 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4365572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1626076 CPUtime=1191.92 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1338921 1256 0 0 118985 207 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 1626076

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.9 s]
/proc/loadavg: 8.19 8.13 7.91 9/288 20757
/proc/meminfo: memFree=22012160/32951124 swapFree=67057896/67111528
[pid=20669] ppid=20667 vsize=1626076 CPUtime=4800.17 cores=1,3,5,7
/proc/20669/stat : 20669 (plingeling) S 20667 20669 19869 0 -1 4202496 1519082 1256 0 0 479789 228 0 0 23 0 5 0 634457433 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 254273419237 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20669/statm: 406519 377955 180 55 0 404405 0
[pid=20669/tid=20676] ppid=20667 vsize=1626076 CPUtime=1202.75 cores=1,3,5,7
/proc/20669/task/20676/stat : 20676 (plingeling) R 20667 20669 19869 0 -1 4202560 40492 1256 0 0 120270 5 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20669/tid=20677] ppid=20667 vsize=1626076 CPUtime=1202.55 cores=1,3,5,7
/proc/20669/task/20677/stat : 20677 (plingeling) R 20667 20669 19869 0 -1 4202560 97822 1256 0 0 120245 10 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20669/tid=20678] ppid=20667 vsize=1626076 CPUtime=1202.74 cores=1,3,5,7
/proc/20669/task/20678/stat : 20678 (plingeling) R 20667 20669 19869 0 -1 4202560 33026 1256 0 0 120271 3 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4365572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20669/tid=20679] ppid=20667 vsize=1626076 CPUtime=1191.92 cores=1,3,5,7
/proc/20669/task/20679/stat : 20679 (plingeling) R 20667 20669 19869 0 -1 4202560 1338921 1256 0 0 118985 207 0 0 25 0 5 0 634457451 1665101824 377955 33554432000 4194304 4417452 140734270784224 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 1626076

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.97
CPU time (s): 4800.26
CPU user time (s): 4797.92
CPU system time (s): 2.34264
CPU usage (%): 399.033
Max. virtual memory (cumulated for all children) (KiB): 2121052

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.92
system time used= 2.34264
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1520338
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 135
involuntary context switches= 36701

runsolver used 3.20751 second user time and 7.62584 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-15 02:47:01
IDJOB=3371151
IDBENCH=71549
IDSOLVER=1848
FILE ID=node140/3371151-1305420421
RUNJOBID= node140-1305419216-20536
PBS_JOBID= 13324552
Free space on /tmp= 73480 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v2400c266431gyes3.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3371151-1305420421/watcher-3371151-1305420421 -o /tmp/evaluation-result-3371151-1305420421/solver-3371151-1305420421 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3371151-1305420421.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= cef3d3066fec910e94448f2404c09548
RANDOM SEED=2072400889

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23849740 kB
Buffers:        303884 kB
Cached:        3858176 kB
SwapCached:       8056 kB
Active:        5984208 kB
Inactive:      2805456 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23849740 kB
SwapTotal:    67111528 kB
SwapFree:     67057896 kB
Dirty:            3528 kB
Writeback:           0 kB
AnonPages:     4624568 kB
Mapped:          23428 kB
Slab:           224700 kB
PageTables:      24436 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 46766512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73480 MiB
End job on node140 at 2011-05-15 03:07:06