Trace number 3394926

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.9 5006.7

General information on the benchmark

NameSAT02/handmade/simon/
satex-challenges/c6288.shuffled.cnf
MD5SUM75bd8319132b47c2d2e6f0889735a248
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark0.774881
Satisfiable
(Un)Satisfiability was proved
Number of variables5008
Number of clauses14813
Sum of the clauses size34485
Maximum clause length33
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 29982
Number of clauses of size 34829
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 51

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394926-1306109479/watcher-3394926-1306109479 -o /tmp/evaluation-result-3394926-1306109479/solver-3394926-1306109479 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394926-1306109479.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.02 8.02 7.99 1/320 23606
/proc/meminfo: memFree=19984408/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=12080 CPUtime=0 cores=0-7
/proc/23606/stat : 23606 (plingeling) R 23604 23606 23530 0 -1 4202496 1098 1252 0 0 0 0 0 0 24 0 1 0 801178501 12369920 1028 33554432000 4194304 4417452 140737398710752 18446744073709551615 260020395776 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/23606/statm: 3020 1028 148 55 0 906 0

[startup+0.0546401 s]
/proc/loadavg: 8.02 8.02 7.99 1/320 23606
/proc/meminfo: memFree=19984408/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=165672 CPUtime=0.12 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3588 1252 0 0 12 0 0 0 18 0 9 0 801178501 169648128 2778 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 41418 2805 174 55 0 39304 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 165672

[startup+0.145512 s]
/proc/loadavg: 8.02 8.02 7.99 1/320 23606
/proc/meminfo: memFree=19984408/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=165396 CPUtime=0.85 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3707 1252 0 0 85 0 0 0 18 0 9 0 801178501 169365504 2579 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 41349 2579 177 55 0 39235 0
Current children cumulated CPU time (s) 0.85
Current children cumulated vsize (KiB) 165396

[startup+0.300483 s]
/proc/loadavg: 8.02 8.02 7.99 1/320 23606
/proc/meminfo: memFree=19984408/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=166428 CPUtime=2.09 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 4162 1252 0 0 209 0 0 0 18 0 9 0 801178501 170422272 2855 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 41607 2855 180 55 0 39493 0
Current children cumulated CPU time (s) 2.09
Current children cumulated vsize (KiB) 166428

[startup+0.70041 s]
/proc/loadavg: 8.02 8.02 7.99 1/320 23606
/proc/meminfo: memFree=19984408/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=172604 CPUtime=5.29 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 6274 1252 0 0 529 0 0 0 18 0 9 0 801178501 176746496 4561 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 43151 4561 180 55 0 41037 0
Current children cumulated CPU time (s) 5.29
Current children cumulated vsize (KiB) 172604

[startup+1.50126 s]
/proc/loadavg: 8.02 8.02 7.99 9/329 23620
/proc/meminfo: memFree=19964000/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=176500 CPUtime=11.69 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 9925 1252 0 0 1167 2 0 0 18 0 9 0 801178501 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 44125 5424 180 55 0 42011 0
[pid=23606/tid=23613] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 955 1252 0 0 145 0 0 0 20 0 9 0 801178505 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4357260 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 618 1252 0 0 145 0 0 0 20 0 9 0 801178505 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4250852 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 935 1252 0 0 145 0 0 0 20 0 9 0 801178505 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364741 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=176500 CPUtime=1.44 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 905 1252 0 0 144 0 0 0 25 0 9 0 801178505 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4357970 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 649 1252 0 0 145 0 0 0 25 0 9 0 801178506 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4236852 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 919 1252 0 0 145 0 0 0 25 0 9 0 801178506 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 1130 1252 0 0 145 0 0 0 23 0 9 0 801178506 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4264888 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=176500 CPUtime=1.45 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 1094 1252 0 0 145 0 0 0 20 0 9 0 801178506 180736000 5424 33554432000 4194304 4417452 140737398710752 18446744073709551615 4357281 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.69
Current children cumulated vsize (KiB) 176500

[startup+3.10097 s]
/proc/loadavg: 8.02 8.01 7.99 9/329 23620
/proc/meminfo: memFree=19955072/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=185312 CPUtime=24.47 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 19757 1252 0 0 2442 5 0 0 18 0 9 0 801178501 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 46328 7630 180 55 0 44214 0
[pid=23606/tid=23613] ppid=23604 vsize=185312 CPUtime=3.05 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 2366 1252 0 0 305 0 0 0 25 0 9 0 801178505 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=185312 CPUtime=3.05 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 1528 1252 0 0 305 0 0 0 25 0 9 0 801178505 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4237237 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=185312 CPUtime=3.05 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 1742 1252 0 0 305 0 0 0 25 0 9 0 801178505 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=185312 CPUtime=3.02 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 2439 1252 0 0 302 0 0 0 25 0 9 0 801178505 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364589 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=185312 CPUtime=3.05 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 1804 1252 0 0 305 0 0 0 25 0 9 0 801178506 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4250762 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=185312 CPUtime=3.04 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 2504 1252 0 0 304 0 0 0 25 0 9 0 801178506 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=185312 CPUtime=3.04 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 2351 1252 0 0 304 0 0 0 25 0 9 0 801178506 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4232626 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=185312 CPUtime=3.05 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 2303 1252 0 0 305 0 0 0 25 0 9 0 801178506 189759488 7630 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.47
Current children cumulated vsize (KiB) 185312

[startup+6.30038 s]
/proc/loadavg: 8.02 8.01 7.99 9/329 23620
/proc/meminfo: memFree=19948620/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=192964 CPUtime=50.04 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 36446 1252 0 0 4995 9 0 0 18 0 9 0 801178501 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 48241 9526 180 55 0 46127 0
[pid=23606/tid=23613] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 5350 1252 0 0 624 1 0 0 25 0 9 0 801178505 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4365188 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 2851 1252 0 0 625 0 0 0 25 0 9 0 801178505 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 3908 1252 0 0 624 1 0 0 25 0 9 0 801178505 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=192964 CPUtime=6.19 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 4448 1252 0 0 618 1 0 0 25 0 9 0 801178505 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 3657 1252 0 0 625 0 0 0 25 0 9 0 801178506 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4237332 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 5281 1252 0 0 624 1 0 0 25 0 9 0 801178506 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 4483 1252 0 0 624 1 0 0 25 0 9 0 801178506 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=192964 CPUtime=6.25 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 3748 1252 0 0 625 0 0 0 25 0 9 0 801178506 197595136 9526 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.04
Current children cumulated vsize (KiB) 192964

[startup+12.7012 s]
/proc/loadavg: 8.02 8.01 7.99 9/329 23620
/proc/meminfo: memFree=19936896/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=207276 CPUtime=101.18 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 65948 1252 0 0 10102 16 0 0 18 0 9 0 801178501 212250624 13133 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 52060 13139 180 55 0 49946 0
[pid=23606/tid=23613] ppid=23604 vsize=208240 CPUtime=12.65 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 9731 1252 0 0 1262 3 0 0 25 0 9 0 801178505 213237760 13151 33554432000 4194304 4417452 140737398710752 18446744073709551615 4366486 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=208240 CPUtime=12.65 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 5976 1252 0 0 1264 1 0 0 25 0 9 0 801178505 213237760 13164 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=208240 CPUtime=12.65 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 7934 1252 0 0 1264 1 0 0 25 0 9 0 801178505 213237760 13180 33554432000 4194304 4417452 140737398710752 18446744073709551615 260020089949 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=208240 CPUtime=12.52 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 6783 1252 0 0 1251 1 0 0 25 0 9 0 801178505 213237760 13193 33554432000 4194304 4417452 140737398710752 18446744073709551615 4358006 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=208240 CPUtime=12.65 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 6974 1252 0 0 1264 1 0 0 25 0 9 0 801178506 213237760 13209 33554432000 4194304 4417452 140737398710752 18446744073709551615 4353585 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=208240 CPUtime=12.64 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 10236 1252 0 0 1262 2 0 0 25 0 9 0 801178506 213237760 13224 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=208240 CPUtime=12.65 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 8286 1252 0 0 1263 2 0 0 25 0 9 0 801178506 213237760 13239 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=208240 CPUtime=12.65 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 7355 1252 0 0 1264 1 0 0 25 0 9 0 801178506 213237760 13255 33554432000 4194304 4417452 140737398710752 18446744073709551615 4354332 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.18
Current children cumulated vsize (KiB) 207276

[startup+25.5008 s]
/proc/loadavg: 8.02 8.01 7.99 9/329 23620
/proc/meminfo: memFree=19926832/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=219376 CPUtime=203.44 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 108495 1252 0 0 20319 25 0 0 18 0 9 0 801178501 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 54844 16175 180 55 0 52730 0
[pid=23606/tid=23613] ppid=23604 vsize=219376 CPUtime=25.45 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 14127 1252 0 0 2542 3 0 0 25 0 9 0 801178505 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=219376 CPUtime=25.45 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 10097 1252 0 0 2543 2 0 0 25 0 9 0 801178505 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=219376 CPUtime=25.45 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 15097 1252 0 0 2542 3 0 0 25 0 9 0 801178505 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=219376 CPUtime=25.2 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 12034 1252 0 0 2518 2 0 0 25 0 9 0 801178505 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4366644 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=219376 CPUtime=25.45 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 10973 1252 0 0 2543 2 0 0 25 0 9 0 801178506 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=219376 CPUtime=25.44 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 15132 1252 0 0 2541 3 0 0 25 0 9 0 801178506 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4237189 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=219376 CPUtime=25.45 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 15838 1252 0 0 2542 3 0 0 25 0 9 0 801178506 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=219376 CPUtime=25.44 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 12477 1252 0 0 2542 2 0 0 25 0 9 0 801178506 224641024 16175 33554432000 4194304 4417452 140737398710752 18446744073709551615 4250807 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.44
Current children cumulated vsize (KiB) 219376

[startup+51.1011 s]
/proc/loadavg: 8.08 8.03 8.00 9/329 23621
/proc/meminfo: memFree=19896112/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=248424 CPUtime=407.99 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 187741 1252 0 0 40758 41 0 0 18 0 9 0 801178501 254386176 23420 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 62106 23420 180 55 0 59992 0
[pid=23606/tid=23613] ppid=23604 vsize=248424 CPUtime=51.05 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 23655 1252 0 0 5100 5 0 0 25 0 9 0 801178505 254386176 23420 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=248424 CPUtime=51.05 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 19096 1252 0 0 5101 4 0 0 25 0 9 0 801178505 254386176 23420 33554432000 4194304 4417452 140737398710752 18446744073709551615 4358118 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=248424 CPUtime=51.05 cores=0-7

################
# More data... #
################

/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 573497 1252 0 0 461369 109 0 0 25 0 9 0 801178505 2470731776 564519 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2412824 CPUtime=4662.35 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 367384 1252 0 0 466159 76 0 0 25 0 9 0 801178506 2470731776 564519 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2412824 CPUtime=4662.4 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 466158 82 0 0 25 0 9 0 801178506 2470731776 564519 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364850 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2412824 CPUtime=4661.64 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 363383 1252 0 0 466095 69 0 0 25 0 9 0 801178506 2470731776 564519 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2412824 CPUtime=4661.6 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 558770 1252 0 0 466059 101 0 0 25 0 9 0 801178506 2470731776 564519 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37249.9
Current children cumulated vsize (KiB) 2412824

[startup+4722.31 s]
/proc/loadavg: 8.03 8.05 8.00 9/329 23758
/proc/meminfo: memFree=17640588/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=2495176 CPUtime=37729.4 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3528393 1252 0 0 3772236 701 0 0 18 0 9 0 801178501 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 623794 585108 180 55 0 621680 0
[pid=23606/tid=23613] ppid=23604 vsize=2495176 CPUtime=4722.4 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 435895 1252 0 0 472150 90 0 0 25 0 9 0 801178505 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364873 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=2495176 CPUtime=4722.28 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 370636 1252 0 0 472152 76 0 0 25 0 9 0 801178505 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=2495176 CPUtime=4722.39 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 454486 1252 0 0 472149 90 0 0 25 0 9 0 801178505 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=2495176 CPUtime=4674.17 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 582669 1252 0 0 467307 110 0 0 25 0 9 0 801178505 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2495176 CPUtime=4722.37 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 367384 1252 0 0 472161 76 0 0 25 0 9 0 801178506 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2495176 CPUtime=4722.41 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 472159 82 0 0 25 0 9 0 801178506 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2495176 CPUtime=4721.64 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 373773 1252 0 0 472092 72 0 0 25 0 9 0 801178506 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2495176 CPUtime=4721.61 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 559797 1252 0 0 472060 101 0 0 25 0 9 0 801178506 2555060224 585108 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37729.4
Current children cumulated vsize (KiB) 2495176

[startup+4782.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/329 23759
/proc/meminfo: memFree=17633716/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=2501384 CPUtime=38208.7 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3546329 1252 0 0 3820163 704 0 0 18 0 9 0 801178501 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 625346 586660 180 55 0 623232 0
[pid=23606/tid=23613] ppid=23604 vsize=2501384 CPUtime=4782.4 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 435895 1252 0 0 478149 91 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=2501384 CPUtime=4782.27 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 370636 1252 0 0 478151 76 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=2501384 CPUtime=4782.38 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 454486 1252 0 0 478148 90 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=2501384 CPUtime=4733.55 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 588992 1252 0 0 473244 111 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2501384 CPUtime=4782.36 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 367384 1252 0 0 478160 76 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2501384 CPUtime=4782.4 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 478158 82 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2501384 CPUtime=4781.61 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 373773 1252 0 0 478089 72 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2501384 CPUtime=4781.61 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 571410 1252 0 0 478058 103 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38208.7
Current children cumulated vsize (KiB) 2501384

[startup+4842.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/329 23761
/proc/meminfo: memFree=17633792/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=2501384 CPUtime=38688 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3546329 1252 0 0 3868100 704 0 0 18 0 9 0 801178501 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 625346 586660 180 55 0 623232 0
[pid=23606/tid=23613] ppid=23604 vsize=2501384 CPUtime=4842.4 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 435895 1252 0 0 484149 91 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=2501384 CPUtime=4842.26 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 370636 1252 0 0 484150 76 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364741 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=2501384 CPUtime=4842.38 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 454486 1252 0 0 484148 90 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=2501384 CPUtime=4792.91 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 588992 1252 0 0 479180 111 0 0 25 0 9 0 801178505 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2501384 CPUtime=4842.36 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 367384 1252 0 0 484160 76 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2501384 CPUtime=4842.4 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 484158 82 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2501384 CPUtime=4841.61 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 373773 1252 0 0 484089 72 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2501384 CPUtime=4841.61 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 571410 1252 0 0 484058 103 0 0 25 0 9 0 801178506 2561417216 586660 33554432000 4194304 4417452 140737398710752 18446744073709551615 4236613 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38688
Current children cumulated vsize (KiB) 2501384

[startup+4902.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/329 23762
/proc/meminfo: memFree=17727848/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=2407264 CPUtime=39167.4 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3570954 1252 0 0 3916034 707 0 0 18 0 9 0 801178501 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 601816 563162 180 55 0 599702 0
[pid=23606/tid=23613] ppid=23604 vsize=2407264 CPUtime=4902.4 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 435895 1252 0 0 490149 91 0 0 25 0 9 0 801178505 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=2407264 CPUtime=4902.27 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 370636 1252 0 0 490151 76 0 0 25 0 9 0 801178505 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=2407264 CPUtime=4902.39 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 454486 1252 0 0 490149 90 0 0 25 0 9 0 801178505 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=2407264 CPUtime=4852.28 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 588992 1252 0 0 485117 111 0 0 25 0 9 0 801178505 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2407264 CPUtime=4902.37 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 392009 1252 0 0 490157 80 0 0 25 0 9 0 801178506 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4354244 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2407264 CPUtime=4902.41 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 490159 82 0 0 25 0 9 0 801178506 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2407264 CPUtime=4901.6 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 373773 1252 0 0 490088 72 0 0 25 0 9 0 801178506 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2407264 CPUtime=4901.61 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 571410 1252 0 0 490058 103 0 0 25 0 9 0 801178506 2465038336 563162 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39167.4
Current children cumulated vsize (KiB) 2407264

[startup+4962.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/329 23764
/proc/meminfo: memFree=17792908/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=2341728 CPUtime=39646.8 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3570954 1252 0 0 3963971 708 0 0 18 0 9 0 801178501 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 585432 546778 180 55 0 583318 0
[pid=23606/tid=23613] ppid=23604 vsize=2341728 CPUtime=4962.4 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 435895 1252 0 0 496149 91 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=2341728 CPUtime=4962.27 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 370636 1252 0 0 496151 76 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=2341728 CPUtime=4962.39 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 454486 1252 0 0 496149 90 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=2341728 CPUtime=4911.65 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 588992 1252 0 0 491054 111 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2341728 CPUtime=4962.37 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 392009 1252 0 0 496157 80 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2341728 CPUtime=4962.41 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 496159 82 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2341728 CPUtime=4961.59 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 373773 1252 0 0 496087 72 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2341728 CPUtime=4961.61 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 571410 1252 0 0 496058 103 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39646.8
Current children cumulated vsize (KiB) 2341728



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.6 s]
/proc/loadavg: 8.06 8.05 8.00 9/329 23765
/proc/meminfo: memFree=17792748/32951124 swapFree=67083432/67111528
[pid=23606] ppid=23604 vsize=2341728 CPUtime=40000.8 cores=0-7
/proc/23606/stat : 23606 (plingeling) S 23604 23606 23530 0 -1 4202496 3570954 1252 0 0 3999373 708 0 0 18 0 9 0 801178501 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/23606/statm: 585432 546778 180 55 0 583318 0
[pid=23606/tid=23613] ppid=23604 vsize=2341728 CPUtime=5006.71 cores=0-7
/proc/23606/task/23613/stat : 23613 (plingeling) R 23604 23606 23530 0 -1 4202560 435895 1252 0 0 500580 91 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23606/tid=23614] ppid=23604 vsize=2341728 CPUtime=5006.58 cores=0-7
/proc/23606/task/23614/stat : 23614 (plingeling) R 23604 23606 23530 0 -1 4202560 370636 1252 0 0 500582 76 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23606/tid=23615] ppid=23604 vsize=2341728 CPUtime=5006.7 cores=0-7
/proc/23606/task/23615/stat : 23615 (plingeling) R 23604 23606 23530 0 -1 4202560 454486 1252 0 0 500580 90 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23606/tid=23616] ppid=23604 vsize=2341728 CPUtime=4955.49 cores=0-7
/proc/23606/task/23616/stat : 23616 (plingeling) R 23604 23606 23530 0 -1 4202560 588992 1252 0 0 495438 111 0 0 25 0 9 0 801178505 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 18446744073699065939 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23606/tid=23617] ppid=23604 vsize=2341728 CPUtime=5006.68 cores=0-7
/proc/23606/task/23617/stat : 23617 (plingeling) R 23604 23606 23530 0 -1 4202560 392009 1252 0 0 500588 80 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23606/tid=23618] ppid=23604 vsize=2341728 CPUtime=5006.72 cores=0-7
/proc/23606/task/23618/stat : 23618 (plingeling) R 23604 23606 23530 0 -1 4202560 381033 1252 0 0 500590 82 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23606/tid=23619] ppid=23604 vsize=2341728 CPUtime=5005.89 cores=0-7
/proc/23606/task/23619/stat : 23619 (plingeling) R 23604 23606 23530 0 -1 4202560 373773 1252 0 0 500517 72 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23606/tid=23620] ppid=23604 vsize=2341728 CPUtime=5005.93 cores=0-7
/proc/23606/task/23620/stat : 23620 (plingeling) R 23604 23606 23530 0 -1 4202560 571410 1252 0 0 500490 103 0 0 25 0 9 0 801178506 2397929472 546778 33554432000 4194304 4417452 140737398710752 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2341728

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5006.7
CPU time (s): 40000.9
CPU user time (s): 39993.8
CPU system time (s): 7.17491
CPU usage (%): 798.948
Max. virtual memory (cumulated for all children) (KiB): 2566920

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.8
system time used= 7.17491
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3572206
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 282
involuntary context switches= 191819

runsolver used 14.7018 second user time and 36.5214 second system time

The end

Launcher Data 

Begin job on node110 at 2011-05-23 02:11:19
IDJOB=3394926
IDBENCH=87734
IDSOLVER=1851
FILE ID=node110/3394926-1306109479
RUNJOBID= node110-1306108889-23544
PBS_JOBID= 13388034
Free space on /tmp= 73344 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT02/handmade/simon/satex-challenges/c6288.shuffled.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394926-1306109479/watcher-3394926-1306109479 -o /tmp/evaluation-result-3394926-1306109479/solver-3394926-1306109479 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394926-1306109479.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 75bd8319132b47c2d2e6f0889735a248
RANDOM SEED=1368151381

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19984308 kB
Buffers:        326424 kB
Cached:        8684004 kB
SwapCached:       1416 kB
Active:        5964180 kB
Inactive:      6670240 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19984308 kB
SwapTotal:    67111528 kB
SwapFree:     67083432 kB
Dirty:             412 kB
Writeback:           0 kB
AnonPages:     3622740 kB
Mapped:          27808 kB
Slab:           239856 kB
PageTables:      26832 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 75581400 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73340 MiB
End job on node110 at 2011-05-23 03:34:48