Trace number 3309750

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 40000.8 5004.78

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k50-eijkbs6669-tseitin.cnf
MD5SUM7886cd26d6e0801ea382a1ffd6224023
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark2052.83
Satisfiable
(Un)Satisfiability was proved
Number of variables192503
Number of clauses564953
Sum of the clauses size1318221
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2376634
Number of clauses of size 3188317
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3309750-1304008200/watcher-3309750-1304008200 -o /tmp/evaluation-result-3309750-1304008200/solver-3309750-1304008200 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3309750-1304008200.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.86 7.85 7.91 2/170 15994
/proc/meminfo: memFree=13513484/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=12316 CPUtime=0 cores=0-7
/proc/15994/stat : 15994 (plingeling) R 15992 15994 15925 0 -1 4202496 1160 1253 0 0 0 0 0 0 21 0 1 0 512417979 12611584 1090 33554432000 4194304 4417332 140734484598320 18446744073709551615 228122407902 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/15994/statm: 3079 1090 146 55 0 965 0

[startup+0.093077 s]
/proc/loadavg: 7.86 7.85 7.91 2/170 15994
/proc/meminfo: memFree=13513484/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=30972 CPUtime=0.08 cores=0-7
/proc/15994/stat : 15994 (plingeling) R 15992 15994 15925 0 -1 4202496 5845 1253 0 0 8 0 0 0 21 0 1 0 512417979 31715328 5732 33554432000 4194304 4417332 140734484598320 18446744073709551615 4209566 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/15994/statm: 7743 5732 148 55 0 5629 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 30972

[startup+0.101077 s]
/proc/loadavg: 7.86 7.85 7.91 2/170 15994
/proc/meminfo: memFree=13513484/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=30972 CPUtime=0.08 cores=0-7
/proc/15994/stat : 15994 (plingeling) R 15992 15994 15925 0 -1 4202496 5845 1253 0 0 8 0 0 0 21 0 1 0 512417979 31715328 5732 33554432000 4194304 4417332 140734484598320 18446744073709551615 4231477 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/15994/statm: 7743 5732 148 55 0 5629 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 30972

[startup+0.30104 s]
/proc/loadavg: 7.86 7.85 7.91 2/170 15994
/proc/meminfo: memFree=13513484/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=81968 CPUtime=0.28 cores=0-7
/proc/15994/stat : 15994 (plingeling) R 15992 15994 15925 0 -1 4202496 18628 1253 0 0 27 1 0 0 22 0 1 0 512417979 83935232 18513 33554432000 4194304 4417332 140734484598320 18446744073709551615 4231387 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/15994/statm: 20492 18513 148 55 0 18378 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 81968

[startup+0.700966 s]
/proc/loadavg: 7.86 7.85 7.91 2/170 15994
/proc/meminfo: memFree=13513484/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=151908 CPUtime=0.68 cores=0-7
/proc/15994/stat : 15994 (plingeling) R 15992 15994 15925 0 -1 4202496 36166 1253 0 0 65 3 0 0 23 0 1 0 512417979 155553792 35998 33554432000 4194304 4417332 140734484598320 18446744073709551615 4259147 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/15994/statm: 37977 35998 148 55 0 35863 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 151908

[startup+1.50079 s]
/proc/loadavg: 7.86 7.85 7.91 2/171 16000
/proc/meminfo: memFree=13294368/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=525852 CPUtime=2.39 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 93232 1253 0 0 228 11 0 0 25 0 9 0 512417979 538472448 67474 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 131463 67474 167 55 0 129349 0
Current children cumulated CPU time (s) 2.39
Current children cumulated vsize (KiB) 525852

[startup+3.1005 s]
/proc/loadavg: 7.86 7.85 7.91 9/179 16009
/proc/meminfo: memFree=13155336/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=614176 CPUtime=14.91 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 134278 1253 0 0 1473 18 0 0 25 0 9 0 512417979 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 153544 87752 173 55 0 151430 0
[pid=15994/tid=16001] ppid=15992 vsize=614176 CPUtime=1.73 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 10158 1253 0 0 172 1 0 0 20 0 9 0 512418112 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4275888 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=614176 CPUtime=1.71 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 10112 1253 0 0 170 1 0 0 20 0 9 0 512418112 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4295187 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=614176 CPUtime=1.76 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 7889 1253 0 0 175 1 0 0 20 0 9 0 512418112 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=614176 CPUtime=1.76 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 4395 1253 0 0 176 0 0 0 20 0 9 0 512418112 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4232598 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=614176 CPUtime=1.51 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 6567 1253 0 0 151 0 0 0 25 0 9 0 512418112 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4277563 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=614176 CPUtime=1.73 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 9037 1253 0 0 172 1 0 0 25 0 9 0 512418112 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=614176 CPUtime=1.73 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 8294 1253 0 0 172 1 0 0 20 0 9 0 512418114 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4302020 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=614176 CPUtime=1.57 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 6567 1253 0 0 157 0 0 0 25 0 9 0 512418118 628916224 87752 33554432000 4194304 4417332 140734484598320 18446744073709551615 4236672 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 14.91
Current children cumulated vsize (KiB) 614176

[startup+6.30092 s]
/proc/loadavg: 7.87 7.85 7.91 9/179 16009
/proc/meminfo: memFree=13149136/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=581268 CPUtime=40.47 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 166892 1253 0 0 4025 22 0 0 25 0 9 0 512417979 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 145317 94092 180 55 0 143203 0
[pid=15994/tid=16001] ppid=15992 vsize=581268 CPUtime=4.92 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 17076 1253 0 0 490 2 0 0 21 0 9 0 512418112 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4253296 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=581268 CPUtime=4.9 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 13549 1253 0 0 488 2 0 0 25 0 9 0 512418112 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4253198 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=581268 CPUtime=4.95 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 9943 1253 0 0 494 1 0 0 25 0 9 0 512418112 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4377889 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=581268 CPUtime=4.96 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 4397 1253 0 0 496 0 0 0 25 0 9 0 512418112 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4232636 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=581268 CPUtime=4.72 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 10197 1253 0 0 471 1 0 0 25 0 9 0 512418112 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4257746 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=581268 CPUtime=4.93 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 15383 1253 0 0 491 2 0 0 25 0 9 0 512418112 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4252349 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=581268 CPUtime=4.93 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 16331 1253 0 0 491 2 0 0 25 0 9 0 512418114 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4376618 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=581268 CPUtime=4.76 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 8757 1253 0 0 476 0 0 0 25 0 9 0 512418118 595218432 94092 33554432000 4194304 4417332 140734484598320 18446744073709551615 4252591 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40.47
Current children cumulated vsize (KiB) 581268

[startup+12.7008 s]
/proc/loadavg: 7.88 7.85 7.91 9/179 16009
/proc/meminfo: memFree=13026024/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=672864 CPUtime=91.63 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 194463 1253 0 0 9136 27 0 0 25 0 9 0 512417979 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 168216 121567 180 55 0 166102 0
[pid=15994/tid=16001] ppid=15992 vsize=672864 CPUtime=11.32 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 22197 1253 0 0 1128 4 0 0 25 0 9 0 512418112 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4273019 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=672864 CPUtime=11.26 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 18670 1253 0 0 1123 3 0 0 25 0 9 0 512418112 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4257654 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=672864 CPUtime=11.35 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 10935 1253 0 0 1134 1 0 0 25 0 9 0 512418112 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4343668 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=672864 CPUtime=11.36 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 5885 1253 0 0 1136 0 0 0 25 0 9 0 512418112 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4265383 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=672864 CPUtime=11.11 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 14293 1253 0 0 1110 1 0 0 25 0 9 0 512418112 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4252431 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=672864 CPUtime=11.33 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 21527 1253 0 0 1130 3 0 0 25 0 9 0 512418112 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4253214 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=672864 CPUtime=11.33 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 16844 1253 0 0 1131 2 0 0 25 0 9 0 512418114 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=672864 CPUtime=11.17 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 12853 1253 0 0 1116 1 0 0 25 0 9 0 512418118 689012736 121567 33554432000 4194304 4417332 140734484598320 18446744073709551615 4252361 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 91.63
Current children cumulated vsize (KiB) 672864

[startup+25.5005 s]
/proc/loadavg: 7.91 7.86 7.91 9/179 16009
/proc/meminfo: memFree=13109912/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=588136 CPUtime=193.96 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 194742 1253 0 0 19368 28 0 0 25 0 9 0 512417979 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 147034 100386 181 55 0 144920 0
[pid=15994/tid=16001] ppid=15992 vsize=588136 CPUtime=24.12 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 22238 1253 0 0 2408 4 0 0 25 0 9 0 512418112 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4253344 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=588136 CPUtime=23.99 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 18670 1253 0 0 2396 3 0 0 25 0 9 0 512418112 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=588136 CPUtime=24.15 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 11091 1253 0 0 2414 1 0 0 25 0 9 0 512418112 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=588136 CPUtime=24.16 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 5885 1253 0 0 2416 0 0 0 25 0 9 0 512418112 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=588136 CPUtime=23.91 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 14334 1253 0 0 2390 1 0 0 25 0 9 0 512418112 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4257368 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=588136 CPUtime=24.13 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 21567 1253 0 0 2410 3 0 0 25 0 9 0 512418112 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4274074 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=588136 CPUtime=24.13 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 16844 1253 0 0 2411 2 0 0 25 0 9 0 512418114 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4236559 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=588136 CPUtime=23.97 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 12854 1253 0 0 2396 1 0 0 25 0 9 0 512418118 602251264 100386 33554432000 4194304 4417332 140734484598320 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 193.96
Current children cumulated vsize (KiB) 588136

[startup+51.1008 s]
/proc/loadavg: 7.99 7.89 7.91 9/179 16010
/proc/meminfo: memFree=13080404/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=619808 CPUtime=398.63 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 203276 1253 0 0 39834 29 0 0 25 0 9 0 512417979 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 154952 108304 181 55 0 152838 0
[pid=15994/tid=16001] ppid=15992 vsize=619808 CPUtime=49.72 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 22313 1253 0 0 4968 4 0 0 25 0 9 0 512418112 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4232695 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=619808 CPUtime=49.46 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 18742 1253 0 0 4943 3 0 0 25 0 9 0 512418112 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4252240 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=619808 CPUtime=49.76 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 13097 1253 0 0 4974 2 0 0 25 0 9 0 512418112 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4250721 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=619808 CPUtime=49.76 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 10721 1253 0 0 4975 1 0 0 25 0 9 0 512418112 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4250651 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=619808 CPUtime=49.5 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 14373 1253 0 0 4949 1 0 0 25 0 9 0 512418112 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=619808 CPUtime=49.73 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 21880 1253 0 0 4970 3 0 0 25 0 9 0 512418112 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=619808 CPUtime=49.73 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 17961 1253 0 0 4971 2 0 0 25 0 9 0 512418114 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4237290 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=619808 CPUtime=49.57 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 12930 1253 0 0 4956 1 0 0 25 0 9 0 512418118 634683392 108304 33554432000 4194304 4417332 140734484598320 18446744073709551615 4271941 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 398.63
Current children cumulated vsize (KiB) 619808

[startup+102.307 s]
/proc/loadavg: 8.06 7.91 7.92 9/179 16011

################
# More data... #
################

/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1218497 1253 0 0 471865 225 0 0 25 0 9 0 512418112 1891000320 411034 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1846680 CPUtime=4720.37 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1196673 1253 0 0 471844 193 0 0 25 0 9 0 512418112 1891000320 411034 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1846680 CPUtime=4721.05 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1062431 1253 0 0 471910 195 0 0 25 0 9 0 512418112 1891000320 411034 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1846680 CPUtime=4720.93 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1077507 1253 0 0 471920 173 0 0 25 0 9 0 512418114 1891000320 411034 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1846680 CPUtime=4720.9 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 714327 1253 0 0 471985 105 0 0 25 0 9 0 512418118 1891000320 411034 33554432000 4194304 4417332 140734484598320 18446744073709551615 4354352 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37743
Current children cumulated vsize (KiB) 1846680

[startup+4782.3 s]
/proc/loadavg: 8.04 8.04 8.00 9/179 16148
/proc/meminfo: memFree=11911616/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=1813828 CPUtime=38222.7 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 8563401 1253 0 0 3820801 1465 0 0 25 0 9 0 512417979 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 453457 402821 181 55 0 451343 0
[pid=15994/tid=16001] ppid=15992 vsize=1813828 CPUtime=4781.05 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 1018399 1253 0 0 477927 178 0 0 25 0 9 0 512418112 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=1813828 CPUtime=4755.65 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 1129727 1253 0 0 475362 203 0 0 25 0 9 0 512418112 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364329 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=1813828 CPUtime=4780.44 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 987374 1253 0 0 477881 163 0 0 25 0 9 0 512418112 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=1813828 CPUtime=4780.9 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1232999 1253 0 0 477862 228 0 0 25 0 9 0 512418112 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364565 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1813828 CPUtime=4780.36 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1232223 1253 0 0 477836 200 0 0 25 0 9 0 512418112 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1813828 CPUtime=4781.04 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1076890 1253 0 0 477906 198 0 0 25 0 9 0 512418112 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1813828 CPUtime=4780.92 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1093839 1253 0 0 477916 176 0 0 25 0 9 0 512418114 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1813828 CPUtime=4780.9 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 720691 1253 0 0 477983 107 0 0 25 0 9 0 512418118 1857359872 402821 33554432000 4194304 4417332 140734484598320 18446744073709551615 4252915 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38222.7
Current children cumulated vsize (KiB) 1813828

[startup+4842.3 s]
/proc/loadavg: 8.05 8.04 8.00 9/179 16150
/proc/meminfo: memFree=11936632/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=1779724 CPUtime=38702.3 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 8655140 1253 0 0 3868755 1479 0 0 25 0 9 0 512417979 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 444931 394295 181 55 0 442817 0
[pid=15994/tid=16001] ppid=15992 vsize=1779724 CPUtime=4841.05 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 1022206 1253 0 0 483927 178 0 0 25 0 9 0 512418112 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=1779724 CPUtime=4815.32 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 1141608 1253 0 0 481328 204 0 0 25 0 9 0 512418112 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364776 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=1779724 CPUtime=4840.44 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 993718 1253 0 0 483880 164 0 0 25 0 9 0 512418112 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=1779724 CPUtime=4840.9 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1245753 1253 0 0 483860 230 0 0 25 0 9 0 512418112 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1779724 CPUtime=4840.37 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1251801 1253 0 0 483834 203 0 0 25 0 9 0 512418112 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1779724 CPUtime=4841.04 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1085855 1253 0 0 483905 199 0 0 25 0 9 0 512418112 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1779724 CPUtime=4840.93 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1107013 1253 0 0 483914 179 0 0 25 0 9 0 512418114 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1779724 CPUtime=4840.9 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 735927 1253 0 0 483980 110 0 0 25 0 9 0 512418118 1822437376 394295 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38702.3
Current children cumulated vsize (KiB) 1779724

[startup+4902.3 s]
/proc/loadavg: 8.05 8.04 8.00 9/179 16151
/proc/meminfo: memFree=11895540/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=1765984 CPUtime=39182 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 8775604 1253 0 0 3916706 1496 0 0 25 0 9 0 512417979 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 441496 390860 181 55 0 439382 0
[pid=15994/tid=16001] ppid=15992 vsize=1765984 CPUtime=4901.05 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 1029859 1253 0 0 489926 179 0 0 25 0 9 0 512418112 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4366303 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=1765984 CPUtime=4875 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 1145436 1253 0 0 487295 205 0 0 25 0 9 0 512418112 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=1765984 CPUtime=4900.44 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 1003963 1253 0 0 489878 166 0 0 25 0 9 0 512418112 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=1765984 CPUtime=4900.9 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1276297 1253 0 0 489854 236 0 0 25 0 9 0 512418112 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4365846 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1765984 CPUtime=4900.36 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1269091 1253 0 0 489831 205 0 0 25 0 9 0 512418112 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364547 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1765984 CPUtime=4901.05 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1102042 1253 0 0 489904 201 0 0 25 0 9 0 512418112 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1765984 CPUtime=4900.92 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1128226 1253 0 0 489911 181 0 0 25 0 9 0 512418114 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364308 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1765984 CPUtime=4900.9 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 749431 1253 0 0 489979 111 0 0 25 0 9 0 512418118 1808367616 390860 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364289 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39182
Current children cumulated vsize (KiB) 1765984

[startup+4962.3 s]
/proc/loadavg: 8.13 8.07 8.01 9/179 16153
/proc/meminfo: memFree=12015160/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=1729912 CPUtime=39661.7 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 8845042 1253 0 0 3964662 1507 0 0 25 0 9 0 512417979 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 432478 381842 181 55 0 430364 0
[pid=15994/tid=16001] ppid=15992 vsize=1729912 CPUtime=4961.06 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 1033069 1253 0 0 495926 180 0 0 25 0 9 0 512418112 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364942 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=1729912 CPUtime=4934.67 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 1159269 1253 0 0 493260 207 0 0 25 0 9 0 512418112 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364544 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=1729912 CPUtime=4960.45 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 1011479 1253 0 0 495878 167 0 0 25 0 9 0 512418112 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4366011 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=1729912 CPUtime=4960.91 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1294605 1253 0 0 495853 238 0 0 25 0 9 0 512418112 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4365869 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1729912 CPUtime=4960.35 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1282340 1253 0 0 495827 208 0 0 25 0 9 0 512418112 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1729912 CPUtime=4961.05 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1103924 1253 0 0 495904 201 0 0 25 0 9 0 512418112 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1729912 CPUtime=4960.92 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1134081 1253 0 0 495909 183 0 0 25 0 9 0 512418114 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1729912 CPUtime=4960.9 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 755016 1253 0 0 495978 112 0 0 25 0 9 0 512418118 1771429888 381842 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364575 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39661.7
Current children cumulated vsize (KiB) 1729912



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5004.7 s]
/proc/loadavg: 8.06 8.05 8.00 9/179 16154
/proc/meminfo: memFree=12060124/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=1655672 CPUtime=40000.7 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 8925792 1253 0 0 3998550 1520 0 0 25 0 9 0 512417979 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 413918 363282 181 55 0 411804 0
[pid=15994/tid=16001] ppid=15992 vsize=1655672 CPUtime=5003.46 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 1037062 1253 0 0 500165 181 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=1655672 CPUtime=4976.84 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 1165419 1253 0 0 497476 208 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=1655672 CPUtime=5002.85 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 1014048 1253 0 0 500118 167 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=1655672 CPUtime=5003.31 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1304400 1253 0 0 500092 239 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364973 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1655672 CPUtime=5002.76 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1288109 1253 0 0 500067 209 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1655672 CPUtime=5003.45 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1118261 1253 0 0 500141 204 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1655672 CPUtime=5003.32 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1147604 1253 0 0 500147 185 0 0 25 0 9 0 512418114 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4357066 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1655672 CPUtime=5003.3 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 779630 1253 0 0 500215 115 0 0 25 0 9 0 512418118 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 1655672

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5004.7 s]
/proc/loadavg: 8.06 8.05 8.00 9/179 16154
/proc/meminfo: memFree=12060124/32951124 swapFree=67111528/67111528
[pid=15994] ppid=15992 vsize=1655672 CPUtime=40000.7 cores=0-7
/proc/15994/stat : 15994 (plingeling) S 15992 15994 15925 0 -1 4202496 8925792 1253 0 0 3998550 1520 0 0 25 0 9 0 512417979 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15994/statm: 413918 363282 181 55 0 411804 0
[pid=15994/tid=16001] ppid=15992 vsize=1655672 CPUtime=5003.46 cores=0-7
/proc/15994/task/16001/stat : 16001 (plingeling) R 15992 15994 15925 0 -1 4202560 1037062 1253 0 0 500165 181 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=15994/tid=16002] ppid=15992 vsize=1655672 CPUtime=4976.84 cores=0-7
/proc/15994/task/16002/stat : 16002 (plingeling) R 15992 15994 15925 0 -1 4202560 1165419 1253 0 0 497476 208 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15994/tid=16003] ppid=15992 vsize=1655672 CPUtime=5002.85 cores=0-7
/proc/15994/task/16003/stat : 16003 (plingeling) R 15992 15994 15925 0 -1 4202560 1014048 1253 0 0 500118 167 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=15994/tid=16004] ppid=15992 vsize=1655672 CPUtime=5003.31 cores=0-7
/proc/15994/task/16004/stat : 16004 (plingeling) R 15992 15994 15925 0 -1 4202560 1304400 1253 0 0 500092 239 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364973 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=15994/tid=16005] ppid=15992 vsize=1655672 CPUtime=5002.76 cores=0-7
/proc/15994/task/16005/stat : 16005 (plingeling) R 15992 15994 15925 0 -1 4202560 1288109 1253 0 0 500067 209 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15994/tid=16006] ppid=15992 vsize=1655672 CPUtime=5003.45 cores=0-7
/proc/15994/task/16006/stat : 16006 (plingeling) R 15992 15994 15925 0 -1 4202560 1118261 1253 0 0 500141 204 0 0 25 0 9 0 512418112 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15994/tid=16007] ppid=15992 vsize=1655672 CPUtime=5003.32 cores=0-7
/proc/15994/task/16007/stat : 16007 (plingeling) R 15992 15994 15925 0 -1 4202560 1147604 1253 0 0 500147 185 0 0 25 0 9 0 512418114 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4357066 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=15994/tid=16008] ppid=15992 vsize=1655672 CPUtime=5003.3 cores=0-7
/proc/15994/task/16008/stat : 16008 (plingeling) R 15992 15994 15925 0 -1 4202560 779630 1253 0 0 500215 115 0 0 25 0 9 0 512418118 1695408128 363282 33554432000 4194304 4417332 140734484598320 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 1655672

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5004.78
CPU time (s): 40000.8
CPU user time (s): 39985.6
CPU system time (s): 15.2767
CPU usage (%): 799.252
Max. virtual memory (cumulated for all children) (KiB): 1974844

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39985.6
system time used= 15.2767
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 8927045
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 560
involuntary context switches= 189881

runsolver used 8.09277 second user time and 18.5322 second system time

The end

Launcher Data 

Begin job on node138 at 2011-04-28 18:30:00
IDJOB=3309750
IDBENCH=82493
IDSOLVER=1695
FILE ID=node138/3309750-1304008200
RUNJOBID= node138-1304008069-15939
PBS_JOBID= 13168840
Free space on /tmp= 71808 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k50-eijkbs6669-tseitin.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3309750-1304008200/watcher-3309750-1304008200 -o /tmp/evaluation-result-3309750-1304008200/solver-3309750-1304008200 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3309750-1304008200.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 7886cd26d6e0801ea382a1ffd6224023
RANDOM SEED=1928600552

node138.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      13514012 kB
Buffers:       2089252 kB
Cached:       16716192 kB
SwapCached:          0 kB
Active:       10717172 kB
Inactive:      8151776 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      13514012 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           10112 kB
Writeback:           0 kB
AnonPages:       63828 kB
Mapped:          14876 kB
Slab:           503540 kB
PageTables:       4228 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   188328 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71804 MiB
End job on node138 at 2011-04-28 19:53:27