Trace number 3394438

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40001.3 5009.65

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-5x41.cnf
MD5SUM420773b7b438dd18c8a88a15fb5bcae1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables820
Number of clauses34235
Sum of the clauses size134480
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21230
Number of clauses of size 30
Number of clauses of size 433005
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394438-1306096033/watcher-3394438-1306096033 -o /tmp/evaluation-result-3394438-1306096033/solver-3394438-1306096033 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394438-1306096033.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.53 7.94 7.96 2/379 30009
/proc/meminfo: memFree=10634632/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=12368 CPUtime=0 cores=0-7
/proc/30009/stat : 30009 (plingeling) R 30007 30009 29863 0 -1 4202496 1169 1251 0 0 0 0 0 0 25 0 1 0 721206028 12664832 1099 33554432000 4194304 4417452 140735452289504 18446744073709551615 218370957056 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/30009/statm: 3092 1099 148 55 0 978 0

[startup+0.0844799 s]
/proc/loadavg: 7.53 7.94 7.96 2/379 30009
/proc/meminfo: memFree=10634632/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=178708 CPUtime=0.07 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 6965 1251 0 0 7 0 0 0 25 0 9 0 721206028 182996992 5848 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 44677 5848 169 55 0 42563 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 178708

[startup+0.107471 s]
/proc/loadavg: 7.53 7.94 7.96 2/379 30009
/proc/meminfo: memFree=10634632/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=179016 CPUtime=0.22 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 7067 1251 0 0 22 0 0 0 25 0 9 0 721206028 183312384 5950 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 44754 5950 174 55 0 42640 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 179016

[startup+0.300403 s]
/proc/loadavg: 7.53 7.94 7.96 2/379 30009
/proc/meminfo: memFree=10634632/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=194640 CPUtime=1.74 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 11517 1251 0 0 172 2 0 0 25 0 9 0 721206028 199311360 9846 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 48660 9846 180 55 0 46546 0
Current children cumulated CPU time (s) 1.74
Current children cumulated vsize (KiB) 194640

[startup+0.700332 s]
/proc/loadavg: 7.53 7.94 7.96 2/379 30009
/proc/meminfo: memFree=10634632/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=211260 CPUtime=4.94 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 16518 1251 0 0 491 3 0 0 25 0 9 0 721206028 216330240 14028 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 52815 14028 180 55 0 50701 0
Current children cumulated CPU time (s) 4.94
Current children cumulated vsize (KiB) 211260

[startup+1.50119 s]
/proc/loadavg: 7.53 7.94 7.96 10/388 30023
/proc/meminfo: memFree=10584952/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=227820 CPUtime=11.32 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 38894 1251 0 0 1125 7 0 0 25 0 9 0 721206028 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 56955 19121 180 55 0 54841 0
[pid=30009/tid=30016] ppid=30007 vsize=227820 CPUtime=1.4 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 2363 1251 0 0 140 0 0 0 19 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4321595 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=227820 CPUtime=1.4 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 2606 1251 0 0 140 0 0 0 25 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4354492 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=227820 CPUtime=1.34 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 5050 1251 0 0 134 0 0 0 25 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4236878 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=227820 CPUtime=1.4 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 5124 1251 0 0 140 0 0 0 25 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=227820 CPUtime=1.4 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3360 1251 0 0 140 0 0 0 19 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=227820 CPUtime=1.41 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 4309 1251 0 0 140 1 0 0 25 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4255645 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=227820 CPUtime=1.41 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 5220 1251 0 0 140 1 0 0 25 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=227820 CPUtime=1.4 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 3957 1251 0 0 140 0 0 0 25 0 9 0 721206036 233287680 19121 33554432000 4194304 4417452 140735452289504 18446744073709551615 4354448 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.32
Current children cumulated vsize (KiB) 227820

[startup+3.10091 s]
/proc/loadavg: 7.57 7.94 7.96 9/388 30023
/proc/meminfo: memFree=10533492/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=289212 CPUtime=24.1 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 80113 1251 0 0 2394 16 0 0 25 0 9 0 721206028 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 72303 34025 180 55 0 70189 0
[pid=30009/tid=30016] ppid=30007 vsize=289212 CPUtime=3 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 6367 1251 0 0 299 1 0 0 18 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=289212 CPUtime=3 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 8155 1251 0 0 299 1 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366090 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=289212 CPUtime=2.93 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 9176 1251 0 0 292 1 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=289212 CPUtime=3.01 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 9443 1251 0 0 299 2 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366079 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=289212 CPUtime=3.01 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 9536 1251 0 0 299 2 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366113 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=289212 CPUtime=3 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 10869 1251 0 0 298 2 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=289212 CPUtime=3.01 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 8578 1251 0 0 300 1 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=289212 CPUtime=2.99 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 11084 1251 0 0 298 1 0 0 25 0 9 0 721206036 296153088 34025 33554432000 4194304 4417452 140735452289504 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.1
Current children cumulated vsize (KiB) 289212

[startup+6.30034 s]
/proc/loadavg: 7.57 7.94 7.96 9/388 30023
/proc/meminfo: memFree=10436352/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=384432 CPUtime=49.66 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 163100 1251 0 0 4931 35 0 0 25 0 9 0 721206028 393658368 57940 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 96108 57966 180 55 0 93994 0
[pid=30009/tid=30016] ppid=30007 vsize=384432 CPUtime=6.2 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 13770 1251 0 0 618 2 0 0 25 0 9 0 721206036 393658368 57980 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364893 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=384432 CPUtime=6.2 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 22977 1251 0 0 616 4 0 0 25 0 9 0 721206036 393658368 57996 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=384432 CPUtime=6.1 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 20277 1251 0 0 606 4 0 0 25 0 9 0 721206036 393658368 58011 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=384432 CPUtime=6.2 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 16096 1251 0 0 617 3 0 0 25 0 9 0 721206036 393658368 58026 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=384432 CPUtime=6.2 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 20153 1251 0 0 616 4 0 0 25 0 9 0 721206036 393658368 58042 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=384432 CPUtime=6.2 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 21270 1251 0 0 615 5 0 0 25 0 9 0 721206036 393658368 58058 33554432000 4194304 4417452 140735452289504 18446744073709551615 218370651199 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=384432 CPUtime=6.2 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 16539 1251 0 0 618 2 0 0 25 0 9 0 721206036 393658368 58072 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=384432 CPUtime=6.19 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 25232 1251 0 0 614 5 0 0 25 0 9 0 721206036 393658368 58088 33554432000 4194304 4417452 140735452289504 18446744073709551615 4365004 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.66
Current children cumulated vsize (KiB) 384432

[startup+12.7012 s]
/proc/loadavg: 7.63 7.94 7.96 9/388 30023
/proc/meminfo: memFree=10239148/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=567488 CPUtime=100.77 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 310695 1251 0 0 10010 67 0 0 25 0 9 0 721206028 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 141872 104536 180 55 0 139758 0
[pid=30009/tid=30016] ppid=30007 vsize=567488 CPUtime=12.6 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 36105 1251 0 0 1254 6 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=567488 CPUtime=12.6 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 38807 1251 0 0 1251 9 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=567488 CPUtime=12.42 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 38916 1251 0 0 1233 9 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=567488 CPUtime=12.6 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 29820 1251 0 0 1254 6 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=567488 CPUtime=12.6 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 44120 1251 0 0 1251 9 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=567488 CPUtime=12.6 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 37734 1251 0 0 1252 8 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364772 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=567488 CPUtime=12.6 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 31330 1251 0 0 1254 6 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=567488 CPUtime=12.59 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 46958 1251 0 0 1250 9 0 0 25 0 9 0 721206036 581107712 104536 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.77
Current children cumulated vsize (KiB) 567488

[startup+25.5009 s]
/proc/loadavg: 7.69 7.94 7.96 9/388 30024
/proc/meminfo: memFree=9934968/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=891540 CPUtime=202.98 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 562084 1251 0 0 20179 119 0 0 25 0 9 0 721206028 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 222885 185843 180 55 0 220771 0
[pid=30009/tid=30016] ppid=30007 vsize=891540 CPUtime=25.41 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 70172 1251 0 0 2528 13 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=891540 CPUtime=25.4 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 61939 1251 0 0 2527 13 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=891540 CPUtime=25.05 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 60885 1251 0 0 2491 14 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364854 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=891540 CPUtime=25.4 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 39967 1251 0 0 2532 8 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=891540 CPUtime=25.4 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 100940 1251 0 0 2520 20 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=891540 CPUtime=25.39 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 76853 1251 0 0 2523 16 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=891540 CPUtime=25.4 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 58568 1251 0 0 2527 13 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=891540 CPUtime=25.38 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 85855 1251 0 0 2520 18 0 0 25 0 9 0 721206036 912936960 185843 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.98
Current children cumulated vsize (KiB) 891540

[startup+51.1014 s]
/proc/loadavg: 7.79 7.94 7.96 9/388 30024
/proc/meminfo: memFree=9497644/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=1292268 CPUtime=407.46 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 980068 1251 0 0 40544 202 0 0 25 0 9 0 721206028 1323282432 285825 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 323067 285825 180 55 0 320953 0
[pid=30009/tid=30016] ppid=30007 vsize=1292268 CPUtime=51 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 109870 1251 0 0 5080 20 0 0 25 0 9 0 721206036 1323282432 285825 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=1292268 CPUtime=51 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 111669 1251 0 0 5077 23 0 0 25 0 9 0 721206036 1323282432 285825 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=1292268 CPUtime=50.33 cores=0-7

################
# More data... #
################

[pid=30009/tid=30018] ppid=30007 vsize=16708548 CPUtime=4678.39 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 1210845 1251 0 0 467597 242 0 0 25 0 9 0 721206036 17109553152 4145069 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=16708548 CPUtime=4722.34 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 1099329 1251 0 0 471930 304 0 0 25 0 9 0 721206036 17109553152 4145069 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=16708548 CPUtime=4705.83 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3122416 1251 0 0 469971 612 0 0 25 0 9 0 721206036 17109553152 4145069 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=16708548 CPUtime=4720.37 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 2034616 1251 0 0 471495 542 0 0 25 0 9 0 721206036 17109553152 4145069 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=16708548 CPUtime=4721.77 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 4477404 1251 0 0 471094 1083 0 0 25 0 9 0 721206036 17109553152 4145069 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=16708548 CPUtime=4722.24 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 2415481 1251 0 0 471676 548 0 0 25 0 9 0 721206036 17109553152 4145069 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37712.8
Current children cumulated vsize (KiB) 16708548

[startup+4782.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/388 30162
/proc/meminfo: memFree=97788/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=17656792 CPUtime=38191.1 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 21414720 1251 0 0 3814320 4794 0 0 25 0 9 0 721206028 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 4414198 4382130 180 55 0 4412084 0
[pid=30009/tid=30016] ppid=30007 vsize=17656792 CPUtime=4780.29 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 2593949 1251 0 0 477464 565 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=17656792 CPUtime=4781.22 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 4295414 1251 0 0 477290 832 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=17656792 CPUtime=4738.02 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 1305405 1251 0 0 473523 279 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=17656792 CPUtime=4782.22 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 1099329 1251 0 0 477918 304 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=17656792 CPUtime=4765.06 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3122416 1251 0 0 475894 612 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=17656792 CPUtime=4780.36 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 2034616 1251 0 0 477494 542 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366113 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=17656792 CPUtime=4781.59 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 4477404 1251 0 0 477076 1083 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=17656792 CPUtime=4782.23 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 2479282 1251 0 0 477650 573 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38191.1
Current children cumulated vsize (KiB) 17656792

[startup+4842.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/388 30164
/proc/meminfo: memFree=97988/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=17656792 CPUtime=38670.4 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 21414720 1251 0 0 3862242 4794 0 0 25 0 9 0 721206028 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 4414198 4382130 180 55 0 4412084 0
[pid=30009/tid=30016] ppid=30007 vsize=17656792 CPUtime=4840.29 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 2593949 1251 0 0 483464 565 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=17656792 CPUtime=4841.22 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 4295414 1251 0 0 483290 832 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=17656792 CPUtime=4798.02 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 1305405 1251 0 0 479523 279 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=17656792 CPUtime=4842.23 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 1099329 1251 0 0 483919 304 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4366391 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=17656792 CPUtime=4824.3 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3122416 1251 0 0 481818 612 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=17656792 CPUtime=4840.36 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 2034616 1251 0 0 483494 542 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364675 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=17656792 CPUtime=4841.57 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 4477404 1251 0 0 483074 1083 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=17656792 CPUtime=4842.23 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 2479282 1251 0 0 483650 573 0 0 25 0 9 0 721206036 18080555008 4382130 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38670.4
Current children cumulated vsize (KiB) 17656792

[startup+4902.3 s]
/proc/loadavg: 8.00 8.01 8.00 11/388 30166
/proc/meminfo: memFree=94504/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=18246964 CPUtime=39149 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 21562263 1251 0 0 3910054 4844 0 0 25 0 9 0 721206028 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 4561741 4529673 180 55 0 4559627 0
[pid=30009/tid=30016] ppid=30007 vsize=18246964 CPUtime=4899.94 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 2593949 1251 0 0 489429 565 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=18246964 CPUtime=4901.23 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 4295414 1251 0 0 489291 832 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=18246964 CPUtime=4857.89 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 1305405 1251 0 0 485510 279 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=18246964 CPUtime=4902.22 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 1146642 1251 0 0 489909 313 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=18246964 CPUtime=4883.54 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3122416 1251 0 0 487742 612 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=18246964 CPUtime=4900.31 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 2034616 1251 0 0 489489 542 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=18246964 CPUtime=4901.48 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 4577634 1251 0 0 489024 1124 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=18246964 CPUtime=4902.23 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 2479282 1251 0 0 489650 573 0 0 25 0 9 0 721206036 18684891136 4529673 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39149
Current children cumulated vsize (KiB) 18246964

[startup+4962.3 s]
/proc/loadavg: 8.00 8.01 8.00 9/388 30167
/proc/meminfo: memFree=566872/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=17776080 CPUtime=39628.2 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 21771944 1251 0 0 3957949 4873 0 0 25 0 9 0 721206028 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 4444020 4411952 180 55 0 4441906 0
[pid=30009/tid=30016] ppid=30007 vsize=17776080 CPUtime=4959.95 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 2593949 1251 0 0 495430 565 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=17776080 CPUtime=4961.23 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 4295414 1251 0 0 495291 832 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=17776080 CPUtime=4917.89 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 1515086 1251 0 0 491481 308 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=17776080 CPUtime=4962.22 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 1146642 1251 0 0 495909 313 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=17776080 CPUtime=4942.77 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3122416 1251 0 0 493665 612 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=17776080 CPUtime=4960.32 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 2034616 1251 0 0 495490 542 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=17776080 CPUtime=4961.47 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 4577634 1251 0 0 495023 1124 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=17776080 CPUtime=4962.23 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 2479282 1251 0 0 495650 573 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39628.2
Current children cumulated vsize (KiB) 17776080



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.9 s]
/proc/loadavg: 8.00 8.00 8.00 9/388 30168
/proc/meminfo: memFree=567432/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=17776080 CPUtime=40000.5 cores=0-7
/proc/30009/stat : 30009 (plingeling) S 30007 30009 29863 0 -1 4202496 21771944 1251 0 0 3995179 4873 0 0 25 0 9 0 721206028 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 4444020 4411952 180 55 0 4441906 0
[pid=30009/tid=30016] ppid=30007 vsize=17776080 CPUtime=5006.56 cores=0-7
/proc/30009/task/30016/stat : 30016 (plingeling) R 30007 30009 29863 0 -1 4202560 2593949 1251 0 0 500091 565 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30009/tid=30017] ppid=30007 vsize=17776080 CPUtime=5007.84 cores=0-7
/proc/30009/task/30017/stat : 30017 (plingeling) R 30007 30009 29863 0 -1 4202560 4295414 1251 0 0 499952 832 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30009/tid=30018] ppid=30007 vsize=17776080 CPUtime=4964.5 cores=0-7
/proc/30009/task/30018/stat : 30018 (plingeling) R 30007 30009 29863 0 -1 4202560 1515086 1251 0 0 496142 308 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30009/tid=30019] ppid=30007 vsize=17776080 CPUtime=5008.84 cores=0-7
/proc/30009/task/30019/stat : 30019 (plingeling) R 30007 30009 29863 0 -1 4202560 1146642 1251 0 0 500571 313 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=17776080 CPUtime=4988.79 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4202560 3122416 1251 0 0 498267 612 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30009/tid=30021] ppid=30007 vsize=17776080 CPUtime=5006.93 cores=0-7
/proc/30009/task/30021/stat : 30021 (plingeling) R 30007 30009 29863 0 -1 4202560 2034616 1251 0 0 500151 542 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30009/tid=30022] ppid=30007 vsize=17776080 CPUtime=5008.07 cores=0-7
/proc/30009/task/30022/stat : 30022 (plingeling) R 30007 30009 29863 0 -1 4202560 4577634 1251 0 0 499683 1124 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30009/tid=30023] ppid=30007 vsize=17776080 CPUtime=5008.85 cores=0-7
/proc/30009/task/30023/stat : 30023 (plingeling) R 30007 30009 29863 0 -1 4202560 2479282 1251 0 0 500312 573 0 0 25 0 9 0 721206036 18202705920 4411952 33554432000 4194304 4417452 140735452289504 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 17776080

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.41 s]
/proc/loadavg: 8.00 8.00 8.00 2/382 30169
/proc/meminfo: memFree=9942996/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=0 CPUtime=40001.1 cores=0-7
/proc/30009/stat : 30009 (plingeling) Z 30007 30009 29863 0 -1 4203548 21771944 1251 0 0 3995183 4923 0 0 18 0 2 0 721206028 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 0 0 0 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=0 CPUtime=4989.28 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4203588 3122416 1251 0 0 498267 661 0 0 25 0 2 0 721206036 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 0

[startup+5009.6 s]
/proc/loadavg: 8.00 8.00 8.00 2/382 30169
/proc/meminfo: memFree=9942996/32951124 swapFree=52307820/67111528
[pid=30009] ppid=30007 vsize=0 CPUtime=40001.2 cores=0-7
/proc/30009/stat : 30009 (plingeling) Z 30007 30009 29863 0 -1 4203548 21771944 1251 0 0 3995183 4941 0 0 18 0 2 0 721206028 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/30009/statm: 0 0 0 0 0 0 0
[pid=30009/tid=30020] ppid=30007 vsize=0 CPUtime=4989.47 cores=0-7
/proc/30009/task/30020/stat : 30020 (plingeling) R 30007 30009 29863 0 -1 4203588 3122416 1251 0 0 498267 680 0 0 25 0 2 0 721206036 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40001.2
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5009.65
CPU time (s): 40001.3
CPU user time (s): 39951.8
CPU system time (s): 49.4765
CPU usage (%): 798.484
Max. virtual memory (cumulated for all children) (KiB): 18246964

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39951.8
system time used= 49.4765
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 21773195
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 471
involuntary context switches= 195603

runsolver used 18.3892 second user time and 45.1841 second system time

The end

Launcher Data 

Begin job on node101 at 2011-05-22 22:27:13
IDJOB=3394438
IDBENCH=83163
IDSOLVER=1851
FILE ID=node101/3394438-1306096033
RUNJOBID= node101-1306096028-29877
PBS_JOBID= 13387973
Free space on /tmp= 73576 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-5x41.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394438-1306096033/watcher-3394438-1306096033 -o /tmp/evaluation-result-3394438-1306096033/solver-3394438-1306096033 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394438-1306096033.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 420773b7b438dd18c8a88a15fb5bcae1
RANDOM SEED=104321313

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      10635036 kB
Buffers:        370456 kB
Cached:       10442092 kB
SwapCached:      14796 kB
Active:       14806716 kB
Inactive:      7193088 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      10635036 kB
SwapTotal:    67111528 kB
SwapFree:     52307820 kB
Dirty:            1484 kB
Writeback:           0 kB
AnonPages:    11184588 kB
Mapped:          20488 kB
Slab:           172988 kB
PageTables:      74128 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 78580244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73576 MiB
End job on node101 at 2011-05-22 23:50:44