Trace number 3372440

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.8 5007.39

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
openstacks-sequencedstrips-nonadl-nonnegated-os-sequencedstrips-p30_3.025-NOTKNOWN.cnf
MD5SUM3c5c73741ba21d5f39c7f436ba834743
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables95456
Number of clauses483561
Sum of the clauses size1149911
Maximum clause length63
Minimum clause length1
Number of clauses of size 1211
Number of clauses of size 2477300
Number of clauses of size 31500
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 54550

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372440-1305437528/watcher-3372440-1305437528 -o /tmp/evaluation-result-3372440-1305437528/solver-3372440-1305437528 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372440-1305437528.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.54 2.64 3.03 1/288 30938
/proc/meminfo: memFree=27546896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=11080 CPUtime=0 cores=0-7
/proc/30938/stat : 30938 (plingeling) R 30936 30938 30869 0 -1 4202496 838 1257 0 0 0 0 0 0 21 0 1 0 655351305 11345920 769 33554432000 4194304 4417452 140734010835136 18446744073709551615 210204647168 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/30938/statm: 2770 769 144 55 0 656 0

[startup+0.073147 s]
/proc/loadavg: 3.54 2.64 3.03 1/288 30938
/proc/meminfo: memFree=27546896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=22212 CPUtime=0.06 cores=0-7
/proc/30938/stat : 30938 (plingeling) R 30936 30938 30869 0 -1 4202496 3646 1257 0 0 6 0 0 0 21 0 1 0 655351305 22745088 3577 33554432000 4194304 4417452 140734010835136 18446744073709551615 210204265984 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/30938/statm: 5553 3577 149 55 0 3439 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 22212

[startup+0.101154 s]
/proc/loadavg: 3.54 2.64 3.03 1/288 30938
/proc/meminfo: memFree=27546896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=26968 CPUtime=0.08 cores=0-7
/proc/30938/stat : 30938 (plingeling) R 30936 30938 30869 0 -1 4202496 4835 1257 0 0 8 0 0 0 21 0 1 0 655351305 27615232 4766 33554432000 4194304 4417452 140734010835136 18446744073709551615 4236012 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/30938/statm: 6742 4766 149 55 0 4628 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26968

[startup+0.301104 s]
/proc/loadavg: 3.54 2.64 3.03 1/288 30938
/proc/meminfo: memFree=27546896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=62496 CPUtime=0.29 cores=0-7
/proc/30938/stat : 30938 (plingeling) R 30936 30938 30869 0 -1 4202496 13717 1257 0 0 28 1 0 0 22 0 1 0 655351305 63995904 13648 33554432000 4194304 4417452 140734010835136 18446744073709551615 4231499 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/30938/statm: 15624 13648 149 55 0 13510 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 62496

[startup+0.701058 s]
/proc/loadavg: 3.54 2.64 3.03 1/288 30938
/proc/meminfo: memFree=27546896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=128128 CPUtime=0.69 cores=0-7
/proc/30938/stat : 30938 (plingeling) R 30936 30938 30869 0 -1 4202496 30141 1257 0 0 66 3 0 0 23 0 1 0 655351305 131203072 30056 33554432000 4194304 4417452 140734010835136 18446744073709551615 4209566 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/30938/statm: 32032 30056 149 55 0 29918 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 128128

[startup+1.50087 s]
/proc/loadavg: 3.54 2.64 3.03 9/297 30952
/proc/meminfo: memFree=27423080/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=320524 CPUtime=6.12 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 60061 1257 0 0 605 7 0 0 18 0 9 0 655351305 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 80131 31918 175 55 0 78017 0
[pid=30938/tid=30945] ppid=30936 vsize=320524 CPUtime=0.71 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 6795 1257 0 0 71 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4287486 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=320524 CPUtime=0.71 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 6425 1257 0 0 71 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4286414 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=320524 CPUtime=0.68 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 5941 1257 0 0 68 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4287459 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=320524 CPUtime=0.7 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 1258 1257 0 0 70 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4358093 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=320524 CPUtime=0.68 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 1780 1257 0 0 68 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4301825 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=320524 CPUtime=0.66 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 3663 1257 0 0 66 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4290912 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=320524 CPUtime=0.54 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 2152 1257 0 0 54 0 0 0 18 0 9 0 655351381 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4288161 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=320524 CPUtime=0.6 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 1889 1257 0 0 60 0 0 0 25 0 9 0 655351385 328216576 31918 33554432000 4194304 4417452 140734010835136 18446744073709551615 4286493 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 6.12
Current children cumulated vsize (KiB) 320524

[startup+3.1006 s]
/proc/loadavg: 3.54 2.64 3.03 9/297 30952
/proc/meminfo: memFree=27417128/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=326888 CPUtime=18.9 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 78296 1257 0 0 1881 9 0 0 18 0 9 0 655351305 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 81722 33503 180 55 0 79608 0
[pid=30938/tid=30945] ppid=30936 vsize=326888 CPUtime=2.31 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 13479 1257 0 0 230 1 0 0 21 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4253176 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=326888 CPUtime=2.31 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 11594 1257 0 0 230 1 0 0 21 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4253198 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=326888 CPUtime=2.28 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 7535 1257 0 0 228 0 0 0 22 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=326888 CPUtime=2.3 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 1508 1257 0 0 230 0 0 0 21 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=326888 CPUtime=2.28 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 1785 1257 0 0 228 0 0 0 21 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4273621 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=326888 CPUtime=2.25 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 6073 1257 0 0 225 0 0 0 21 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4261973 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=326888 CPUtime=2.14 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3249 1257 0 0 214 0 0 0 21 0 9 0 655351381 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4258491 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=326888 CPUtime=2.2 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2915 1257 0 0 220 0 0 0 25 0 9 0 655351385 334733312 33503 33554432000 4194304 4417452 140734010835136 18446744073709551615 4253075 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 18.9
Current children cumulated vsize (KiB) 326888

[startup+6.30107 s]
/proc/loadavg: 3.90 2.73 3.06 9/297 30952
/proc/meminfo: memFree=27393592/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=344072 CPUtime=44.47 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 83963 1257 0 0 4437 10 0 0 18 0 9 0 655351305 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 86018 38716 180 55 0 83904 0
[pid=30938/tid=30945] ppid=30936 vsize=344072 CPUtime=5.51 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 14366 1257 0 0 550 1 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4320821 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=344072 CPUtime=5.51 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 11595 1257 0 0 550 1 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4232664 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=344072 CPUtime=5.48 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 7926 1257 0 0 548 0 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=344072 CPUtime=5.5 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 4343 1257 0 0 550 0 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4358947 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=344072 CPUtime=5.48 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 1786 1257 0 0 548 0 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=344072 CPUtime=5.42 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 6747 1257 0 0 542 0 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=344072 CPUtime=5.34 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3505 1257 0 0 534 0 0 0 25 0 9 0 655351381 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=344072 CPUtime=5.4 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 3537 1257 0 0 540 0 0 0 25 0 9 0 655351385 352329728 38716 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 44.47
Current children cumulated vsize (KiB) 344072

[startup+12.701 s]
/proc/loadavg: 4.23 2.82 3.09 9/297 30952
/proc/meminfo: memFree=27305084/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=400588 CPUtime=95.61 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 115958 1257 0 0 9546 15 0 0 18 0 9 0 655351305 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 100147 60388 180 55 0 98033 0
[pid=30938/tid=30945] ppid=30936 vsize=400588 CPUtime=11.91 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 22208 1257 0 0 1189 2 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=400588 CPUtime=11.91 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 17861 1257 0 0 1189 2 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4357024 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=400588 CPUtime=11.87 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 10180 1257 0 0 1187 0 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4339846 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=400588 CPUtime=11.9 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 12997 1257 0 0 1188 2 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4356867 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=400588 CPUtime=11.88 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 2961 1257 0 0 1188 0 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4307930 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=400588 CPUtime=11.76 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 8262 1257 0 0 1175 1 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4236870 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=400588 CPUtime=11.74 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 6122 1257 0 0 1173 1 0 0 25 0 9 0 655351381 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=400588 CPUtime=11.8 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 5209 1257 0 0 1180 0 0 0 25 0 9 0 655351385 410202112 60388 33554432000 4194304 4417452 140734010835136 18446744073709551615 4237180 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 95.61
Current children cumulated vsize (KiB) 400588

[startup+25.5009 s]
/proc/loadavg: 5.06 3.07 3.17 9/297 30953
/proc/meminfo: memFree=27191376/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=552600 CPUtime=197.89 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 193511 1257 0 0 19762 27 0 0 18 0 9 0 655351305 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 138150 89341 180 55 0 136036 0
[pid=30938/tid=30945] ppid=30936 vsize=552600 CPUtime=24.7 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 41634 1257 0 0 2465 5 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4320800 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=552600 CPUtime=24.71 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 32320 1257 0 0 2468 3 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4277545 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=552600 CPUtime=24.67 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 10714 1257 0 0 2467 0 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4264909 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=552600 CPUtime=24.7 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 37196 1257 0 0 2464 6 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364558 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=552600 CPUtime=24.67 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 4661 1257 0 0 2467 0 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=552600 CPUtime=24.44 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 11888 1257 0 0 2443 1 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=552600 CPUtime=24.54 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 11235 1257 0 0 2452 2 0 0 25 0 9 0 655351381 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=552600 CPUtime=24.6 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 13705 1257 0 0 2459 1 0 0 25 0 9 0 655351385 565862400 89341 33554432000 4194304 4417452 140734010835136 18446744073709551615 4237194 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 197.89
Current children cumulated vsize (KiB) 552600

[startup+51.1006 s]
/proc/loadavg: 6.14 3.48 3.30 9/297 30954
/proc/meminfo: memFree=27070764/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=681524 CPUtime=402.44 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 387394 1257 0 0 40184 60 0 0 18 0 9 0 655351305 697880576 124529 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 170381 124529 180 55 0 168267 0
[pid=30938/tid=30945] ppid=30936 vsize=681524 CPUtime=50.3 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 74720 1257 0 0 5019 11 0 0 25 0 9 0 655351381 697880576 124529 33554432000 4194304 4417452 140734010835136 18446744073709551615 4366657 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=681524 CPUtime=50.31 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 51527 1257 0 0 5025 6 0 0 25 0 9 0 655351381 697880576 124529 33554432000 4194304 4417452 140734010835136 18446744073709551615 4237002 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=681524 CPUtime=50.27 cores=0-7

################
# More data... #
################

/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2786785 1257 0 0 471543 541 0 0 25 0 9 0 655351385 22543577088 5467256 33554432000 4194304 4417452 140734010835136 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37727.5
Current children cumulated vsize (KiB) 22015212

[startup+4782.3 s]
/proc/loadavg: 8.07 8.02 7.94 9/297 31107
/proc/meminfo: memFree=6823040/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=20830440 CPUtime=38206.9 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 27849583 1257 0 0 3814930 5756 0 0 18 0 9 0 655351305 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 5207610 5171063 180 55 0 5205496 0
[pid=30938/tid=30945] ppid=30936 vsize=20830440 CPUtime=4760.99 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 3512926 1257 0 0 475446 653 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4263109 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=20830440 CPUtime=4781.48 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 3260191 1257 0 0 477463 685 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=20830440 CPUtime=4780.55 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 2988111 1257 0 0 477429 626 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=20830440 CPUtime=4781.42 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 5057356 1257 0 0 477028 1114 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=20830440 CPUtime=4781.26 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 2316309 1257 0 0 477646 480 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=20830440 CPUtime=4758.18 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 4032013 1257 0 0 475009 809 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4309899 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=20830440 CPUtime=4781.31 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3818588 1257 0 0 477300 831 0 0 25 0 9 0 655351381 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4355841 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=20830440 CPUtime=4780.83 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2833931 1257 0 0 477533 550 0 0 25 0 9 0 655351385 21330370560 5171063 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364446 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38206.9
Current children cumulated vsize (KiB) 20830440

[startup+4842.3 s]
/proc/loadavg: 8.07 8.03 7.94 9/297 31108
/proc/meminfo: memFree=7643764/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=20009796 CPUtime=38686.3 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 28115660 1257 0 0 3862825 5806 0 0 18 0 9 0 655351305 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 5002449 4965902 180 55 0 5000335 0
[pid=30938/tid=30945] ppid=30936 vsize=20009796 CPUtime=4820.99 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 3583059 1257 0 0 481432 667 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4263172 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=20009796 CPUtime=4841.48 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 3260191 1257 0 0 483463 685 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4199406 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=20009796 CPUtime=4840.54 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 3020672 1257 0 0 483419 635 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=20009796 CPUtime=4841.42 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 5057356 1257 0 0 483028 1114 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=20009796 CPUtime=4841.26 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 2316322 1257 0 0 483646 480 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=20009796 CPUtime=4817.63 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 4186791 1257 0 0 480929 834 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=20009796 CPUtime=4841.31 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3818588 1257 0 0 483299 832 0 0 25 0 9 0 655351381 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364696 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=20009796 CPUtime=4840.83 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2842523 1257 0 0 483532 551 0 0 25 0 9 0 655351385 20490031104 4965902 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38686.3
Current children cumulated vsize (KiB) 20009796

[startup+4902.3 s]
/proc/loadavg: 8.02 8.02 7.94 9/297 31110
/proc/meminfo: memFree=7012200/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=20624072 CPUtime=39165.8 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 28414920 1257 0 0 3910697 5880 0 0 18 0 9 0 655351305 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 5156018 5123535 180 55 0 5153904 0
[pid=30938/tid=30945] ppid=30936 vsize=20624072 CPUtime=4880.99 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 3600893 1257 0 0 487429 670 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=20624072 CPUtime=4901.49 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 3260191 1257 0 0 489463 686 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=20624072 CPUtime=4900.53 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 3262383 1257 0 0 489354 699 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=20624072 CPUtime=4901.43 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 5057356 1257 0 0 489029 1114 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=20624072 CPUtime=4901.25 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 2327106 1257 0 0 489642 483 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364861 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=20624072 CPUtime=4877.09 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 4202116 1257 0 0 486873 836 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=20624072 CPUtime=4901.32 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3818602 1257 0 0 489300 832 0 0 25 0 9 0 655351381 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=20624072 CPUtime=4900.83 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2856115 1257 0 0 489531 552 0 0 25 0 9 0 655351385 21119049728 5123535 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364958 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39165.8
Current children cumulated vsize (KiB) 20624072

[startup+4962.3 s]
/proc/loadavg: 8.01 8.02 7.94 9/297 31111
/proc/meminfo: memFree=6841048/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=20855856 CPUtime=39645.2 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 28476963 1257 0 0 3958625 5896 0 0 18 0 9 0 655351305 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 5213964 5169194 180 55 0 5211850 0
[pid=30938/tid=30945] ppid=30936 vsize=20855856 CPUtime=4940.99 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 3609214 1257 0 0 493428 671 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=20855856 CPUtime=4961.49 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 3260191 1257 0 0 495463 686 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=20855856 CPUtime=4960.52 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 3262383 1257 0 0 495353 699 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=20855856 CPUtime=4961.43 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 5057356 1257 0 0 495029 1114 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4366400 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=20855856 CPUtime=4961.26 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 2359922 1257 0 0 495633 493 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=20855856 CPUtime=4936.54 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 4214190 1257 0 0 492815 839 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=20855856 CPUtime=4961.32 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3818602 1257 0 0 495300 832 0 0 25 0 9 0 655351381 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=20855856 CPUtime=4960.83 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2864947 1257 0 0 495530 553 0 0 25 0 9 0 655351385 21356396544 5169194 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39645.2
Current children cumulated vsize (KiB) 20855856



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.7 s]
/proc/loadavg: 8.00 8.01 7.94 9/297 31113
/proc/meminfo: memFree=10033896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=17639720 CPUtime=40000.1 cores=0-7
/proc/30938/stat : 30938 (plingeling) S 30936 30938 30869 0 -1 4202496 28623044 1257 0 0 3994079 5927 0 0 18 0 9 0 655351305 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 4409930 4369768 180 55 0 4407816 0
[pid=30938/tid=30945] ppid=30936 vsize=17639720 CPUtime=4985.4 cores=0-7
/proc/30938/task/30945/stat : 30945 (plingeling) R 30936 30938 30869 0 -1 4202560 3627446 1257 0 0 497866 674 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4342461 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30938/tid=30946] ppid=30936 vsize=17639720 CPUtime=5005.9 cores=0-7
/proc/30938/task/30946/stat : 30946 (plingeling) R 30936 30938 30869 0 -1 4202560 3326627 1257 0 0 499890 700 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30938/tid=30947] ppid=30936 vsize=17639720 CPUtime=5004.92 cores=0-7
/proc/30938/task/30947/stat : 30947 (plingeling) R 30936 30938 30869 0 -1 4202560 3262383 1257 0 0 499793 699 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=30938/tid=30948] ppid=30936 vsize=17639720 CPUtime=5005.84 cores=0-7
/proc/30938/task/30948/stat : 30948 (plingeling) R 30936 30938 30869 0 -1 4202560 5057356 1257 0 0 499470 1114 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30938/tid=30949] ppid=30936 vsize=17639720 CPUtime=5005.67 cores=0-7
/proc/30938/task/30949/stat : 30949 (plingeling) R 30936 30938 30869 0 -1 4202560 2359922 1257 0 0 500074 493 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=17639720 CPUtime=4980.55 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4202560 4230574 1257 0 0 497214 841 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30938/tid=30951] ppid=30936 vsize=17639720 CPUtime=5005.72 cores=0-7
/proc/30938/task/30951/stat : 30951 (plingeling) R 30936 30938 30869 0 -1 4202560 3849295 1257 0 0 499729 843 0 0 25 0 9 0 655351381 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30938/tid=30952] ppid=30936 vsize=17639720 CPUtime=5005.25 cores=0-7
/proc/30938/task/30952/stat : 30952 (plingeling) R 30936 30938 30869 0 -1 4202560 2879283 1257 0 0 499970 555 0 0 25 0 9 0 655351385 18063073280 4369768 33554432000 4194304 4417452 140734010835136 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 17639720

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007 s]
/proc/loadavg: 8.00 8.01 7.94 9/297 31113
/proc/meminfo: memFree=10033896/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=0 CPUtime=40000.4 cores=0-7
/proc/30938/stat : 30938 (plingeling) Z 30936 30938 30869 0 -1 4203548 28623044 1257 0 0 3994086 5955 0 0 18 0 2 0 655351305 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 0 0 0 0 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=0 CPUtime=4980.83 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4203588 4230574 1257 0 0 497214 869 0 0 25 0 2 0 655351381 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 0

[startup+5007.21 s]
/proc/loadavg: 8.00 8.01 7.94 2/291 31115
/proc/meminfo: memFree=19637456/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=0 CPUtime=40000.6 cores=0-7
/proc/30938/stat : 30938 (plingeling) Z 30936 30938 30869 0 -1 4203548 28623044 1257 0 0 3994086 5977 0 0 18 0 2 0 655351305 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 0 0 0 0 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=0 CPUtime=4981.04 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4203588 4230574 1257 0 0 497214 890 0 0 25 0 2 0 655351381 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 0

[startup+5007.3 s]
/proc/loadavg: 8.00 8.01 7.94 2/291 31115
/proc/meminfo: memFree=19637456/32951124 swapFree=63649232/67111528
[pid=30938] ppid=30936 vsize=0 CPUtime=40000.7 cores=0-7
/proc/30938/stat : 30938 (plingeling) Z 30936 30938 30869 0 -1 4203548 28623044 1257 0 0 3994086 5986 0 0 18 0 2 0 655351305 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 7 0 0 0
/proc/30938/statm: 0 0 0 0 0 0 0
[pid=30938/tid=30950] ppid=30936 vsize=0 CPUtime=4981.13 cores=0-7
/proc/30938/task/30950/stat : 30950 (plingeling) R 30936 30938 30869 0 -1 4203588 4230574 1257 0 0 497214 899 0 0 25 0 2 0 655351381 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5007.39
CPU time (s): 40000.8
CPU user time (s): 39940.9
CPU system time (s): 59.9549
CPU usage (%): 798.835
Max. virtual memory (cumulated for all children) (KiB): 23920696

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39940.9
system time used= 59.9549
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28624301
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1163
involuntary context switches= 191937

runsolver used 12.929 second user time and 32.615 second system time

The end

Launcher Data 

Begin job on node130 at 2011-05-15 07:32:08
IDJOB=3372440
IDBENCH=82381
IDSOLVER=1851
FILE ID=node130/3372440-1305437528
RUNJOBID= node130-1305437407-30883
PBS_JOBID= 13324772
Free space on /tmp= 73460 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/openstacks-sequencedstrips-nonadl-nonnegated-os-sequencedstrips-p30_3.025-NOTKNOWN.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372440-1305437528/watcher-3372440-1305437528 -o /tmp/evaluation-result-3372440-1305437528/solver-3372440-1305437528 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372440-1305437528.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3c5c73741ba21d5f39c7f436ba834743
RANDOM SEED=1540506464

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27547300 kB
Buffers:        275004 kB
Cached:        3800664 kB
SwapCached:      10248 kB
Active:        1317992 kB
Inactive:      3813692 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27547300 kB
SwapTotal:    67111528 kB
SwapFree:     63649232 kB
Dirty:            8680 kB
Writeback:           0 kB
AnonPages:     1053704 kB
Mapped:          12968 kB
Slab:           183672 kB
PageTables:      24420 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 51458492 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73460 MiB
End job on node130 at 2011-05-15 08:55:37