Trace number 3395040

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.7 5011.72

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q918.75-n50-p15-s98191766.cnf
MD5SUM71638ad618721ad48c6bd2d48b828020
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.32195
Satisfiable
(Un)Satisfiability was proved
Number of variables2377
Number of clauses566749
Sum of the clauses size1135775
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2566699
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3395040-1306111824/watcher-3395040-1306111824 -o /tmp/evaluation-result-3395040-1306111824/solver-3395040-1306111824 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3395040-1306111824.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.51 7.94 7.97 2/491 11517
/proc/meminfo: memFree=14096496/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=11376 CPUtime=0 cores=0-7
/proc/11517/stat : 11517 (plingeling) R 11515 11517 11491 0 -1 4202496 925 1251 0 0 0 0 0 0 23 0 1 0 801413098 11649024 855 33554432000 4194304 4417452 140734722068496 18446744073709551615 251367207621 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/11517/statm: 2880 867 147 55 0 766 0

[startup+0.110795 s]
/proc/loadavg: 7.51 7.94 7.97 2/491 11517
/proc/meminfo: memFree=14096496/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=27840 CPUtime=0.09 cores=0-7
/proc/11517/stat : 11517 (plingeling) R 11515 11517 11491 0 -1 4202496 5201 1251 0 0 9 0 0 0 23 0 1 0 801413098 28508160 4983 33554432000 4194304 4417452 140734722068496 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/11517/statm: 6960 4983 148 55 0 4846 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27840

[startup+0.200777 s]
/proc/loadavg: 7.51 7.94 7.97 2/491 11517
/proc/meminfo: memFree=14096496/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=44224 CPUtime=0.19 cores=0-7
/proc/11517/stat : 11517 (plingeling) R 11515 11517 11491 0 -1 4202496 9297 1251 0 0 18 1 0 0 23 0 1 0 801413098 45285376 9079 33554432000 4194304 4417452 140734722068496 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/11517/statm: 11056 9079 148 55 0 8942 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 44224

[startup+0.300757 s]
/proc/loadavg: 7.51 7.94 7.97 2/491 11517
/proc/meminfo: memFree=14096496/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=72896 CPUtime=0.29 cores=0-7
/proc/11517/stat : 11517 (plingeling) R 11515 11517 11491 0 -1 4202496 15450 1251 0 0 27 2 0 0 24 0 1 0 801413098 74645504 15233 33554432000 4194304 4417452 140734722068496 18446744073709551615 251367240797 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/11517/statm: 18224 15264 148 55 0 16110 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 72896

[startup+0.700683 s]
/proc/loadavg: 7.51 7.94 7.97 2/491 11517
/proc/meminfo: memFree=14096496/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=142528 CPUtime=0.69 cores=0-7
/proc/11517/stat : 11517 (plingeling) R 11515 11517 11491 0 -1 4202496 33873 1251 0 0 65 4 0 0 25 0 1 0 801413098 145948672 33655 33554432000 4194304 4417452 140734722068496 18446744073709551615 4227801 0 0 4096 17442 0 0 0 17 0 0 0 0
/proc/11517/statm: 35632 33655 148 55 0 33518 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 142528

[startup+1.50046 s]
/proc/loadavg: 7.51 7.94 7.97 9/500 11531
/proc/meminfo: memFree=13954816/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=323636 CPUtime=6.4 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 67064 1251 0 0 629 11 0 0 18 0 9 0 801413098 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 80909 42139 178 55 0 78795 0
[pid=11517/tid=11524] ppid=11515 vsize=323636 CPUtime=0.69 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 2073 1251 0 0 69 0 0 0 18 0 9 0 801413176 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4346458 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=323636 CPUtime=0.7 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 6090 1251 0 0 69 1 0 0 18 0 9 0 801413176 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4302020 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=323636 CPUtime=0.7 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 4115 1251 0 0 70 0 0 0 25 0 9 0 801413176 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4339868 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=323636 CPUtime=0.7 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 4932 1251 0 0 69 1 0 0 25 0 9 0 801413176 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4259140 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=323636 CPUtime=0.7 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 5538 1251 0 0 69 1 0 0 18 0 9 0 801413176 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4302036 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=323636 CPUtime=0.69 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 4149 1251 0 0 69 0 0 0 25 0 9 0 801413177 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4346677 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=323636 CPUtime=0.68 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 4163 1251 0 0 68 0 0 0 18 0 9 0 801413177 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4346677 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=323636 CPUtime=0.68 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2115 1251 0 0 68 0 0 0 21 0 9 0 801413177 331403264 42139 33554432000 4194304 4417452 140734722068496 18446744073709551615 4346501 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.4
Current children cumulated vsize (KiB) 323636

[startup+3.10116 s]
/proc/loadavg: 7.51 7.94 7.97 9/500 11531
/proc/meminfo: memFree=13895792/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=357860 CPUtime=19.18 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 84507 1251 0 0 1905 13 0 0 18 0 9 0 801413098 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 89465 50665 180 55 0 87351 0
[pid=11517/tid=11524] ppid=11515 vsize=357860 CPUtime=2.29 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 3801 1251 0 0 229 0 0 0 21 0 9 0 801413176 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4236239 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=357860 CPUtime=2.3 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 7304 1251 0 0 229 1 0 0 20 0 9 0 801413176 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=357860 CPUtime=2.3 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 5330 1251 0 0 230 0 0 0 25 0 9 0 801413176 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=357860 CPUtime=2.3 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 13970 1251 0 0 228 2 0 0 25 0 9 0 801413176 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4354524 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=357860 CPUtime=2.3 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 6752 1251 0 0 229 1 0 0 20 0 9 0 801413176 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4258632 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=357860 CPUtime=2.3 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 5970 1251 0 0 229 1 0 0 25 0 9 0 801413177 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=357860 CPUtime=2.28 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 5376 1251 0 0 228 0 0 0 20 0 9 0 801413177 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4250003 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=357860 CPUtime=2.26 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2115 1251 0 0 226 0 0 0 25 0 9 0 801413177 366448640 50665 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 19.18
Current children cumulated vsize (KiB) 357860

[startup+6.30057 s]
/proc/loadavg: 7.55 7.94 7.97 9/500 11532
/proc/meminfo: memFree=13865916/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=364312 CPUtime=44.73 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 100525 1251 0 0 4457 16 0 0 18 0 9 0 801413098 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 91078 52278 180 55 0 88964 0
[pid=11517/tid=11524] ppid=11515 vsize=364312 CPUtime=5.49 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 3801 1251 0 0 549 0 0 0 25 0 9 0 801413176 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4239312 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=364312 CPUtime=5.5 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 7304 1251 0 0 549 1 0 0 25 0 9 0 801413176 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4239605 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=364312 CPUtime=5.5 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 5330 1251 0 0 550 0 0 0 25 0 9 0 801413176 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4376607 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=364312 CPUtime=5.5 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 29988 1251 0 0 545 5 0 0 25 0 9 0 801413176 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=364312 CPUtime=5.5 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 6752 1251 0 0 549 1 0 0 25 0 9 0 801413176 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=364312 CPUtime=5.5 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 5970 1251 0 0 549 1 0 0 25 0 9 0 801413177 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4250003 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=364312 CPUtime=5.48 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 5376 1251 0 0 548 0 0 0 25 0 9 0 801413177 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=364312 CPUtime=5.41 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2115 1251 0 0 541 0 0 0 25 0 9 0 801413177 373055488 52278 33554432000 4194304 4417452 140734722068496 18446744073709551615 4235999 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 44.73
Current children cumulated vsize (KiB) 364312

[startup+12.7004 s]
/proc/loadavg: 7.59 7.94 7.97 9/500 11532
/proc/meminfo: memFree=13841672/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=421756 CPUtime=95.82 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 126385 1251 0 0 9563 19 0 0 18 0 9 0 801413098 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 105439 66639 180 55 0 103325 0
[pid=11517/tid=11524] ppid=11515 vsize=421756 CPUtime=11.89 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 3801 1251 0 0 1189 0 0 0 25 0 9 0 801413176 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=421756 CPUtime=11.9 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 7304 1251 0 0 1189 1 0 0 25 0 9 0 801413176 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4235968 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=421756 CPUtime=11.9 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 7520 1251 0 0 1190 0 0 0 25 0 9 0 801413176 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4356934 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=421756 CPUtime=11.9 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 51610 1251 0 0 1183 7 0 0 25 0 9 0 801413176 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4355827 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=421756 CPUtime=11.9 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 6752 1251 0 0 1189 1 0 0 25 0 9 0 801413176 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4248897 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=421756 CPUtime=11.9 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 5970 1251 0 0 1189 1 0 0 25 0 9 0 801413177 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=421756 CPUtime=11.88 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 7424 1251 0 0 1188 0 0 0 25 0 9 0 801413177 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=421756 CPUtime=11.7 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2115 1251 0 0 1170 0 0 0 25 0 9 0 801413177 431878144 66639 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 95.82
Current children cumulated vsize (KiB) 421756

[startup+25.501 s]
/proc/loadavg: 7.68 7.94 7.97 9/500 11532
/proc/meminfo: memFree=13735536/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=517720 CPUtime=197.99 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 166706 1251 0 0 19776 23 0 0 18 0 9 0 801413098 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 129430 90664 180 55 0 127316 0
[pid=11517/tid=11524] ppid=11515 vsize=517720 CPUtime=24.7 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 4147 1251 0 0 2469 1 0 0 25 0 9 0 801413176 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4236008 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=517720 CPUtime=24.69 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 9870 1251 0 0 2468 1 0 0 25 0 9 0 801413176 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=517720 CPUtime=24.69 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 7522 1251 0 0 2469 0 0 0 25 0 9 0 801413176 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=517720 CPUtime=24.7 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 86548 1251 0 0 2459 11 0 0 25 0 9 0 801413176 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=517720 CPUtime=24.7 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 8800 1251 0 0 2469 1 0 0 25 0 9 0 801413176 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4248897 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=517720 CPUtime=24.69 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 6391 1251 0 0 2468 1 0 0 25 0 9 0 801413177 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=517720 CPUtime=24.68 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 7424 1251 0 0 2468 0 0 0 25 0 9 0 801413177 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=517720 CPUtime=24.28 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2115 1251 0 0 2428 0 0 0 25 0 9 0 801413177 530145280 90664 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 197.99
Current children cumulated vsize (KiB) 517720

[startup+51.1013 s]
/proc/loadavg: 7.95 7.98 7.98 9/500 11533
/proc/meminfo: memFree=13598888/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=654056 CPUtime=402.36 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 231707 1251 0 0 40202 34 0 0 18 0 9 0 801413098 669753344 124811 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 163514 124811 180 55 0 161400 0
[pid=11517/tid=11524] ppid=11515 vsize=654056 CPUtime=50.3 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 7480 1251 0 0 5029 1 0 0 25 0 9 0 801413176 669753344 124811 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249181 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=654056 CPUtime=50.29 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 9870 1251 0 0 5028 1 0 0 25 0 9 0 801413176 669753344 124811 33554432000 4194304 4417452 140734722068496 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=654056 CPUtime=50.3 cores=0-7

################
# More data... #
################

[pid=11517/tid=11526] ppid=11515 vsize=8210288 CPUtime=4721.13 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 2537602 1251 0 0 471732 381 0 0 25 0 9 0 801413176 8407334912 2013731 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=8210288 CPUtime=4720.22 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 1402751 1251 0 0 471693 329 0 0 25 0 9 0 801413176 8407334912 2013731 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=8210288 CPUtime=4721.63 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 1225169 1251 0 0 471913 250 0 0 25 0 9 0 801413176 8407334912 2013731 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=8210288 CPUtime=4657.51 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 1486374 1251 0 0 465522 229 0 0 25 0 9 0 801413177 8407334912 2013731 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=8210288 CPUtime=4709.43 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 1457670 1251 0 0 470696 247 0 0 25 0 9 0 801413177 8407334912 2013731 33554432000 4194304 4417452 140734722068496 18446744073709551615 4263118 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=8210288 CPUtime=4720.26 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2194439 1251 0 0 471726 300 0 0 25 0 9 0 801413177 8407334912 2013731 33554432000 4194304 4417452 140734722068496 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37692.2
Current children cumulated vsize (KiB) 8210288

[startup+4782.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/500 12152
/proc/meminfo: memFree=6049732/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=8348684 CPUtime=38171.2 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 13940815 1251 0 0 3814780 2340 0 0 18 0 9 0 801413098 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 2087171 2051770 180 55 0 2085057 0
[pid=11517/tid=11524] ppid=11515 vsize=8348684 CPUtime=4780.5 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 2034462 1251 0 0 477726 324 0 0 25 0 9 0 801413176 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=8348684 CPUtime=4780.73 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 1497651 1251 0 0 477809 264 0 0 25 0 9 0 801413176 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=8348684 CPUtime=4781.11 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 2586754 1251 0 0 477725 386 0 0 25 0 9 0 801413176 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=8348684 CPUtime=4780.21 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 1402751 1251 0 0 477692 329 0 0 25 0 9 0 801413176 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=8348684 CPUtime=4781.61 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 1225169 1251 0 0 477911 250 0 0 25 0 9 0 801413176 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4365088 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=8348684 CPUtime=4716.52 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 1502758 1251 0 0 471421 231 0 0 25 0 9 0 801413177 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=8348684 CPUtime=4769.41 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 1457670 1251 0 0 476694 247 0 0 25 0 9 0 801413177 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=8348684 CPUtime=4780.25 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2199711 1251 0 0 477724 301 0 0 25 0 9 0 801413177 8549052416 2051770 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38171.2
Current children cumulated vsize (KiB) 8348684

[startup+4842.32 s]
/proc/loadavg: 8.00 8.02 8.00 9/500 12154
/proc/meminfo: memFree=5734392/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=8571784 CPUtime=38650.3 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 14189563 1251 0 0 3862641 2392 0 0 18 0 9 0 801413098 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 2142946 2114219 180 55 0 2140832 0
[pid=11517/tid=11524] ppid=11515 vsize=8571784 CPUtime=4840.51 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 2054367 1251 0 0 483724 327 0 0 25 0 9 0 801413176 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=8571784 CPUtime=4840.75 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 1546803 1251 0 0 483805 270 0 0 25 0 9 0 801413176 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=8571784 CPUtime=4841.13 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 2619522 1251 0 0 483720 393 0 0 25 0 9 0 801413176 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=8571784 CPUtime=4840.23 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 1402751 1251 0 0 483694 329 0 0 25 0 9 0 801413176 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=8571784 CPUtime=4841.63 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 1347164 1251 0 0 483880 283 0 0 25 0 9 0 801413176 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=8571784 CPUtime=4775.55 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 1502758 1251 0 0 477324 231 0 0 25 0 9 0 801413177 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=8571784 CPUtime=4829.43 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 1457670 1251 0 0 482696 247 0 0 25 0 9 0 801413177 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=8571784 CPUtime=4840.23 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2224639 1251 0 0 483719 304 0 0 25 0 9 0 801413177 8777506816 2114219 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38650.3
Current children cumulated vsize (KiB) 8571784

[startup+4902.3 s]
/proc/loadavg: 8.05 8.03 8.00 9/500 12155
/proc/meminfo: memFree=5778504/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=8575528 CPUtime=39129.2 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 14309283 1251 0 0 3910514 2406 0 0 18 0 9 0 801413098 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 2143882 2119251 180 55 0 2141768 0
[pid=11517/tid=11524] ppid=11515 vsize=8575528 CPUtime=4900.47 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 2070751 1251 0 0 489718 329 0 0 25 0 9 0 801413176 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=8575528 CPUtime=4900.74 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 1595955 1251 0 0 489799 275 0 0 25 0 9 0 801413176 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=8575528 CPUtime=4901.11 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 2652290 1251 0 0 489714 397 0 0 25 0 9 0 801413176 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=8575528 CPUtime=4900.21 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 1402751 1251 0 0 489692 329 0 0 25 0 9 0 801413176 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=8575528 CPUtime=4901.62 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 1347164 1251 0 0 489879 283 0 0 25 0 9 0 801413176 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=8575528 CPUtime=4834.56 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 1502758 1251 0 0 483225 231 0 0 25 0 9 0 801413177 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=8575528 CPUtime=4889.42 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 1457670 1251 0 0 488695 247 0 0 25 0 9 0 801413177 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=8575528 CPUtime=4900.22 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2246055 1251 0 0 489715 307 0 0 25 0 9 0 801413177 8781340672 2119251 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39129.2
Current children cumulated vsize (KiB) 8575528

[startup+4962.3 s]
/proc/loadavg: 8.10 8.05 8.01 9/500 12157
/proc/meminfo: memFree=5750576/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=8586888 CPUtime=39608.2 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 14398140 1251 0 0 3958401 2419 0 0 18 0 9 0 801413098 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 2146722 2109804 180 55 0 2144608 0
[pid=11517/tid=11524] ppid=11515 vsize=8586888 CPUtime=4960.47 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 2103519 1251 0 0 495714 333 0 0 25 0 9 0 801413176 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=8586888 CPUtime=4960.74 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 1595955 1251 0 0 495799 275 0 0 25 0 9 0 801413176 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=8586888 CPUtime=4961.12 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 2668674 1251 0 0 495709 403 0 0 25 0 9 0 801413176 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 251367604650 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=8586888 CPUtime=4960.22 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 1402751 1251 0 0 495693 329 0 0 25 0 9 0 801413176 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=8586888 CPUtime=4961.62 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 1347164 1251 0 0 495879 283 0 0 25 0 9 0 801413176 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=8586888 CPUtime=4893.57 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 1502758 1251 0 0 489126 231 0 0 25 0 9 0 801413177 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=8586888 CPUtime=4949.42 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 1457670 1251 0 0 494695 247 0 0 25 0 9 0 801413177 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4307907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=8586888 CPUtime=4960.21 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2285760 1251 0 0 495710 311 0 0 25 0 9 0 801413177 8792973312 2109804 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39608.2
Current children cumulated vsize (KiB) 8586888



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.4 s]
/proc/loadavg: 8.09 8.06 8.01 9/500 12158
/proc/meminfo: memFree=5416124/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=8922540 CPUtime=40000.3 cores=0-7
/proc/11517/stat : 11517 (plingeling) S 11515 11517 11491 0 -1 4202496 14593570 1251 0 0 3997573 2454 0 0 18 0 9 0 801413098 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 251379349477 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 2230635 2193718 180 55 0 2228521 0
[pid=11517/tid=11524] ppid=11515 vsize=8922540 CPUtime=5009.58 cores=0-7
/proc/11517/task/11524/stat : 11524 (plingeling) R 11515 11517 11491 0 -1 4202560 2119903 1251 0 0 500623 335 0 0 25 0 9 0 801413176 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11517/tid=11525] ppid=11515 vsize=8922540 CPUtime=5009.86 cores=0-7
/proc/11517/task/11525/stat : 11525 (plingeling) R 11515 11517 11491 0 -1 4202560 1612339 1251 0 0 500709 277 0 0 25 0 9 0 801413176 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11517/tid=11526] ppid=11515 vsize=8922540 CPUtime=5010.24 cores=0-7
/proc/11517/task/11526/stat : 11526 (plingeling) R 11515 11517 11491 0 -1 4202560 2685058 1251 0 0 500619 405 0 0 25 0 9 0 801413176 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11517/tid=11527] ppid=11515 vsize=8922540 CPUtime=5009.33 cores=0-7
/proc/11517/task/11527/stat : 11527 (plingeling) R 11515 11517 11491 0 -1 4202560 1408303 1251 0 0 500603 330 0 0 25 0 9 0 801413176 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11517/tid=11528] ppid=11515 vsize=8922540 CPUtime=5010.74 cores=0-7
/proc/11517/task/11528/stat : 11528 (plingeling) R 11515 11517 11491 0 -1 4202560 1347164 1251 0 0 500791 283 0 0 25 0 9 0 801413176 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=8922540 CPUtime=4941.88 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4202560 1502758 1251 0 0 493957 231 0 0 25 0 9 0 801413177 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11517/tid=11530] ppid=11515 vsize=8922540 CPUtime=4998.54 cores=0-7
/proc/11517/task/11530/stat : 11530 (plingeling) R 11515 11517 11491 0 -1 4202560 1563756 1251 0 0 499584 270 0 0 25 0 9 0 801413177 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11517/tid=11531] ppid=11515 vsize=8922540 CPUtime=5009.33 cores=0-7
/proc/11517/task/11531/stat : 11531 (plingeling) R 11515 11517 11491 0 -1 4202560 2320400 1251 0 0 500618 315 0 0 25 0 9 0 801413177 9136680960 2193718 33554432000 4194304 4417452 140734722068496 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 8922540

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011.52 s]
/proc/loadavg: 8.09 8.06 8.01 9/500 12158
/proc/meminfo: memFree=5416124/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=0 CPUtime=40000.5 cores=0-7
/proc/11517/stat : 11517 (plingeling) Z 11515 11517 11491 0 -1 4203548 14593570 1251 0 0 3997585 2464 0 0 18 0 2 0 801413098 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 0 0 0 0 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=0 CPUtime=4941.98 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4203588 1502758 1251 0 0 493957 241 0 0 25 0 2 0 801413177 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 0

[startup+5011.72 s]
/proc/loadavg: 8.09 8.06 8.01 2/494 12159
/proc/meminfo: memFree=10620328/32950928 swapFree=38710952/67111528
[pid=11517] ppid=11515 vsize=0 CPUtime=40000.7 cores=0-7
/proc/11517/stat : 11517 (plingeling) Z 11515 11517 11491 0 -1 4203548 14593570 1251 0 0 3997585 2483 0 0 18 0 2 0 801413098 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/11517/statm: 0 0 0 0 0 0 0
[pid=11517/tid=11529] ppid=11515 vsize=0 CPUtime=4942.17 cores=0-7
/proc/11517/task/11529/stat : 11529 (plingeling) R 11515 11517 11491 0 -1 4203588 1502758 1251 0 0 493957 260 0 0 25 0 2 0 801413177 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5011.72
CPU time (s): 40000.7
CPU user time (s): 39975.9
CPU system time (s): 24.8402
CPU usage (%): 798.143
Max. virtual memory (cumulated for all children) (KiB): 8922540

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39975.9
system time used= 24.8402
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 14594821
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 338
involuntary context switches= 198961

runsolver used 22.6746 second user time and 59.437 second system time

The end

Launcher Data 

Begin job on node113 at 2011-05-23 02:50:24
IDJOB=3395040
IDBENCH=88228
IDSOLVER=1851
FILE ID=node113/3395040-1306111824
RUNJOBID= node113-1306111824-11505
PBS_JOBID= 13388053
Free space on /tmp= 73308 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q918.75-n50-p15-s98191766.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3395040-1306111824/watcher-3395040-1306111824 -o /tmp/evaluation-result-3395040-1306111824/solver-3395040-1306111824 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3395040-1306111824.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 71638ad618721ad48c6bd2d48b828020
RANDOM SEED=1577465065

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      14096900 kB
Buffers:        335232 kB
Cached:        8782460 kB
SwapCached:      93732 kB
Active:        8077724 kB
Inactive:     10436108 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      14096900 kB
SwapTotal:    67111528 kB
SwapFree:     38710952 kB
Dirty:            7392 kB
Writeback:           0 kB
AnonPages:     9314644 kB
Mapped:          24156 kB
Slab:           152160 kB
PageTables:     116988 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 154760684 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73416 MiB
End job on node113 at 2011-05-23 04:13:58