Trace number 3394124

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.6 5008.28

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v1560c133795gyes1.cnf
MD5SUM118632220c6f3388c45a2582e9a2ac0e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark3144.24
Satisfiable
(Un)Satisfiability was proved
Number of variables1560
Number of clauses133795
Sum of the clauses size269030
Maximum clause length26
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2133735
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394124-1306091016/watcher-3394124-1306091016 -o /tmp/evaluation-result-3394124-1306091016/solver-3394124-1306091016 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394124-1306091016.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.42 7.90 7.96 2/380 29680
/proc/meminfo: memFree=10619884/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=10500 CPUtime=0.01 cores=0-7
/proc/29680/stat : 29680 (plingeling) R 29678 29680 29455 0 -1 4202496 693 1251 0 0 1 0 0 0 25 0 1 0 720704347 10752000 623 33554432000 4194304 4417452 140736756900768 18446744073709551615 218370957056 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/29680/statm: 2625 623 147 55 0 511 0

[startup+0.078304 s]
/proc/loadavg: 7.42 7.90 7.96 2/380 29680
/proc/meminfo: memFree=10619884/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=26724 CPUtime=0.06 cores=0-7
/proc/29680/stat : 29680 (plingeling) R 29678 29680 29455 0 -1 4202496 4758 1251 0 0 6 0 0 0 25 0 1 0 720704347 27365376 4688 33554432000 4194304 4417452 140736756900768 18446744073709551615 4209661 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/29680/statm: 6681 4688 148 55 0 4567 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 26724

[startup+0.100311 s]
/proc/loadavg: 7.42 7.90 7.96 2/380 29680
/proc/meminfo: memFree=10619884/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=26724 CPUtime=0.09 cores=0-7
/proc/29680/stat : 29680 (plingeling) R 29678 29680 29455 0 -1 4202496 4758 1251 0 0 9 0 0 0 25 0 1 0 720704347 27365376 4688 33554432000 4194304 4417452 140736756900768 18446744073709551615 218370957056 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/29680/statm: 6681 4688 148 55 0 4567 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26724

[startup+0.307129 s]
/proc/loadavg: 7.42 7.90 7.96 2/380 29680
/proc/meminfo: memFree=10619884/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=203560 CPUtime=1.08 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 12941 1251 0 0 107 1 0 0 21 0 9 0 720704347 208445440 11972 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 50890 11972 177 55 0 48776 0
Current children cumulated CPU time (s) 1.08
Current children cumulated vsize (KiB) 203560

[startup+1.17707 s]
/proc/loadavg: 7.42 7.90 7.96 2/380 29680
/proc/meminfo: memFree=10619884/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=277860 CPUtime=7.99 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 26012 1251 0 0 796 3 0 0 21 0 9 0 720704347 284528640 14183 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 69465 14183 180 55 0 67351 0
Current children cumulated CPU time (s) 7.99
Current children cumulated vsize (KiB) 277860

[startup+1.50102 s]
/proc/loadavg: 7.42 7.90 7.96 2/380 29680
/proc/meminfo: memFree=10619884/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=282996 CPUtime=10.58 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 27296 1251 0 0 1054 4 0 0 21 0 9 0 720704347 289787904 15467 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 70749 15467 180 55 0 68635 0
Current children cumulated CPU time (s) 10.58
Current children cumulated vsize (KiB) 282996

[startup+3.10073 s]
/proc/loadavg: 7.42 7.90 7.96 10/389 29694
/proc/meminfo: memFree=10522976/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=304228 CPUtime=23.27 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 39644 1251 0 0 2321 6 0 0 21 0 9 0 720704347 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 76057 20934 180 55 0 73943 0
[pid=29680/tid=29687] ppid=29678 vsize=304228 CPUtime=2.8 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 1758 1251 0 0 280 0 0 0 25 0 9 0 720704364 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=304228 CPUtime=2.91 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 2739 1251 0 0 291 0 0 0 23 0 9 0 720704364 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=304228 CPUtime=2.92 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 1515 1251 0 0 292 0 0 0 24 0 9 0 720704364 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4259618 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=304228 CPUtime=2.92 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 16927 1251 0 0 289 3 0 0 25 0 9 0 720704364 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4250651 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=304228 CPUtime=2.89 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 2029 1251 0 0 289 0 0 0 23 0 9 0 720704364 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4236017 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=304228 CPUtime=2.92 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 2698 1251 0 0 292 0 0 0 24 0 9 0 720704364 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4258641 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=304228 CPUtime=2.92 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 1217 1251 0 0 292 0 0 0 25 0 9 0 720704365 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=304228 CPUtime=2.78 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 1892 1251 0 0 278 0 0 0 25 0 9 0 720704365 311529472 20934 33554432000 4194304 4417452 140736756900768 18446744073709551615 4249280 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.27
Current children cumulated vsize (KiB) 304228

[startup+6.30116 s]
/proc/loadavg: 7.55 7.92 7.97 10/389 29694
/proc/meminfo: memFree=10508220/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=312420 CPUtime=48.68 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 51892 1251 0 0 4860 8 0 0 21 0 9 0 720704347 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 78105 23028 180 55 0 75991 0
[pid=29680/tid=29687] ppid=29678 vsize=312420 CPUtime=6 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 1758 1251 0 0 600 0 0 0 25 0 9 0 720704364 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=312420 CPUtime=6.12 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 2739 1251 0 0 612 0 0 0 25 0 9 0 720704364 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=312420 CPUtime=6.12 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 1515 1251 0 0 612 0 0 0 25 0 9 0 720704364 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=312420 CPUtime=6.12 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 29129 1251 0 0 607 5 0 0 25 0 9 0 720704364 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4232569 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=312420 CPUtime=6.05 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 2045 1251 0 0 605 0 0 0 25 0 9 0 720704364 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4258604 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=312420 CPUtime=6.12 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 2698 1251 0 0 612 0 0 0 25 0 9 0 720704364 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4336723 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=312420 CPUtime=6.12 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 1247 1251 0 0 612 0 0 0 25 0 9 0 720704365 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=312420 CPUtime=5.82 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 1892 1251 0 0 582 0 0 0 25 0 9 0 720704365 319918080 23028 33554432000 4194304 4417452 140736756900768 18446744073709551615 4233229 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 48.68
Current children cumulated vsize (KiB) 312420

[startup+12.714 s]
/proc/loadavg: 7.58 7.92 7.97 9/388 29694
/proc/meminfo: memFree=10505300/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=351048 CPUtime=99.9 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 84785 1251 0 0 9977 13 0 0 21 0 9 0 720704347 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 87762 32655 180 55 0 85648 0
[pid=29680/tid=29687] ppid=29678 vsize=351048 CPUtime=12.41 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 2568 1251 0 0 1241 0 0 0 25 0 9 0 720704364 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4250003 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=351048 CPUtime=12.53 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 3549 1251 0 0 1253 0 0 0 25 0 9 0 720704364 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4336683 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=351048 CPUtime=12.53 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 2126 1251 0 0 1253 0 0 0 25 0 9 0 720704364 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=351048 CPUtime=12.52 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 55489 1251 0 0 1243 9 0 0 25 0 9 0 720704364 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=351048 CPUtime=12.39 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 3666 1251 0 0 1239 0 0 0 25 0 9 0 720704364 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4248926 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=351048 CPUtime=12.53 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 3508 1251 0 0 1253 0 0 0 25 0 9 0 720704364 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4258267 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=351048 CPUtime=12.53 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 2308 1251 0 0 1253 0 0 0 25 0 9 0 720704365 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4258389 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=351048 CPUtime=12.23 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 2702 1251 0 0 1223 0 0 0 25 0 9 0 720704365 359473152 32655 33554432000 4194304 4417452 140736756900768 18446744073709551615 4320498 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 99.9
Current children cumulated vsize (KiB) 351048

[startup+25.5007 s]
/proc/loadavg: 7.75 7.94 7.97 9/388 29695
/proc/meminfo: memFree=10429608/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=430480 CPUtime=202.05 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 134199 1251 0 0 20182 23 0 0 21 0 9 0 720704347 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 107620 52491 180 55 0 105506 0
[pid=29680/tid=29687] ppid=29678 vsize=430480 CPUtime=25.19 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 3508 1251 0 0 2519 0 0 0 25 0 9 0 720704364 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=430480 CPUtime=25.32 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 5180 1251 0 0 2531 1 0 0 25 0 9 0 720704364 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=430480 CPUtime=25.32 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 7792 1251 0 0 2531 1 0 0 25 0 9 0 720704364 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=430480 CPUtime=25.32 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 89716 1251 0 0 2515 17 0 0 25 0 9 0 720704364 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=430480 CPUtime=25.03 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 3804 1251 0 0 2503 0 0 0 25 0 9 0 720704364 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=430480 CPUtime=25.31 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 5198 1251 0 0 2531 0 0 0 25 0 9 0 720704364 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=430480 CPUtime=25.31 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 6264 1251 0 0 2531 0 0 0 25 0 9 0 720704365 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=430480 CPUtime=25.02 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 3868 1251 0 0 2502 0 0 0 25 0 9 0 720704365 440811520 52491 33554432000 4194304 4417452 140736756900768 18446744073709551615 4250961 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 202.05
Current children cumulated vsize (KiB) 430480

[startup+51.1012 s]
/proc/loadavg: 7.83 7.94 7.97 9/388 29695
/proc/meminfo: memFree=10386436/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=467824 CPUtime=406.64 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 219418 1251 0 0 40626 38 0 0 21 0 9 0 720704347 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 116956 62597 180 55 0 114842 0
[pid=29680/tid=29687] ppid=29678 vsize=467824 CPUtime=50.81 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 9013 1251 0 0 5080 1 0 0 25 0 9 0 720704364 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=467824 CPUtime=50.93 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 5838 1251 0 0 5092 1 0 0 25 0 9 0 720704364 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4250876 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=467824 CPUtime=50.93 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 15623 1251 0 0 5091 2 0 0 25 0 9 0 720704364 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=467824 CPUtime=50.93 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 144775 1251 0 0 5066 27 0 0 25 0 9 0 720704364 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364488 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=467824 CPUtime=50.33 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 4824 1251 0 0 5033 0 0 0 25 0 9 0 720704364 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=467824 CPUtime=50.93 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 12752 1251 0 0 5091 2 0 0 25 0 9 0 720704364 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=467824 CPUtime=50.93 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 13527 1251 0 0 5092 1 0 0 25 0 9 0 720704365 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366079 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=467824 CPUtime=50.62 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 4197 1251 0 0 5062 0 0 0 25 0 9 0 720704365 479051776 62597 33554432000 4194304 4417452 140736756900768 18446744073709551615 4250936 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.64
Current children cumulated vsize (KiB) 467824

[startup+102.301 s]
/proc/loadavg: 8.03 7.98 7.98 9/388 29697

################
# More data... #
################


[startup+4722.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/388 29832
/proc/meminfo: memFree=6314364/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=4534164 CPUtime=37717.5 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 6198636 1251 0 0 3770631 1115 0 0 21 0 9 0 720704347 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 1133541 1078412 180 55 0 1131427 0
[pid=29680/tid=29687] ppid=29678 vsize=4534164 CPUtime=4721.98 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 633894 1251 0 0 472064 134 0 0 25 0 9 0 720704364 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4262948 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=4534164 CPUtime=4694.66 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 561599 1251 0 0 469360 106 0 0 25 0 9 0 720704364 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=4534164 CPUtime=4722.2 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 843582 1251 0 0 472083 137 0 0 25 0 9 0 720704364 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=4534164 CPUtime=4722.24 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 1591576 1251 0 0 471905 319 0 0 25 0 9 0 720704364 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=4534164 CPUtime=4719.24 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 611580 1251 0 0 471825 99 0 0 25 0 9 0 720704364 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=4534164 CPUtime=4722.19 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 674541 1251 0 0 472111 108 0 0 25 0 9 0 720704364 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=4534164 CPUtime=4722.06 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 663897 1251 0 0 472085 121 0 0 25 0 9 0 720704365 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=4534164 CPUtime=4692.66 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 609098 1251 0 0 469179 87 0 0 25 0 9 0 720704365 4642983936 1078412 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37717.5
Current children cumulated vsize (KiB) 4534164

[startup+4782.31 s]
/proc/loadavg: 8.00 8.03 8.00 9/388 29834
/proc/meminfo: memFree=6149264/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=4698792 CPUtime=38196.8 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 6271171 1251 0 0 3818551 1129 0 0 21 0 9 0 720704347 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 1174698 1119569 180 55 0 1172584 0
[pid=29680/tid=29687] ppid=29678 vsize=4698792 CPUtime=4781.99 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 633894 1251 0 0 478065 134 0 0 25 0 9 0 720704364 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=4698792 CPUtime=4754.65 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 561599 1251 0 0 475359 106 0 0 25 0 9 0 720704364 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=4698792 CPUtime=4782.21 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 916117 1251 0 0 478070 151 0 0 25 0 9 0 720704364 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4365058 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=4698792 CPUtime=4782.26 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 1591576 1251 0 0 477907 319 0 0 25 0 9 0 720704364 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=4698792 CPUtime=4779.26 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 611580 1251 0 0 477827 99 0 0 25 0 9 0 720704364 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=4698792 CPUtime=4782.2 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 674541 1251 0 0 478112 108 0 0 25 0 9 0 720704364 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=4698792 CPUtime=4782.08 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 663897 1251 0 0 478087 121 0 0 25 0 9 0 720704365 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=4698792 CPUtime=4751.92 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 609098 1251 0 0 475105 87 0 0 25 0 9 0 720704365 4811563008 1119569 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38196.8
Current children cumulated vsize (KiB) 4698792

[startup+4842.3 s]
/proc/loadavg: 8.11 8.05 8.01 9/388 29835
/proc/meminfo: memFree=5596160/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=5057976 CPUtime=38675.9 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 6471916 1251 0 0 3866409 1183 0 0 21 0 9 0 720704347 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 1264494 1209397 180 55 0 1262380 0
[pid=29680/tid=29687] ppid=29678 vsize=5057976 CPUtime=4841.98 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 723988 1251 0 0 484033 165 0 0 25 0 9 0 720704364 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=5057976 CPUtime=4814.63 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 561599 1251 0 0 481357 106 0 0 25 0 9 0 720704364 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=5057976 CPUtime=4842.2 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 916117 1251 0 0 484069 151 0 0 25 0 9 0 720704364 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=5057976 CPUtime=4842.24 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 1591576 1251 0 0 483905 319 0 0 25 0 9 0 720704364 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=5057976 CPUtime=4839.25 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 635825 1251 0 0 483822 103 0 0 25 0 9 0 720704364 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4250721 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=5057976 CPUtime=4842.19 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 680493 1251 0 0 484109 110 0 0 25 0 9 0 720704364 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=5057976 CPUtime=4842.06 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 663897 1251 0 0 484085 121 0 0 25 0 9 0 720704365 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=5057976 CPUtime=4811.14 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 689552 1251 0 0 481010 104 0 0 25 0 9 0 720704365 5179367424 1209397 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38675.9
Current children cumulated vsize (KiB) 5057976

[startup+4902.3 s]
/proc/loadavg: 8.11 8.06 8.00 9/388 29837
/proc/meminfo: memFree=5661864/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=5185264 CPUtime=39155.2 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 6559523 1251 0 0 3914316 1202 0 0 21 0 9 0 720704347 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 1296316 1241219 180 55 0 1294202 0
[pid=29680/tid=29687] ppid=29678 vsize=5185264 CPUtime=4901.98 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 723988 1251 0 0 490032 166 0 0 25 0 9 0 720704364 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=5185264 CPUtime=4874.62 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 569193 1251 0 0 487355 107 0 0 25 0 9 0 720704364 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=5185264 CPUtime=4902.2 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 916117 1251 0 0 490069 151 0 0 25 0 9 0 720704364 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=5185264 CPUtime=4902.25 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 1591576 1251 0 0 489906 319 0 0 25 0 9 0 720704364 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=5185264 CPUtime=4899.25 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 638977 1251 0 0 489822 103 0 0 25 0 9 0 720704364 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=5185264 CPUtime=4902.19 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 757354 1251 0 0 490093 126 0 0 25 0 9 0 720704364 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366450 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=5185264 CPUtime=4902.06 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 663897 1251 0 0 490085 121 0 0 25 0 9 0 720704365 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=5185264 CPUtime=4870.39 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 689552 1251 0 0 486935 104 0 0 25 0 9 0 720704365 5309710336 1241219 33554432000 4194304 4417452 140736756900768 18446744073709551615 4344164 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39155.2
Current children cumulated vsize (KiB) 5185264

[startup+4962.3 s]
/proc/loadavg: 8.11 8.06 8.00 9/388 29838
/proc/meminfo: memFree=6167500/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=4681060 CPUtime=39634.4 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 6661822 1251 0 0 3962212 1230 0 0 21 0 9 0 720704347 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 1170265 1115168 180 55 0 1168151 0
[pid=29680/tid=29687] ppid=29678 vsize=4681060 CPUtime=4961.98 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 743073 1251 0 0 496028 170 0 0 25 0 9 0 720704364 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=4681060 CPUtime=4934.6 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 569193 1251 0 0 493353 107 0 0 25 0 9 0 720704364 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=4681060 CPUtime=4962.2 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 930003 1251 0 0 496066 154 0 0 25 0 9 0 720704364 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=4681060 CPUtime=4962.25 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 1660904 1251 0 0 495885 340 0 0 25 0 9 0 720704364 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=4681060 CPUtime=4959.25 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 638977 1251 0 0 495822 103 0 0 25 0 9 0 720704364 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=4681060 CPUtime=4962.19 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 757354 1251 0 0 496093 126 0 0 25 0 9 0 720704364 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=4681060 CPUtime=4962.06 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 663897 1251 0 0 496085 121 0 0 25 0 9 0 720704365 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4365050 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=4681060 CPUtime=4929.65 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 689552 1251 0 0 492861 104 0 0 25 0 9 0 720704365 4793405440 1115168 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39634.4
Current children cumulated vsize (KiB) 4681060



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.1 s]
/proc/loadavg: 8.05 8.05 8.00 9/388 29840
/proc/meminfo: memFree=6306036/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=4542936 CPUtime=40000.4 cores=0-7
/proc/29680/stat : 29680 (plingeling) S 29678 29680 29455 0 -1 4202496 6679415 1251 0 0 3998802 1233 0 0 21 0 9 0 720704347 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 218382759909 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 1135734 1080637 180 55 0 1133620 0
[pid=29680/tid=29687] ppid=29678 vsize=4542936 CPUtime=5007.8 cores=0-7
/proc/29680/task/29687/stat : 29687 (plingeling) R 29678 29680 29455 0 -1 4202560 743073 1251 0 0 500610 170 0 0 25 0 9 0 720704364 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=29680/tid=29688] ppid=29678 vsize=4542936 CPUtime=4980.41 cores=0-7
/proc/29680/task/29688/stat : 29688 (plingeling) R 29678 29680 29455 0 -1 4202560 569193 1251 0 0 497934 107 0 0 25 0 9 0 720704364 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29680/tid=29689] ppid=29678 vsize=4542936 CPUtime=5008.01 cores=0-7
/proc/29680/task/29689/stat : 29689 (plingeling) R 29678 29680 29455 0 -1 4202560 932363 1251 0 0 500647 154 0 0 25 0 9 0 720704364 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=29680/tid=29690] ppid=29678 vsize=4542936 CPUtime=5008.06 cores=0-7
/proc/29680/task/29690/stat : 29690 (plingeling) R 29678 29680 29455 0 -1 4202560 1660904 1251 0 0 500466 340 0 0 25 0 9 0 720704364 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29680/tid=29691] ppid=29678 vsize=4542936 CPUtime=5005.07 cores=0-7
/proc/29680/task/29691/stat : 29691 (plingeling) R 29678 29680 29455 0 -1 4202560 638977 1251 0 0 500404 103 0 0 25 0 9 0 720704364 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29680/tid=29692] ppid=29678 vsize=4542936 CPUtime=5008.01 cores=0-7
/proc/29680/task/29692/stat : 29692 (plingeling) R 29678 29680 29455 0 -1 4202560 772587 1251 0 0 500672 129 0 0 25 0 9 0 720704364 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29680/tid=29693] ppid=29678 vsize=4542936 CPUtime=5007.88 cores=0-7
/proc/29680/task/29693/stat : 29693 (plingeling) R 29678 29680 29455 0 -1 4202560 663897 1251 0 0 500667 121 0 0 25 0 9 0 720704365 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=4542936 CPUtime=4974.88 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4202560 689552 1251 0 0 497384 104 0 0 25 0 9 0 720704365 4651966464 1080637 33554432000 4194304 4417452 140736756900768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 4542936

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5008.22 s]
/proc/loadavg: 8.05 8.05 8.00 9/388 29840
/proc/meminfo: memFree=6306036/32951124 swapFree=52307820/67111528
[pid=29680] ppid=29678 vsize=0 CPUtime=40000.5 cores=0-7
/proc/29680/stat : 29680 (plingeling) Z 29678 29680 29455 0 -1 4203548 6679415 1251 0 0 3998806 1243 0 0 18 0 2 0 720704347 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 6 0 0 0
/proc/29680/statm: 0 0 0 0 0 0 0
[pid=29680/tid=29694] ppid=29678 vsize=0 CPUtime=4974.98 cores=0-7
/proc/29680/task/29694/stat : 29694 (plingeling) R 29678 29680 29455 0 -1 4203588 689552 1251 0 0 497384 114 0 0 25 0 2 0 720704365 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5008.28
CPU time (s): 40000.6
CPU user time (s): 39988.1
CPU system time (s): 12.4901
CPU usage (%): 798.689
Max. virtual memory (cumulated for all children) (KiB): 5544696

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39988.1
system time used= 12.4901
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6680666
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 630
involuntary context switches= 193814

runsolver used 17.7423 second user time and 43.7184 second system time

The end

Launcher Data 

Begin job on node101 at 2011-05-22 21:03:36
IDJOB=3394124
IDBENCH=71593
IDSOLVER=1851
FILE ID=node101/3394124-1306091016
RUNJOBID= node101-1306086005-29469
PBS_JOBID= 13387921
Free space on /tmp= 73576 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v1560c133795gyes1.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394124-1306091016/watcher-3394124-1306091016 -o /tmp/evaluation-result-3394124-1306091016/solver-3394124-1306091016 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394124-1306091016.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 118632220c6f3388c45a2582e9a2ac0e
RANDOM SEED=658254935

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      10620172 kB
Buffers:        370140 kB
Cached:       10442424 kB
SwapCached:      14796 kB
Active:       14821108 kB
Inactive:      7193012 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      10620172 kB
SwapTotal:    67111528 kB
SwapFree:     52307820 kB
Dirty:            1808 kB
Writeback:           0 kB
AnonPages:    11198212 kB
Mapped:          24596 kB
Slab:           172664 kB
PageTables:      74964 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 78598244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73576 MiB
End job on node101 at 2011-05-22 22:27:06