Trace number 3394100

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.5 5010.82

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v945c61409g3.cnf
MD5SUM4de3b21cb74a7ec60763201555524736
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.619904
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61409
Sum of the clauses size123673
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261364
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394100-1306089431/watcher-3394100-1306089431 -o /tmp/evaluation-result-3394100-1306089431/solver-3394100-1306089431 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394100-1306089431.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.41 7.92 7.96 1/481 3799
/proc/meminfo: memFree=16957728/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=9140 CPUtime=0 cores=0-7
/proc/3799/stat : 3799 (plingeling) R 3797 3799 3773 0 -1 4202496 374 1262 0 0 0 0 0 0 25 0 1 0 453134367 9359360 304 33554432000 4194304 4417452 140734717713568 18446744073709551615 268660662016 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/3799/statm: 2285 304 147 55 0 171 0

[startup+0.0684901 s]
/proc/loadavg: 7.41 7.92 7.96 1/481 3799
/proc/meminfo: memFree=16957728/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=25200 CPUtime=0.06 cores=0-7
/proc/3799/stat : 3799 (plingeling) R 3797 3799 3773 0 -1 4202496 4228 1262 0 0 6 0 0 0 25 0 1 0 453134367 25804800 4158 33554432000 4194304 4417452 140734717713568 18446744073709551615 268660356189 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/3799/statm: 6300 4185 148 55 0 4186 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 25200

[startup+0.10043 s]
/proc/loadavg: 7.41 7.92 7.96 1/481 3799
/proc/meminfo: memFree=16957728/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=174324 CPUtime=0.2 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 4838 1262 0 0 20 0 0 0 25 0 9 0 453134367 178507776 4758 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 43581 4758 173 55 0 41467 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 174324

[startup+0.300397 s]
/proc/loadavg: 7.41 7.92 7.96 1/481 3799
/proc/meminfo: memFree=16957728/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=180388 CPUtime=1.77 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 9212 1262 0 0 176 1 0 0 25 0 9 0 453134367 184717312 6268 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 45097 6268 180 55 0 42983 0
Current children cumulated CPU time (s) 1.77
Current children cumulated vsize (KiB) 180388

[startup+0.700325 s]
/proc/loadavg: 7.41 7.92 7.96 1/481 3799
/proc/meminfo: memFree=16957728/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=191276 CPUtime=4.96 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 12938 1262 0 0 495 1 0 0 25 0 9 0 453134367 195866624 9019 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 47819 9019 180 55 0 45705 0
Current children cumulated CPU time (s) 4.96
Current children cumulated vsize (KiB) 191276

[startup+1.50119 s]
/proc/loadavg: 7.41 7.92 7.96 9/490 3813
/proc/meminfo: memFree=16925076/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=192216 CPUtime=11.35 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 16152 1262 0 0 1133 2 0 0 25 0 9 0 453134367 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 48054 9254 180 55 0 45940 0
[pid=3799/tid=3806] ppid=3797 vsize=192216 CPUtime=1.42 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 959 1262 0 0 142 0 0 0 20 0 9 0 453134374 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249609 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=192216 CPUtime=1.4 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 476 1262 0 0 140 0 0 0 25 0 9 0 453134374 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=192216 CPUtime=1.42 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 822 1262 0 0 142 0 0 0 25 0 9 0 453134374 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4365782 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=192216 CPUtime=1.41 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 6652 1262 0 0 140 1 0 0 25 0 9 0 453134374 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4365204 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=192216 CPUtime=1.42 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 701 1262 0 0 142 0 0 0 25 0 9 0 453134374 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=192216 CPUtime=1.38 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 432 1262 0 0 138 0 0 0 25 0 9 0 453134375 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=192216 CPUtime=1.39 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 611 1262 0 0 139 0 0 0 25 0 9 0 453134375 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=192216 CPUtime=1.4 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 854 1262 0 0 140 0 0 0 25 0 9 0 453134375 196829184 9254 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.35
Current children cumulated vsize (KiB) 192216

[startup+3.1009 s]
/proc/loadavg: 7.41 7.92 7.96 9/490 3813
/proc/meminfo: memFree=16910196/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=204800 CPUtime=24.14 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 23545 1262 0 0 2411 3 0 0 25 0 9 0 453134367 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 51200 12400 180 55 0 49086 0
[pid=3799/tid=3806] ppid=3797 vsize=204800 CPUtime=3.02 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1455 1262 0 0 302 0 0 0 25 0 9 0 453134374 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 4235985 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=204800 CPUtime=2.98 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 476 1262 0 0 298 0 0 0 25 0 9 0 453134374 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 4250080 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=204800 CPUtime=3.02 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 1078 1262 0 0 302 0 0 0 25 0 9 0 453134374 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=204800 CPUtime=3.01 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 11586 1262 0 0 300 1 0 0 25 0 9 0 453134374 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=204800 CPUtime=3.02 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 1197 1262 0 0 302 0 0 0 25 0 9 0 453134374 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=204800 CPUtime=2.98 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 928 1262 0 0 298 0 0 0 25 0 9 0 453134375 209715200 12400 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=206080 CPUtime=2.99 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 830 1262 0 0 299 0 0 0 25 0 9 0 453134375 211025920 12401 33554432000 4194304 4417452 140734717713568 18446744073709551615 4277563 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=206080 CPUtime=3 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1350 1262 0 0 300 0 0 0 25 0 9 0 453134375 211025920 12414 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.14
Current children cumulated vsize (KiB) 204800

[startup+6.30034 s]
/proc/loadavg: 7.46 7.92 7.96 9/490 3813
/proc/meminfo: memFree=16894200/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=212748 CPUtime=49.68 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 37067 1262 0 0 4963 5 0 0 25 0 9 0 453134367 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 53187 14419 180 55 0 51073 0
[pid=3799/tid=3806] ppid=3797 vsize=212748 CPUtime=6.22 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1455 1262 0 0 622 0 0 0 25 0 9 0 453134374 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4258620 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=212748 CPUtime=6.13 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 476 1262 0 0 613 0 0 0 25 0 9 0 453134374 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4235968 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=212748 CPUtime=6.22 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 2665 1262 0 0 622 0 0 0 25 0 9 0 453134374 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=212748 CPUtime=6.21 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 21060 1262 0 0 618 3 0 0 25 0 9 0 453134374 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=212748 CPUtime=6.22 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 1197 1262 0 0 622 0 0 0 25 0 9 0 453134374 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4336695 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=212748 CPUtime=6.18 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 928 1262 0 0 618 0 0 0 25 0 9 0 453134375 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4236302 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=212748 CPUtime=6.19 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 3291 1262 0 0 619 0 0 0 25 0 9 0 453134375 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=212748 CPUtime=6.2 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1350 1262 0 0 620 0 0 0 25 0 9 0 453134375 217853952 14419 33554432000 4194304 4417452 140734717713568 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.68
Current children cumulated vsize (KiB) 212748

[startup+12.7012 s]
/proc/loadavg: 7.50 7.92 7.96 9/490 3813
/proc/meminfo: memFree=16864756/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=236116 CPUtime=100.77 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 66474 1262 0 0 10066 11 0 0 25 0 9 0 453134367 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 59029 20229 180 55 0 56915 0
[pid=3799/tid=3806] ppid=3797 vsize=236116 CPUtime=12.61 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 4692 1262 0 0 1261 0 0 0 25 0 9 0 453134374 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=236116 CPUtime=12.42 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 2151 1262 0 0 1242 0 0 0 19 0 9 0 453134374 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=236116 CPUtime=12.6 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 4765 1262 0 0 1260 0 0 0 25 0 9 0 453134374 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4236544 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=236116 CPUtime=12.62 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 34867 1262 0 0 1256 6 0 0 25 0 9 0 453134374 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=236116 CPUtime=12.61 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 2880 1262 0 0 1261 0 0 0 25 0 9 0 453134374 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=236116 CPUtime=12.57 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 3974 1262 0 0 1257 0 0 0 25 0 9 0 453134375 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4232595 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=236116 CPUtime=12.59 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 4746 1262 0 0 1259 0 0 0 25 0 9 0 453134375 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=236116 CPUtime=12.6 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 3754 1262 0 0 1260 0 0 0 25 0 9 0 453134375 241782784 20229 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364474 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 100.77
Current children cumulated vsize (KiB) 236116

[startup+25.501 s]
/proc/loadavg: 7.61 7.92 7.96 9/490 3813
/proc/meminfo: memFree=16807160/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=310248 CPUtime=202.96 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 159657 1262 0 0 20265 31 0 0 25 0 9 0 453134367 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 77562 38768 180 55 0 75448 0
[pid=3799/tid=3806] ppid=3797 vsize=310248 CPUtime=25.41 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 16560 1262 0 0 2539 2 0 0 25 0 9 0 453134374 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=310248 CPUtime=25.02 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 10063 1262 0 0 2500 2 0 0 25 0 9 0 453134374 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4232693 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=310248 CPUtime=25.41 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 12106 1262 0 0 2539 2 0 0 25 0 9 0 453134374 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4365663 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=310248 CPUtime=25.41 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 55028 1262 0 0 2531 10 0 0 25 0 9 0 453134374 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4321595 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=310248 CPUtime=25.41 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 15380 1262 0 0 2538 3 0 0 25 0 9 0 453134374 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=310248 CPUtime=25.38 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 15602 1262 0 0 2535 3 0 0 25 0 9 0 453134375 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=310248 CPUtime=25.38 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 15731 1262 0 0 2536 2 0 0 25 0 9 0 453134375 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4365042 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=310248 CPUtime=25.4 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 14542 1262 0 0 2538 2 0 0 25 0 9 0 453134375 317693952 38768 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366094 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.96
Current children cumulated vsize (KiB) 310248

[startup+51.1005 s]
/proc/loadavg: 7.74 7.93 7.96 9/490 3814
/proc/meminfo: memFree=16779940/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=328888 CPUtime=407.36 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 302576 1262 0 0 40679 57 0 0 25 0 9 0 453134367 336781312 43159 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 82222 43159 180 55 0 80108 0
[pid=3799/tid=3806] ppid=3797 vsize=328888 CPUtime=51.01 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 34272 1262 0 0 5095 6 0 0 25 0 9 0 453134374 336781312 43159 33554432000 4194304 4417452 140734717713568 18446744073709551615 4306336 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=328888 CPUtime=50.22 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 30367 1262 0 0 5017 5 0 0 25 0 9 0 453134374 336781312 43159 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=328888 CPUtime=51.01 cores=0-7

################
# More data... #
################


[startup+4722.32 s]
/proc/loadavg: 8.22 8.09 8.01 11/491 3954
/proc/meminfo: memFree=12009640/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=5080620 CPUtime=37699.3 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 7684900 1262 0 0 3768475 1456 0 0 25 0 9 0 453134367 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 1270155 1231354 180 55 0 1268041 0
[pid=3799/tid=3806] ppid=3797 vsize=5080620 CPUtime=4721.76 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 975329 1262 0 0 472011 165 0 0 25 0 9 0 453134374 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=5080620 CPUtime=4644.83 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 1052320 1262 0 0 464288 195 0 0 25 0 9 0 453134374 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4236544 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=5080620 CPUtime=4721.01 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 978914 1262 0 0 471917 184 0 0 25 0 9 0 453134374 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=5080620 CPUtime=4722.34 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 864047 1262 0 0 472043 191 0 0 25 0 9 0 453134374 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=5080620 CPUtime=4722.33 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 908749 1262 0 0 472060 173 0 0 25 0 9 0 453134374 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=5080620 CPUtime=4722.28 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 987538 1262 0 0 472061 167 0 0 25 0 9 0 453134375 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=5080620 CPUtime=4722.32 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 881353 1262 0 0 472045 187 0 0 25 0 9 0 453134375 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=5080620 CPUtime=4722.28 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1032005 1262 0 0 472039 189 0 0 25 0 9 0 453134375 5202554880 1231354 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37699.3
Current children cumulated vsize (KiB) 5080620

[startup+4782.3 s]
/proc/loadavg: 8.31 8.14 8.04 9/490 3955
/proc/meminfo: memFree=11928096/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=5175820 CPUtime=38177.5 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 7735084 1262 0 0 3816289 1461 0 0 25 0 9 0 453134367 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 1293955 1255154 180 55 0 1291841 0
[pid=3799/tid=3806] ppid=3797 vsize=5175820 CPUtime=4781.06 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1025513 1262 0 0 477936 170 0 0 25 0 9 0 453134374 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=5175820 CPUtime=4703.84 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 1052320 1262 0 0 470189 195 0 0 25 0 9 0 453134374 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=5175820 CPUtime=4780.97 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 978914 1262 0 0 477913 184 0 0 25 0 9 0 453134374 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=5175820 CPUtime=4782.32 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 864047 1262 0 0 478041 191 0 0 25 0 9 0 453134374 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=5175820 CPUtime=4782.32 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 908749 1262 0 0 478059 173 0 0 25 0 9 0 453134374 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=5175820 CPUtime=4782.27 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 987538 1262 0 0 478060 167 0 0 25 0 9 0 453134375 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364583 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=5175820 CPUtime=4782.3 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 881353 1262 0 0 478043 187 0 0 25 0 9 0 453134375 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=5175820 CPUtime=4782.26 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1032005 1262 0 0 478037 189 0 0 25 0 9 0 453134375 5300039680 1255154 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38177.5
Current children cumulated vsize (KiB) 5175820

[startup+4842.3 s]
/proc/loadavg: 8.16 8.13 8.03 9/490 3957
/proc/meminfo: memFree=11481792/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=5621108 CPUtime=38656.5 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 7909927 1262 0 0 3864143 1507 0 0 25 0 9 0 453134367 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 1405277 1366475 180 55 0 1403163 0
[pid=3799/tid=3806] ppid=3797 vsize=5621108 CPUtime=4841.06 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1025513 1262 0 0 483936 170 0 0 25 0 9 0 453134374 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366102 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=5621108 CPUtime=4762.85 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 1052320 1262 0 0 476090 195 0 0 25 0 9 0 453134374 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=5621108 CPUtime=4840.96 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 978914 1262 0 0 483912 184 0 0 25 0 9 0 453134374 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=5621108 CPUtime=4842.32 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 864047 1262 0 0 484041 191 0 0 25 0 9 0 453134374 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=5621108 CPUtime=4842.32 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 1021614 1262 0 0 484025 207 0 0 25 0 9 0 453134374 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=5621108 CPUtime=4842.27 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 987538 1262 0 0 484060 167 0 0 25 0 9 0 453134375 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=5621108 CPUtime=4842.3 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 943331 1262 0 0 484031 199 0 0 25 0 9 0 453134375 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=5621108 CPUtime=4842.26 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1032005 1262 0 0 484037 189 0 0 25 0 9 0 453134375 5756014592 1366475 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38656.5
Current children cumulated vsize (KiB) 5621108

[startup+4902.3 s]
/proc/loadavg: 8.11 8.12 8.03 9/490 3959
/proc/meminfo: memFree=11416660/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=5686644 CPUtime=39135.5 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 7926311 1262 0 0 3912041 1512 0 0 25 0 9 0 453134367 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 1421661 1382859 180 55 0 1419547 0
[pid=3799/tid=3806] ppid=3797 vsize=5686644 CPUtime=4901.06 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1025513 1262 0 0 489936 170 0 0 25 0 9 0 453134374 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=5686644 CPUtime=4821.88 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 1052320 1262 0 0 481993 195 0 0 25 0 9 0 453134374 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=5686644 CPUtime=4900.95 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 978914 1262 0 0 489911 184 0 0 25 0 9 0 453134374 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=5686644 CPUtime=4902.32 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 880431 1262 0 0 490036 196 0 0 25 0 9 0 453134374 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=5686644 CPUtime=4902.32 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 1021614 1262 0 0 490025 207 0 0 25 0 9 0 453134374 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364903 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=5686644 CPUtime=4902.27 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 987538 1262 0 0 490060 167 0 0 25 0 9 0 453134375 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=5686644 CPUtime=4902.3 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 943331 1262 0 0 490031 199 0 0 25 0 9 0 453134375 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=5686644 CPUtime=4902.26 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1032005 1262 0 0 490037 189 0 0 25 0 9 0 453134375 5823123456 1382859 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39135.5
Current children cumulated vsize (KiB) 5686644

[startup+4962.3 s]
/proc/loadavg: 8.08 8.11 8.03 9/490 3960
/proc/meminfo: memFree=11224416/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=5878132 CPUtime=39614.5 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 7974183 1262 0 0 3959925 1527 0 0 25 0 9 0 453134367 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 1469533 1430731 180 55 0 1467419 0
[pid=3799/tid=3806] ppid=3797 vsize=5878132 CPUtime=4961.06 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1025513 1262 0 0 495936 170 0 0 25 0 9 0 453134374 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=5878132 CPUtime=4880.89 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 1052320 1262 0 0 487894 195 0 0 25 0 9 0 453134374 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=5878132 CPUtime=4960.92 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 978914 1262 0 0 495908 184 0 0 25 0 9 0 453134374 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=5878132 CPUtime=4962.33 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 928303 1262 0 0 496021 212 0 0 25 0 9 0 453134374 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=5878132 CPUtime=4962.32 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 1021614 1262 0 0 496025 207 0 0 25 0 9 0 453134374 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4365027 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=5878132 CPUtime=4962.27 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 987538 1262 0 0 496060 167 0 0 25 0 9 0 453134375 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=5878132 CPUtime=4962.3 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 943331 1262 0 0 496031 199 0 0 25 0 9 0 453134375 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=5878132 CPUtime=4962.26 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1032005 1262 0 0 496037 189 0 0 25 0 9 0 453134375 6019207168 1430731 33554432000 4194304 4417452 140734717713568 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39614.5
Current children cumulated vsize (KiB) 5878132



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.6 s]
/proc/loadavg: 8.09 8.11 8.03 9/490 3961
/proc/meminfo: memFree=11703516/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=5399956 CPUtime=40000.2 cores=0-7
/proc/3799/stat : 3799 (plingeling) S 3797 3799 3773 0 -1 4202496 8081065 1262 0 0 3998470 1547 0 0 25 0 9 0 453134367 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 1349989 1311188 180 55 0 1347875 0
[pid=3799/tid=3806] ppid=3797 vsize=5399956 CPUtime=5009.38 cores=0-7
/proc/3799/task/3806/stat : 3806 (plingeling) R 3797 3799 3773 0 -1 4202560 1025513 1262 0 0 500768 170 0 0 25 0 9 0 453134374 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=5399956 CPUtime=4928.4 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4202560 1052320 1262 0 0 492645 195 0 0 25 0 9 0 453134374 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3799/tid=3808] ppid=3797 vsize=5399956 CPUtime=5009.23 cores=0-7
/proc/3799/task/3808/stat : 3808 (plingeling) R 3797 3799 3773 0 -1 4202560 978914 1262 0 0 500739 184 0 0 25 0 9 0 453134374 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3799/tid=3809] ppid=3797 vsize=5399956 CPUtime=5010.65 cores=0-7
/proc/3799/task/3809/stat : 3809 (plingeling) R 3797 3799 3773 0 -1 4202560 928303 1262 0 0 500853 212 0 0 25 0 9 0 453134374 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3799/tid=3810] ppid=3797 vsize=5399956 CPUtime=5010.64 cores=0-7
/proc/3799/task/3810/stat : 3810 (plingeling) R 3797 3799 3773 0 -1 4202560 1062863 1262 0 0 500851 213 0 0 25 0 9 0 453134374 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3799/tid=3811] ppid=3797 vsize=5399956 CPUtime=5010.59 cores=0-7
/proc/3799/task/3811/stat : 3811 (plingeling) R 3797 3799 3773 0 -1 4202560 987538 1262 0 0 500892 167 0 0 25 0 9 0 453134375 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3799/tid=3812] ppid=3797 vsize=5399956 CPUtime=5010.62 cores=0-7
/proc/3799/task/3812/stat : 3812 (plingeling) R 3797 3799 3773 0 -1 4202560 1008964 1262 0 0 500850 212 0 0 25 0 9 0 453134375 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3799/tid=3813] ppid=3797 vsize=5399956 CPUtime=5010.58 cores=0-7
/proc/3799/task/3813/stat : 3813 (plingeling) R 3797 3799 3773 0 -1 4202560 1032005 1262 0 0 500869 189 0 0 25 0 9 0 453134375 5529554944 1311188 33554432000 4194304 4417452 140734717713568 18446744073709551615 4364972 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 5399956

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010.8 s]
/proc/loadavg: 8.09 8.11 8.03 9/490 3961
/proc/meminfo: memFree=11703516/32951124 swapFree=46575304/67111528
[pid=3799] ppid=3797 vsize=0 CPUtime=40000.5 cores=0-7
/proc/3799/stat : 3799 (plingeling) Z 3797 3799 3773 0 -1 4203548 8081065 1262 0 0 3998484 1564 0 0 18 0 2 0 453134367 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/3799/statm: 0 0 0 0 0 0 0
[pid=3799/tid=3807] ppid=3797 vsize=0 CPUtime=4928.59 cores=0-7
/proc/3799/task/3807/stat : 3807 (plingeling) R 3797 3799 3773 0 -1 4203588 1052320 1262 0 0 492646 213 0 0 25 0 2 0 453134374 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5010.82
CPU time (s): 40000.5
CPU user time (s): 39984.8
CPU system time (s): 15.6656
CPU usage (%): 798.283
Max. virtual memory (cumulated for all children) (KiB): 5907368

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39984.8
system time used= 15.6656
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 8082327
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 301
involuntary context switches= 193212

runsolver used 23.0255 second user time and 58.8231 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-22 20:37:11
IDJOB=3394100
IDBENCH=71481
IDSOLVER=1851
FILE ID=node107/3394100-1306089431
RUNJOBID= node107-1306089431-3787
PBS_JOBID= 13387927
Free space on /tmp= 73192 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v945c61409g3.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394100-1306089431/watcher-3394100-1306089431 -o /tmp/evaluation-result-3394100-1306089431/solver-3394100-1306089431 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394100-1306089431.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4de3b21cb74a7ec60763201555524736
RANDOM SEED=1151338706

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16958008 kB
Buffers:        298400 kB
Cached:        6683764 kB
SwapCached:       9840 kB
Active:       11693120 kB
Inactive:      3976692 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16958008 kB
SwapTotal:    67111528 kB
SwapFree:     46575304 kB
Dirty:             784 kB
Writeback:           0 kB
AnonPages:     8684964 kB
Mapped:          31904 kB
Slab:           156884 kB
PageTables:      95156 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 145073124 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73192 MiB
End job on node107 at 2011-05-22 22:00:44