Trace number 3264058

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.12 1201.75

General information on the benchmark

NameSAT04/handmade/kexu/frb-0.6/
frb75-13-2.used-as.sat04-878.cnf
MD5SUM00e6f1cbbaaa8f43dc6451d49bfa1cef
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.241962
Satisfiable
(Un)Satisfiability was proved
Number of variables975
Number of clauses34275
Sum of the clauses size69375
Maximum clause length13
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 234200
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264058-1304102889/watcher-3264058-1304102889 -o /tmp/evaluation-result-3264058-1304102889/solver-3264058-1304102889 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3264058-1304102889.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.95 7.88 7.75 3/173 16028
/proc/meminfo: memFree=23094656/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=8728 CPUtime=0 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 215 0 0 0 0 0 0 0 21 0 1 0 521889351 8937472 146 33554432000 4194304 4417332 140734867966032 18446744073709551615 244446945024 0 0 4096 0 18446744071562362840 0 0 17 2 0 0 0
/proc/16027/statm: 2182 146 122 55 0 68 0

[startup+0.035699 s]
/proc/loadavg: 6.95 7.88 7.75 3/173 16028
/proc/meminfo: memFree=23094656/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=8728 CPUtime=0 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 215 0 0 0 0 0 0 0 21 0 1 0 521889351 8937472 146 33554432000 4194304 4417332 140734867966032 18446744073709551615 244446945024 0 0 4096 0 18446744071562362840 0 0 17 2 0 0 0
/proc/16027/statm: 2182 146 122 55 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8728

[startup+0.104331 s]
/proc/loadavg: 6.95 7.88 7.75 3/173 16028
/proc/meminfo: memFree=23094656/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=55628 CPUtime=0.16 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 2131 1250 0 4 16 0 0 0 21 0 5 0 521889351 56963072 1628 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 13907 1652 177 55 0 11793 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 55628

[startup+0.300579 s]
/proc/loadavg: 6.95 7.88 7.75 3/173 16028
/proc/meminfo: memFree=23094656/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=125332 CPUtime=0.94 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 3725 1250 0 4 94 0 0 0 21 0 5 0 521889351 128339968 2739 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 31333 2739 181 55 0 29219 0
Current children cumulated CPU time (s) 0.94
Current children cumulated vsize (KiB) 125332

[startup+0.700505 s]
/proc/loadavg: 6.95 7.88 7.75 3/173 16028
/proc/meminfo: memFree=23094656/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=125080 CPUtime=2.54 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 4901 1250 0 4 254 0 0 0 21 0 5 0 521889351 128081920 2701 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 31270 2701 181 55 0 29156 0
Current children cumulated CPU time (s) 2.54
Current children cumulated vsize (KiB) 125080

[startup+1.50036 s]
/proc/loadavg: 6.95 7.88 7.75 9/186 16051
/proc/meminfo: memFree=23063272/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=133504 CPUtime=5.74 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 8408 1250 0 4 574 0 0 0 21 0 5 0 521889351 136708096 4788 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 33376 4788 181 55 0 31262 0
[pid=16027/tid=16044] ppid=16025 vsize=133504 CPUtime=1.43 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 719 1250 0 4 143 0 0 0 25 0 5 0 521889358 136708096 4788 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=133504 CPUtime=1.43 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 637 1250 0 4 143 0 0 0 25 0 5 0 521889358 136708096 4788 33554432000 4194304 4417332 140734867966032 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=133504 CPUtime=1.42 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 1448 1250 0 4 142 0 0 0 25 0 5 0 521889358 136708096 4788 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364541 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=133504 CPUtime=1.42 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 4098 1250 0 4 142 0 0 0 25 0 5 0 521889358 136708096 4788 33554432000 4194304 4417332 140734867966032 18446744073709551615 4354414 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.74
Current children cumulated vsize (KiB) 133504

[startup+3.10106 s]
/proc/loadavg: 6.95 7.88 7.75 9/186 16051
/proc/meminfo: memFree=23055212/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=139384 CPUtime=12.13 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 15868 1250 0 4 1211 2 0 0 21 0 5 0 521889351 142729216 6285 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 34846 6285 181 55 0 32732 0
[pid=16027/tid=16044] ppid=16025 vsize=139384 CPUtime=3.03 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 1530 1250 0 4 303 0 0 0 25 0 5 0 521889358 142729216 6285 33554432000 4194304 4417332 140734867966032 18446744073709551615 4336603 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=139384 CPUtime=3.02 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 1646 1250 0 4 302 0 0 0 25 0 5 0 521889358 142729216 6285 33554432000 4194304 4417332 140734867966032 18446744073709551615 4331757 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=139384 CPUtime=3.01 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 2818 1250 0 4 301 0 0 0 25 0 5 0 521889358 142729216 6285 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=139384 CPUtime=3.03 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 8368 1250 0 4 302 1 0 0 25 0 5 0 521889358 142729216 6285 33554432000 4194304 4417332 140734867966032 18446744073709551615 4366011 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 12.13
Current children cumulated vsize (KiB) 139384

[startup+6.30046 s]
/proc/loadavg: 7.04 7.88 7.75 9/186 16051
/proc/meminfo: memFree=23034668/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=155232 CPUtime=24.91 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 28856 1250 0 4 2487 4 0 0 21 0 5 0 521889351 158957568 10278 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 38808 10278 181 55 0 36694 0
[pid=16027/tid=16044] ppid=16025 vsize=155232 CPUtime=6.22 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 4960 1250 0 4 622 0 0 0 25 0 5 0 521889358 158957568 10278 33554432000 4194304 4417332 140734867966032 18446744073709551615 4355987 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=155232 CPUtime=6.22 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 4623 1250 0 4 622 0 0 0 25 0 5 0 521889358 158957568 10278 33554432000 4194304 4417332 140734867966032 18446744073709551615 4365963 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=155232 CPUtime=6.19 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 3876 1250 0 4 619 0 0 0 25 0 5 0 521889358 158957568 10278 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364568 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=155232 CPUtime=6.23 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 13891 1250 0 4 621 2 0 0 25 0 5 0 521889358 158957568 10278 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364776 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.91
Current children cumulated vsize (KiB) 155232

[startup+12.7003 s]
/proc/loadavg: 7.11 7.89 7.75 9/186 16051
/proc/meminfo: memFree=22981236/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=155736 CPUtime=50.48 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 51800 1250 0 4 5040 8 0 0 21 0 5 0 521889351 159473664 10386 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 38934 10386 181 55 0 36820 0
[pid=16027/tid=16044] ppid=16025 vsize=155736 CPUtime=12.62 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 9107 1250 0 4 1261 1 0 0 25 0 5 0 521889358 159473664 10386 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=155736 CPUtime=12.62 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 10519 1250 0 4 1261 1 0 0 25 0 5 0 521889358 159473664 10386 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=155736 CPUtime=12.56 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 8054 1250 0 4 1255 1 0 0 25 0 5 0 521889358 159473664 10386 33554432000 4194304 4417332 140734867966032 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=155736 CPUtime=12.63 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 22614 1250 0 4 1260 3 0 0 25 0 5 0 521889358 159473664 10386 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 50.48
Current children cumulated vsize (KiB) 155736

[startup+25.5009 s]
/proc/loadavg: 7.46 7.92 7.77 9/186 16052
/proc/meminfo: memFree=22972552/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=170524 CPUtime=101.61 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 90818 1250 0 4 10148 13 0 0 21 0 5 0 521889351 174616576 14070 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 42631 14070 181 55 0 40517 0
[pid=16027/tid=16044] ppid=16025 vsize=170524 CPUtime=25.42 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 17726 1250 0 4 2540 2 0 0 25 0 5 0 521889358 174616576 14070 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=170524 CPUtime=25.43 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 19109 1250 0 4 2540 3 0 0 25 0 5 0 521889358 174616576 14070 33554432000 4194304 4417332 140734867966032 18446744073709551615 4236945 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=170524 CPUtime=25.29 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 15939 1250 0 4 2527 2 0 0 25 0 5 0 521889358 174616576 14070 33554432000 4194304 4417332 140734867966032 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=170524 CPUtime=25.43 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 36538 1250 0 4 2538 5 0 0 25 0 5 0 521889358 174616576 14070 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.61
Current children cumulated vsize (KiB) 170524

[startup+51.1011 s]
/proc/loadavg: 7.64 7.93 7.77 9/186 16053
/proc/meminfo: memFree=22927524/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=202496 CPUtime=203.87 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 158141 1250 0 4 20364 23 0 0 21 0 5 0 521889351 207355904 22063 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 50624 22063 181 55 0 48510 0
[pid=16027/tid=16044] ppid=16025 vsize=202496 CPUtime=51.03 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 32568 1250 0 4 5099 4 0 0 25 0 5 0 521889358 207355904 22063 33554432000 4194304 4417332 140734867966032 18446744073709551615 4308204 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=202496 CPUtime=51.02 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 37804 1250 0 4 5097 5 0 0 25 0 5 0 521889358 207355904 22063 33554432000 4194304 4417332 140734867966032 18446744073709551615 4235999 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=202496 CPUtime=50.75 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 31096 1250 0 4 5071 4 0 0 25 0 5 0 521889358 207355904 22063 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364639 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=202496 CPUtime=51.03 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 55167 1250 0 4 5095 8 0 0 25 0 5 0 521889358 207355904 22063 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.87
Current children cumulated vsize (KiB) 202496

[startup+102.307 s]
/proc/loadavg: 7.95 7.97 7.79 9/186 16054
/proc/meminfo: memFree=22852976/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=227488 CPUtime=408.42 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 255686 1250 0 4 40803 39 0 0 21 0 5 0 521889351 232947712 28311 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 56872 28311 181 55 0 54758 0
[pid=16027/tid=16044] ppid=16025 vsize=227488 CPUtime=102.23 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 52143 1250 0 4 10216 7 0 0 25 0 5 0 521889358 232947712 28311 33554432000 4194304 4417332 140734867966032 18446744073709551615 4278002 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=227488 CPUtime=102.22 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 61717 1250 0 4 10213 9 0 0 25 0 5 0 521889358 232947712 28311 33554432000 4194304 4417332 140734867966032 18446744073709551615 4260714 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=227488 CPUtime=101.68 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 52170 1250 0 4 10160 8 0 0 25 0 5 0 521889358 232947712 28311 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=227488 CPUtime=102.23 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 88150 1250 0 4 10210 13 0 0 25 0 5 0 521889358 232947712 28311 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 408.42
Current children cumulated vsize (KiB) 227488

[startup+162.3 s]
/proc/loadavg: 8.08 8.00 7.81 9/186 16055
/proc/meminfo: memFree=22725828/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=297032 CPUtime=648.07 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 329330 1250 0 4 64757 50 0 0 21 0 5 0 521889351 304160768 45697 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 74258 45697 181 55 0 72144 0
[pid=16027/tid=16044] ppid=16025 vsize=297032 CPUtime=162.23 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 72751 1250 0 4 16212 11 0 0 25 0 5 0 521889358 304160768 45697 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=297032 CPUtime=162.21 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 83989 1250 0 4 16209 12 0 0 25 0 5 0 521889358 304160768 45697 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=297032 CPUtime=161.34 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 70961 1250 0 4 16124 10 0 0 25 0 5 0 521889358 304160768 45697 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=297032 CPUtime=162.23 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 100123 1250 0 4 16208 15 0 0 25 0 5 0 521889358 304160768 45697 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 648.07
Current children cumulated vsize (KiB) 297032


################
# More data... #
################

[pid=16027/tid=16045] ppid=16025 vsize=812100 CPUtime=762.18 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 221323 1250 0 4 76183 35 0 0 25 0 5 0 521889358 831590400 174462 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364656 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=812100 CPUtime=758.1 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 183835 1250 0 4 75780 30 0 0 25 0 5 0 521889358 831590400 174462 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364722 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=812100 CPUtime=762.25 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 222329 1250 0 4 76189 36 0 0 25 0 5 0 521889358 831590400 174462 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3044.83
Current children cumulated vsize (KiB) 812100

[startup+822.301 s]
/proc/loadavg: 8.02 8.06 7.93 9/186 16082
/proc/meminfo: memFree=21553256/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=867920 CPUtime=3284.49 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 873240 1250 0 4 328306 143 0 0 21 0 5 0 521889351 888750080 188417 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 216980 188417 181 55 0 214866 0
[pid=16027/tid=16044] ppid=16025 vsize=867920 CPUtime=822.24 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 244247 1250 0 4 82184 40 0 0 25 0 5 0 521889358 888750080 188417 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364608 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=867920 CPUtime=822.17 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 221323 1250 0 4 82182 35 0 0 25 0 5 0 521889358 888750080 188417 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=867920 CPUtime=817.77 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 183835 1250 0 4 81747 30 0 0 25 0 5 0 521889358 888750080 188417 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=867920 CPUtime=822.25 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 222329 1250 0 4 82189 36 0 0 25 0 5 0 521889358 888750080 188417 33554432000 4194304 4417332 140734867966032 18446744073709551615 4237194 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3284.49
Current children cumulated vsize (KiB) 867920

[startup+882.301 s]
/proc/loadavg: 8.01 8.05 7.93 9/186 16084
/proc/meminfo: memFree=21472880/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=963152 CPUtime=3524.16 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 897048 1250 0 4 352270 146 0 0 21 0 5 0 521889351 986267648 212225 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 240788 212225 181 55 0 238674 0
[pid=16027/tid=16044] ppid=16025 vsize=963152 CPUtime=882.25 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 244247 1250 0 4 88185 40 0 0 25 0 5 0 521889358 986267648 212225 33554432000 4194304 4417332 140734867966032 18446744073709551615 4311312 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=963152 CPUtime=882.17 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 238750 1250 0 4 88179 38 0 0 25 0 5 0 521889358 986267648 212225 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=963152 CPUtime=877.45 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 183835 1250 0 4 87715 30 0 0 25 0 5 0 521889358 986267648 212225 33554432000 4194304 4417332 140734867966032 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=963152 CPUtime=882.25 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 228710 1250 0 4 88188 37 0 0 25 0 5 0 521889358 986267648 212225 33554432000 4194304 4417332 140734867966032 18446744073709551615 4365986 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3524.16
Current children cumulated vsize (KiB) 963152

[startup+942.301 s]
/proc/loadavg: 8.00 8.04 7.93 9/186 16087
/proc/meminfo: memFree=21510064/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=923192 CPUtime=3763.83 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 955543 1250 0 4 376228 155 0 0 21 0 5 0 521889351 945348608 202235 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 230798 202235 181 55 0 228684 0
[pid=16027/tid=16044] ppid=16025 vsize=923192 CPUtime=942.25 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 244247 1250 0 4 94185 40 0 0 25 0 5 0 521889358 945348608 202235 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=923192 CPUtime=942.16 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 240088 1250 0 4 94178 38 0 0 25 0 5 0 521889358 945348608 202235 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=923192 CPUtime=937.12 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 207658 1250 0 4 93678 34 0 0 25 0 5 0 521889358 945348608 202235 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=923192 CPUtime=942.26 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 262044 1250 0 4 94184 42 0 0 25 0 5 0 521889358 945348608 202235 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3763.83
Current children cumulated vsize (KiB) 923192

[startup+1002.3 s]
/proc/loadavg: 8.21 8.10 7.95 9/186 16090
/proc/meminfo: memFree=21560176/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=768028 CPUtime=4003.51 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 1015025 1250 0 4 400185 166 0 0 21 0 5 0 521889351 786460672 162730 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 192007 162730 181 55 0 189893 0
[pid=16027/tid=16044] ppid=16025 vsize=768028 CPUtime=1002.25 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 252996 1250 0 4 100183 42 0 0 25 0 5 0 521889358 786460672 162730 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=768028 CPUtime=1002.15 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 259574 1250 0 4 100174 41 0 0 25 0 5 0 521889358 786460672 162730 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=768028 CPUtime=996.79 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 234809 1250 0 4 99640 39 0 0 25 0 5 0 521889358 786460672 162730 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=768028 CPUtime=1002.25 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 266140 1250 0 4 100183 42 0 0 25 0 5 0 521889358 786460672 162730 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364713 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4003.51
Current children cumulated vsize (KiB) 768028

[startup+1062.3 s]
/proc/loadavg: 8.12 8.09 7.96 9/186 16091
/proc/meminfo: memFree=21531024/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=802088 CPUtime=4243.19 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 1028031 1250 0 4 424151 168 0 0 21 0 5 0 521889351 821338112 171245 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 200522 171245 181 55 0 198408 0
[pid=16027/tid=16044] ppid=16025 vsize=802088 CPUtime=1062.25 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 266002 1250 0 4 106182 43 0 0 25 0 5 0 521889358 821338112 171245 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=802088 CPUtime=1062.15 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 259574 1250 0 4 106174 41 0 0 25 0 5 0 521889358 821338112 171245 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=802088 CPUtime=1056.47 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 234809 1250 0 4 105608 39 0 0 25 0 5 0 521889358 821338112 171245 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=802088 CPUtime=1062.26 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 266140 1250 0 4 106183 43 0 0 25 0 5 0 521889358 821338112 171245 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364631 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4243.19
Current children cumulated vsize (KiB) 802088

[startup+1122.3 s]
/proc/loadavg: 8.09 8.09 7.97 9/186 16093
/proc/meminfo: memFree=21530376/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=875168 CPUtime=4482.86 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 1055589 1250 0 4 448116 170 0 0 21 0 5 0 521889351 896172032 189515 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 218792 189515 181 55 0 216678 0
[pid=16027/tid=16044] ppid=16025 vsize=875168 CPUtime=1122.26 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 282727 1250 0 4 112181 45 0 0 25 0 5 0 521889358 896172032 189515 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=875168 CPUtime=1122.15 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 259574 1250 0 4 112174 41 0 0 25 0 5 0 521889358 896172032 189515 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364695 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=875168 CPUtime=1116.14 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 234809 1250 0 4 111575 39 0 0 25 0 5 0 521889358 896172032 189515 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=875168 CPUtime=1122.26 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 276973 1250 0 4 112183 43 0 0 25 0 5 0 521889358 896172032 189515 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4482.86
Current children cumulated vsize (KiB) 875168

[startup+1182.3 s]
/proc/loadavg: 8.07 8.08 7.97 9/186 16095
/proc/meminfo: memFree=21493772/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=896360 CPUtime=4722.54 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 1060887 1250 0 4 472083 171 0 0 21 0 5 0 521889351 917872640 194813 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 224090 194813 181 55 0 221976 0
[pid=16027/tid=16044] ppid=16025 vsize=896360 CPUtime=1182.26 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 282727 1250 0 4 118181 45 0 0 25 0 5 0 521889358 917872640 194813 33554432000 4194304 4417332 140734867966032 18446744073709551615 4366025 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=896360 CPUtime=1182.14 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 259721 1250 0 4 118173 41 0 0 25 0 5 0 521889358 917872640 194813 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364722 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=896360 CPUtime=1175.82 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 239960 1250 0 4 117542 40 0 0 25 0 5 0 521889358 917872640 194813 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=896360 CPUtime=1182.26 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 276973 1250 0 4 118183 43 0 0 25 0 5 0 521889358 917872640 194813 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4722.54
Current children cumulated vsize (KiB) 896360



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 8.20 8.11 7.98 9/186 16095
/proc/meminfo: memFree=21443420/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=946672 CPUtime=4800.04 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 1073465 1250 0 4 479830 174 0 0 21 0 5 0 521889351 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 236668 207391 181 55 0 234554 0
[pid=16027/tid=16044] ppid=16025 vsize=946672 CPUtime=1201.67 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 282727 1250 0 4 120122 45 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=946672 CPUtime=1201.55 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 259721 1250 0 4 120113 42 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=946672 CPUtime=1195.11 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 239960 1250 0 4 119471 40 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=946672 CPUtime=1201.67 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 289551 1250 0 4 120121 46 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 946672

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 8.20 8.11 7.98 9/186 16095
/proc/meminfo: memFree=21443420/32951124 swapFree=67111524/67111528
[pid=16027] ppid=16025 vsize=946672 CPUtime=4800.04 cores=0,2,4,6
/proc/16027/stat : 16027 (plingeling) S 16025 16027 14150 0 -1 4202496 1073465 1250 0 4 479830 174 0 0 21 0 5 0 521889351 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16027/statm: 236668 207391 181 55 0 234554 0
[pid=16027/tid=16044] ppid=16025 vsize=946672 CPUtime=1201.67 cores=0,2,4,6
/proc/16027/task/16044/stat : 16044 (plingeling) R 16025 16027 14150 0 -1 4202560 282727 1250 0 4 120122 45 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16027/tid=16045] ppid=16025 vsize=946672 CPUtime=1201.55 cores=0,2,4,6
/proc/16027/task/16045/stat : 16045 (plingeling) R 16025 16027 14150 0 -1 4202560 259721 1250 0 4 120113 42 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16027/tid=16046] ppid=16025 vsize=946672 CPUtime=1195.11 cores=0,2,4,6
/proc/16027/task/16046/stat : 16046 (plingeling) R 16025 16027 14150 0 -1 4202560 239960 1250 0 4 119471 40 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16027/tid=16047] ppid=16025 vsize=946672 CPUtime=1201.67 cores=0,2,4,6
/proc/16027/task/16047/stat : 16047 (plingeling) R 16025 16027 14150 0 -1 4202560 289551 1250 0 4 120121 46 0 0 25 0 5 0 521889358 969392128 207391 33554432000 4194304 4417332 140734867966032 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 946672

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.75
CPU time (s): 4800.12
CPU user time (s): 4798.33
CPU system time (s): 1.79273
CPU usage (%): 399.427
Max. virtual memory (cumulated for all children) (KiB): 1054532

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4798.33
system time used= 1.79273
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1074716
page faults= 4
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 141
involuntary context switches= 23151

runsolver used 1.78373 second user time and 4.77627 second system time

The end

Launcher Data 

Begin job on node105 at 2011-04-29 20:48:09
IDJOB=3264058
IDBENCH=84551
IDSOLVER=1605
FILE ID=node105/3264058-1304102889
RUNJOBID= node105-1304102889-16012
PBS_JOBID= 13170561
Free space on /tmp= 71940 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT04/handmade/kexu/frb-0.6/frb75-13-2.used-as.sat04-878.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264058-1304102889/watcher-3264058-1304102889 -o /tmp/evaluation-result-3264058-1304102889/solver-3264058-1304102889 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3264058-1304102889.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 00e6f1cbbaaa8f43dc6451d49bfa1cef
RANDOM SEED=1472946558

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23095052 kB
Buffers:       2157548 kB
Cached:        7088872 kB
SwapCached:          4 kB
Active:         544884 kB
Inactive:      8764736 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23095052 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             892 kB
Writeback:           0 kB
AnonPages:       63412 kB
Mapped:          14492 kB
Slab:           483488 kB
PageTables:       4908 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   182444 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71936 MiB
End job on node105 at 2011-04-29 21:08:13