Trace number 3371111

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.19 1203.35

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-3-5_528.cnf
MD5SUM7c7070bd15b57ddd9a06dd3709baef54
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark157.607
Satisfiable
(Un)Satisfiability was proved
Number of variables527
Number of clauses12692
Sum of the clauses size38748
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 312356
Number of clauses of size 40
Number of clauses of size 5336
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3371111-1305427692/watcher-3371111-1305427692 -o /tmp/evaluation-result-3371111-1305427692/solver-3371111-1305427692 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3371111-1305427692.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.84 7.99 8.39 2/344 3209
/proc/meminfo: memFree=21555624/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=8728 CPUtime=0 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 214 0 0 0 0 0 0 0 22 0 1 0 635185367 8937472 146 33554432000 4194304 4417452 140734155553552 18446744073709551615 255106768640 0 0 4096 0 18446744071562362840 0 0 17 0 0 0 0
/proc/3209/statm: 2182 146 122 55 0 68 0

[startup+0.0387621 s]
/proc/loadavg: 6.84 7.99 8.39 2/344 3209
/proc/meminfo: memFree=21555624/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=55888 CPUtime=0.07 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1863 1256 0 0 7 0 0 0 22 0 5 0 635185367 57229312 1725 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 13972 1725 177 55 0 11858 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 55888

[startup+0.100721 s]
/proc/loadavg: 6.84 7.99 8.39 2/344 3209
/proc/meminfo: memFree=21555624/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=55480 CPUtime=0.32 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 2083 1256 0 0 32 0 0 0 22 0 5 0 635185367 56811520 1655 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 13870 1655 178 55 0 11756 0
Current children cumulated CPU time (s) 0.32
Current children cumulated vsize (KiB) 55480

[startup+0.300687 s]
/proc/loadavg: 6.84 7.99 8.39 2/344 3209
/proc/meminfo: memFree=21555624/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=122752 CPUtime=1.12 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 2518 1256 0 0 112 0 0 0 22 0 5 0 635185367 125698048 2090 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 30688 2090 178 55 0 28574 0
Current children cumulated CPU time (s) 1.12
Current children cumulated vsize (KiB) 122752

[startup+0.700617 s]
/proc/loadavg: 6.84 7.99 8.39 2/344 3209
/proc/meminfo: memFree=21555624/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=124708 CPUtime=2.72 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 3039 1256 0 0 272 0 0 0 22 0 5 0 635185367 127700992 2611 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 31177 2611 178 55 0 29063 0
Current children cumulated CPU time (s) 2.72
Current children cumulated vsize (KiB) 124708

[startup+1.50048 s]
/proc/loadavg: 6.84 7.99 8.39 9/358 3239
/proc/meminfo: memFree=21529216/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=127492 CPUtime=5.9 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 4224 1256 0 0 590 0 0 0 22 0 5 0 635185367 130551808 3307 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 31873 3307 178 55 0 29759 0
[pid=3209/tid=3232] ppid=3207 vsize=127492 CPUtime=1.47 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 357 1256 0 0 147 0 0 0 25 0 5 0 635185369 130551808 3307 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=127492 CPUtime=1.47 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 466 1256 0 0 147 0 0 0 25 0 5 0 635185369 130551808 3307 33554432000 4194304 4417452 140734155553552 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=127492 CPUtime=1.47 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 370 1256 0 0 147 0 0 0 25 0 5 0 635185369 130551808 3307 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=127492 CPUtime=1.46 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 1628 1256 0 0 146 0 0 0 25 0 5 0 635185369 130551808 3307 33554432000 4194304 4417452 140734155553552 18446744073709551615 4365973 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 5.9
Current children cumulated vsize (KiB) 127492

[startup+3.1002 s]
/proc/loadavg: 6.93 7.99 8.39 9/358 3239
/proc/meminfo: memFree=21521652/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=132008 CPUtime=12.29 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 7873 1256 0 0 1228 1 0 0 22 0 5 0 635185367 135176192 4451 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 33002 4451 178 55 0 30888 0
[pid=3209/tid=3232] ppid=3207 vsize=132008 CPUtime=3.07 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 1389 1256 0 0 307 0 0 0 25 0 5 0 635185369 135176192 4451 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=132008 CPUtime=3.07 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 1721 1256 0 0 307 0 0 0 25 0 5 0 635185369 135176192 4451 33554432000 4194304 4417452 140734155553552 18446744073709551615 4365150 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=132008 CPUtime=3.07 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 882 1256 0 0 307 0 0 0 25 0 5 0 635185369 135176192 4451 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=132008 CPUtime=3.04 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 2478 1256 0 0 304 0 0 0 25 0 5 0 635185369 135176192 4451 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12.29
Current children cumulated vsize (KiB) 132008

[startup+6.30063 s]
/proc/loadavg: 6.93 7.99 8.39 9/358 3239
/proc/meminfo: memFree=21509596/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=134372 CPUtime=25.06 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 14857 1256 0 0 2504 2 0 0 22 0 5 0 635185367 137596928 5042 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 33593 5042 178 55 0 31479 0
[pid=3209/tid=3232] ppid=3207 vsize=134372 CPUtime=6.27 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 2371 1256 0 0 627 0 0 0 25 0 5 0 635185369 137596928 5042 33554432000 4194304 4417452 140734155553552 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=134372 CPUtime=6.26 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 4160 1256 0 0 626 0 0 0 25 0 5 0 635185369 137596928 5042 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364693 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=134372 CPUtime=6.27 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 1839 1256 0 0 627 0 0 0 25 0 5 0 635185369 137596928 5042 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=134372 CPUtime=6.21 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 5084 1256 0 0 620 1 0 0 25 0 5 0 635185369 137596928 5042 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 25.06
Current children cumulated vsize (KiB) 134372

[startup+12.7005 s]
/proc/loadavg: 7.09 7.99 8.38 9/358 3240
/proc/meminfo: memFree=21495932/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=144220 CPUtime=50.58 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 32352 1256 0 0 5054 4 0 0 22 0 5 0 635185367 147681280 7504 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 36055 7504 178 55 0 33941 0
[pid=3209/tid=3232] ppid=3207 vsize=144220 CPUtime=12.67 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 6616 1256 0 0 1266 1 0 0 25 0 5 0 635185369 147681280 7504 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=144220 CPUtime=12.67 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 7992 1256 0 0 1266 1 0 0 25 0 5 0 635185369 147681280 7504 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364597 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=144220 CPUtime=12.67 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 6685 1256 0 0 1267 0 0 0 25 0 5 0 635185369 147681280 7504 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=144220 CPUtime=12.54 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 9656 1256 0 0 1253 1 0 0 25 0 5 0 635185369 147681280 7504 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.58
Current children cumulated vsize (KiB) 144220

[startup+25.5003 s]
/proc/loadavg: 7.38 8.03 8.39 9/358 3240
/proc/meminfo: memFree=21480068/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=150308 CPUtime=101.64 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 60161 1256 0 0 10155 9 0 0 22 0 5 0 635185367 153915392 9026 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 37577 9026 178 55 0 35463 0
[pid=3209/tid=3232] ppid=3207 vsize=150308 CPUtime=25.47 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 11935 1256 0 0 2546 1 0 0 25 0 5 0 635185369 153915392 9026 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=150308 CPUtime=25.46 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 16630 1256 0 0 2544 2 0 0 25 0 5 0 635185369 153915392 9026 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=150308 CPUtime=25.47 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 14383 1256 0 0 2546 1 0 0 25 0 5 0 635185369 153915392 9026 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=150308 CPUtime=25.19 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 15810 1256 0 0 2517 2 0 0 25 0 5 0 635185369 153915392 9026 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.64
Current children cumulated vsize (KiB) 150308

[startup+51.1008 s]
/proc/loadavg: 6.97 7.89 8.34 5/348 3258
/proc/meminfo: memFree=21507876/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=159152 CPUtime=203.76 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 107921 1256 0 0 20359 17 0 0 22 0 5 0 635185367 162971648 11246 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 39788 11246 178 55 0 37674 0
[pid=3209/tid=3232] ppid=3207 vsize=159152 CPUtime=51.07 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 23178 1256 0 0 5104 3 0 0 25 0 5 0 635185369 162971648 11246 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=159152 CPUtime=51.06 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 29574 1256 0 0 5101 5 0 0 25 0 5 0 635185369 162971648 11246 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=159152 CPUtime=51.07 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 24835 1256 0 0 5104 3 0 0 25 0 5 0 635185369 162971648 11246 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=159152 CPUtime=50.52 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 28931 1256 0 0 5047 5 0 0 25 0 5 0 635185369 162971648 11246 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.76
Current children cumulated vsize (KiB) 159152

[startup+102.313 s]
/proc/loadavg: 5.23 7.25 8.09 5/348 3259
/proc/meminfo: memFree=21489532/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=179972 CPUtime=408.04 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 237210 1256 0 0 40766 38 0 0 22 0 5 0 635185367 184291328 16452 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 44993 16452 178 55 0 42879 0
[pid=3209/tid=3232] ppid=3207 vsize=179972 CPUtime=102.28 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 61752 1256 0 0 10219 9 0 0 25 0 5 0 635185369 184291328 16452 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364699 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=179972 CPUtime=102.27 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 68687 1256 0 0 10216 11 0 0 25 0 5 0 635185369 184291328 16452 33554432000 4194304 4417452 140734155553552 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=179972 CPUtime=102.28 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 59738 1256 0 0 10219 9 0 0 25 0 5 0 635185369 184291328 16452 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=179972 CPUtime=101.17 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 45630 1256 0 0 10109 8 0 0 25 0 5 0 635185369 184291328 16452 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 408.04
Current children cumulated vsize (KiB) 179972

[startup+162.3 s]
/proc/loadavg: 4.50 6.67 7.84 5/348 3261
/proc/meminfo: memFree=21488592/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=180156 CPUtime=647.34 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 331230 1256 0 0 64681 53 0 0 22 0 5 0 635185367 184479744 16500 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 45039 16500 178 55 0 42925 0
[pid=3209/tid=3232] ppid=3207 vsize=180156 CPUtime=162.27 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 84870 1256 0 0 16214 13 0 0 25 0 5 0 635185369 184479744 16500 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=180156 CPUtime=162.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 98814 1256 0 0 16209 16 0 0 25 0 5 0 635185369 184479744 16500 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=180156 CPUtime=162.27 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 83236 1256 0 0 16215 12 0 0 25 0 5 0 635185369 184479744 16500 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=180156 CPUtime=160.51 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 62907 1256 0 0 16040 11 0 0 25 0 5 0 635185369 184479744 16500 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 647.34
Current children cumulated vsize (KiB) 180156


################
# More data... #
################

[pid=3209/tid=3233] ppid=3207 vsize=344384 CPUtime=762.26 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 322849 1256 0 0 76178 48 0 0 25 0 5 0 635185369 352649216 57530 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=344384 CPUtime=762.29 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 294691 1256 0 0 76186 43 0 0 25 0 5 0 635185369 352649216 57530 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=344384 CPUtime=753.98 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 166285 1256 0 0 75373 25 0 0 25 0 5 0 635185369 352649216 57530 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3040.87
Current children cumulated vsize (KiB) 344384

[startup+822.3 s]
/proc/loadavg: 4.08 4.36 5.91 5/348 3288
/proc/meminfo: memFree=21323592/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=345660 CPUtime=3280.19 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1100364 1256 0 0 327853 166 0 0 22 0 5 0 635185367 353955840 57849 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 86415 57849 178 55 0 84301 0
[pid=3209/tid=3232] ppid=3207 vsize=345660 CPUtime=822.29 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 291285 1256 0 0 82186 43 0 0 25 0 5 0 635185369 353955840 57849 33554432000 4194304 4417452 140734155553552 18446744073709551615 4366400 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=345660 CPUtime=822.24 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 329382 1256 0 0 82175 49 0 0 25 0 5 0 635185369 353955840 57849 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=345660 CPUtime=822.28 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 306162 1256 0 0 82183 45 0 0 25 0 5 0 635185369 353955840 57849 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=345660 CPUtime=813.32 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 172132 1256 0 0 81306 26 0 0 25 0 5 0 635185369 353955840 57849 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3280.19
Current children cumulated vsize (KiB) 345660

[startup+882.301 s]
/proc/loadavg: 4.08 4.30 5.79 5/348 3291
/proc/meminfo: memFree=21333696/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=335316 CPUtime=3519.53 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1144990 1256 0 0 351781 172 0 0 22 0 5 0 635185367 343363584 55263 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 83829 55263 178 55 0 81715 0
[pid=3209/tid=3232] ppid=3207 vsize=335316 CPUtime=882.29 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 293850 1256 0 0 88185 44 0 0 25 0 5 0 635185369 343363584 55263 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=335316 CPUtime=882.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 343016 1256 0 0 88174 51 0 0 25 0 5 0 635185369 343363584 55263 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=335316 CPUtime=882.29 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 323796 1256 0 0 88181 48 0 0 25 0 5 0 635185369 343363584 55263 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=335316 CPUtime=872.67 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 182925 1256 0 0 87239 28 0 0 25 0 5 0 635185369 343363584 55263 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3519.53
Current children cumulated vsize (KiB) 335316

[startup+942.301 s]
/proc/loadavg: 4.07 4.26 5.68 5/348 3292
/proc/meminfo: memFree=21308584/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=360900 CPUtime=3758.89 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1200184 1256 0 0 375709 180 0 0 22 0 5 0 635185367 369561600 61659 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 90225 61659 178 55 0 88111 0
[pid=3209/tid=3232] ppid=3207 vsize=360900 CPUtime=942.29 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 311548 1256 0 0 94183 46 0 0 25 0 5 0 635185369 369561600 61659 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=360900 CPUtime=942.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 353900 1256 0 0 94172 53 0 0 25 0 5 0 635185369 369561600 61659 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=360900 CPUtime=942.29 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 344276 1256 0 0 94178 51 0 0 25 0 5 0 635185369 369561600 61659 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=360900 CPUtime=932.02 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 189057 1256 0 0 93173 29 0 0 25 0 5 0 635185369 369561600 61659 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3758.89
Current children cumulated vsize (KiB) 360900

[startup+1002.3 s]
/proc/loadavg: 4.07 4.23 5.58 5/348 3294
/proc/meminfo: memFree=21279132/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=390612 CPUtime=3998.23 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1223408 1256 0 0 399640 183 0 0 22 0 5 0 635185367 399986688 69087 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 97653 69087 178 55 0 95539 0
[pid=3209/tid=3232] ppid=3207 vsize=390612 CPUtime=1002.29 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 318271 1256 0 0 100182 47 0 0 25 0 5 0 635185369 399986688 69087 33554432000 4194304 4417452 140734155553552 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=390612 CPUtime=1002.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 357113 1256 0 0 100172 53 0 0 25 0 5 0 635185369 399986688 69087 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=390612 CPUtime=1002.29 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 351508 1256 0 0 100177 52 0 0 25 0 5 0 635185369 399986688 69087 33554432000 4194304 4417452 140734155553552 18446744073709551615 4237317 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=390612 CPUtime=991.35 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 195113 1256 0 0 99106 29 0 0 25 0 5 0 635185369 399986688 69087 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3998.23
Current children cumulated vsize (KiB) 390612

[startup+1062.3 s]
/proc/loadavg: 4.06 4.20 5.48 5/348 3296
/proc/meminfo: memFree=21256244/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=413060 CPUtime=4237.59 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1241098 1256 0 0 423573 186 0 0 22 0 5 0 635185367 422973440 74699 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 103265 74699 178 55 0 101151 0
[pid=3209/tid=3232] ppid=3207 vsize=413060 CPUtime=1062.29 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 318271 1256 0 0 106182 47 0 0 25 0 5 0 635185369 422973440 74699 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=413060 CPUtime=1062.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 357113 1256 0 0 106172 53 0 0 25 0 5 0 635185369 422973440 74699 33554432000 4194304 4417452 140734155553552 18446744073709551615 4353879 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=413060 CPUtime=1062.3 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 359700 1256 0 0 106176 54 0 0 25 0 5 0 635185369 422973440 74699 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=413060 CPUtime=1050.71 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 204611 1256 0 0 105041 30 0 0 25 0 5 0 635185369 422973440 74699 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4237.59
Current children cumulated vsize (KiB) 413060

[startup+1122.3 s]
/proc/loadavg: 4.06 4.17 5.39 5/348 3297
/proc/meminfo: memFree=21225188/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=444172 CPUtime=4476.94 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1284955 1256 0 0 447499 195 0 0 22 0 5 0 635185367 454832128 82477 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 111043 82477 178 55 0 108929 0
[pid=3209/tid=3232] ppid=3207 vsize=444172 CPUtime=1122.3 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 329491 1256 0 0 112180 50 0 0 25 0 5 0 635185369 454832128 82477 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=444172 CPUtime=1122.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 365305 1256 0 0 112170 55 0 0 25 0 5 0 635185369 454832128 82477 33554432000 4194304 4417452 140734155553552 18446744073709551615 4366074 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=444172 CPUtime=1122.29 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 379248 1256 0 0 112172 57 0 0 25 0 5 0 635185369 454832128 82477 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=444172 CPUtime=1110.05 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 209508 1256 0 0 110974 31 0 0 25 0 5 0 635185369 454832128 82477 33554432000 4194304 4417452 140734155553552 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4476.94
Current children cumulated vsize (KiB) 444172

[startup+1182.3 s]
/proc/loadavg: 4.02 4.14 5.30 5/348 3299
/proc/meminfo: memFree=21268280/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=399544 CPUtime=4716.29 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1320579 1256 0 0 471430 199 0 0 22 0 5 0 635185367 409133056 71320 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 99886 71320 178 55 0 97772 0
[pid=3209/tid=3232] ppid=3207 vsize=399544 CPUtime=1182.3 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 337683 1256 0 0 118179 51 0 0 25 0 5 0 635185369 409133056 71320 33554432000 4194304 4417452 140734155553552 18446744073709551615 4263180 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=399544 CPUtime=1182.25 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 371368 1256 0 0 118169 56 0 0 25 0 5 0 635185369 409133056 71320 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=399544 CPUtime=1182.29 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 395632 1256 0 0 118170 59 0 0 25 0 5 0 635185369 409133056 71320 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=399544 CPUtime=1169.41 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 214493 1256 0 0 116909 32 0 0 25 0 5 0 635185369 409133056 71320 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4716.29
Current children cumulated vsize (KiB) 399544



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.3 s]
/proc/loadavg: 4.01 4.13 5.27 6/348 3299
/proc/meminfo: memFree=21197572/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=471304 CPUtime=4800.11 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1338519 1256 0 0 479810 201 0 0 22 0 5 0 635185367 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 117826 89260 178 55 0 115712 0
[pid=3209/tid=3232] ppid=3207 vsize=471304 CPUtime=1203.32 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 345875 1256 0 0 120280 52 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=471304 CPUtime=1203.28 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 371368 1256 0 0 120272 56 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=471304 CPUtime=1203.32 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 403824 1256 0 0 120272 60 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364567 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=471304 CPUtime=1190.18 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 216049 1256 0 0 118986 32 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 471304

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.3 s]
/proc/loadavg: 4.01 4.13 5.27 6/348 3299
/proc/meminfo: memFree=21197572/32951124 swapFree=67111348/67111528
[pid=3209] ppid=3207 vsize=471304 CPUtime=4800.11 cores=0,2,4,6
/proc/3209/stat : 3209 (plingeling) S 3207 3209 27864 0 -1 4202496 1338519 1256 0 0 479810 201 0 0 22 0 5 0 635185367 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/3209/statm: 117826 89260 178 55 0 115712 0
[pid=3209/tid=3232] ppid=3207 vsize=471304 CPUtime=1203.32 cores=0,2,4,6
/proc/3209/task/3232/stat : 3232 (plingeling) R 3207 3209 27864 0 -1 4202560 345875 1256 0 0 120280 52 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3209/tid=3233] ppid=3207 vsize=471304 CPUtime=1203.28 cores=0,2,4,6
/proc/3209/task/3233/stat : 3233 (plingeling) R 3207 3209 27864 0 -1 4202560 371368 1256 0 0 120272 56 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3209/tid=3234] ppid=3207 vsize=471304 CPUtime=1203.32 cores=0,2,4,6
/proc/3209/task/3234/stat : 3234 (plingeling) R 3207 3209 27864 0 -1 4202560 403824 1256 0 0 120272 60 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364567 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3209/tid=3235] ppid=3207 vsize=471304 CPUtime=1190.18 cores=0,2,4,6
/proc/3209/task/3235/stat : 3235 (plingeling) R 3207 3209 27864 0 -1 4202560 216049 1256 0 0 118986 32 0 0 25 0 5 0 635185369 482615296 89260 33554432000 4194304 4417452 140734155553552 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 471304

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.35
CPU time (s): 4800.19
CPU user time (s): 4798.15
CPU system time (s): 2.03969
CPU usage (%): 398.903
Max. virtual memory (cumulated for all children) (KiB): 523180

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4798.15
system time used= 2.03969
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1339775
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 194
involuntary context switches= 21049

runsolver used 4.18936 second user time and 8.96464 second system time

The end

Launcher Data 

Begin job on node144 at 2011-05-15 04:48:12
IDJOB=3371111
IDBENCH=83256
IDSOLVER=1848
FILE ID=node144/3371111-1305427692
RUNJOBID= node144-1305427692-3195
PBS_JOBID= 13324583
Free space on /tmp= 73256 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-3-5_528.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3371111-1305427692/watcher-3371111-1305427692 -o /tmp/evaluation-result-3371111-1305427692/solver-3371111-1305427692 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3371111-1305427692.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7c7070bd15b57ddd9a06dd3709baef54
RANDOM SEED=431906235

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21554672 kB
Buffers:        293788 kB
Cached:        3628072 kB
SwapCached:          0 kB
Active:        8229360 kB
Inactive:      2821648 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21554672 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:             872 kB
Writeback:           0 kB
AnonPages:     7129176 kB
Mapped:          26800 kB
Slab:           244660 kB
PageTables:      37312 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 81975024 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73256 MiB
End job on node144 at 2011-05-15 05:08:18