Trace number 3372731

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.9 5006.18

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_3_keyfind_1.cnf
MD5SUMff59ad0e88befc51bfa041e45143dc53
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2808
Number of clauses30432
Sum of the clauses size167020
Maximum clause length8
Minimum clause length1
Number of clauses of size 1280
Number of clauses of size 2384
Number of clauses of size 35024
Number of clauses of size 4960
Number of clauses of size 56144
Number of clauses of size over 517640

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372731-1305441709/watcher-3372731-1305441709 -o /tmp/evaluation-result-3372731-1305441709/solver-3372731-1305441709 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372731-1305441709.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.00 8.00 8.00 1/281 12490
/proc/meminfo: memFree=29218672/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=9876 CPUtime=0 cores=0-7
/proc/12490/stat : 12490 (plingeling) R 12488 12490 12278 0 -1 4202496 562 1261 0 0 0 0 0 0 18 0 1 0 655768836 10113024 493 33554432000 4194304 4417452 140735602319424 18446744073709551615 228122332700 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/12490/statm: 2469 493 148 55 0 355 0

[startup+0.06763 s]
/proc/loadavg: 8.00 8.00 8.00 1/281 12490
/proc/meminfo: memFree=29218672/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=23668 CPUtime=0.05 cores=0-7
/proc/12490/stat : 12490 (plingeling) R 12488 12490 12278 0 -1 4202496 4011 1261 0 0 5 0 0 0 18 0 1 0 655768836 24236032 3942 33554432000 4194304 4417452 140735602319424 18446744073709551615 4231554 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/12490/statm: 5917 3942 149 55 0 3803 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 23668

[startup+0.100619 s]
/proc/loadavg: 8.00 8.00 8.00 1/281 12490
/proc/meminfo: memFree=29218672/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=107672 CPUtime=0.12 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 5116 1261 0 0 11 1 0 0 18 0 9 0 655768836 110256128 4480 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 26918 4480 172 55 0 24804 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 107672

[startup+0.316549 s]
/proc/loadavg: 8.00 8.00 8.00 1/281 12490
/proc/meminfo: memFree=29218672/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=115680 CPUtime=1.4 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 7417 1261 0 0 139 1 0 0 18 0 9 0 655768836 118456320 6472 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 28920 6472 180 55 0 26806 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 115680

[startup+0.700477 s]
/proc/loadavg: 8.00 8.00 8.00 1/281 12490
/proc/meminfo: memFree=29218672/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=183576 CPUtime=4.39 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 8388 1261 0 0 438 1 0 0 18 0 9 0 655768836 187981824 7114 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 45894 7114 180 55 0 43780 0
Current children cumulated CPU time (s) 4.39
Current children cumulated vsize (KiB) 183576

[startup+1.50034 s]
/proc/loadavg: 7.52 7.90 7.97 9/290 12504
/proc/meminfo: memFree=29187112/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=191508 CPUtime=10.78 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 14739 1261 0 0 1075 3 0 0 18 0 9 0 655768836 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 47877 9107 180 55 0 45763 0
[pid=12490/tid=12497] ppid=12488 vsize=191508 CPUtime=1.37 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 1529 1261 0 0 137 0 0 0 23 0 9 0 655768845 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365996 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=191508 CPUtime=1.4 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 661 1261 0 0 140 0 0 0 23 0 9 0 655768845 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=191508 CPUtime=1.4 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 2143 1261 0 0 140 0 0 0 23 0 9 0 655768845 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=191508 CPUtime=1.4 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 1205 1261 0 0 140 0 0 0 20 0 9 0 655768845 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4354481 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=191508 CPUtime=1.3 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 875 1261 0 0 130 0 0 0 23 0 9 0 655768845 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4356831 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=191508 CPUtime=1.15 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 449 1261 0 0 115 0 0 0 22 0 9 0 655768846 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=191508 CPUtime=1.22 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 1082 1261 0 0 122 0 0 0 22 0 9 0 655768846 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=191508 CPUtime=1.4 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 1744 1261 0 0 140 0 0 0 23 0 9 0 655768846 196104192 9107 33554432000 4194304 4417452 140735602319424 18446744073709551615 4357970 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 10.78
Current children cumulated vsize (KiB) 191508

[startup+3.10105 s]
/proc/loadavg: 7.52 7.90 7.97 9/290 12504
/proc/meminfo: memFree=29177812/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=201960 CPUtime=23.57 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 30404 1261 0 0 2351 6 0 0 18 0 9 0 655768836 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 50490 11711 180 55 0 48376 0
[pid=12490/tid=12497] ppid=12488 vsize=201960 CPUtime=2.96 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 3596 1261 0 0 296 0 0 0 25 0 9 0 655768845 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4273971 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=201960 CPUtime=3 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 2520 1261 0 0 300 0 0 0 25 0 9 0 655768845 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4321362 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=201960 CPUtime=3 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 5395 1261 0 0 299 1 0 0 25 0 9 0 655768845 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=201960 CPUtime=2.99 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 2770 1261 0 0 299 0 0 0 25 0 9 0 655768845 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=201960 CPUtime=2.9 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 2777 1261 0 0 290 0 0 0 25 0 9 0 655768845 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=201960 CPUtime=2.75 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 1988 1261 0 0 275 0 0 0 25 0 9 0 655768846 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=201960 CPUtime=2.81 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 2325 1261 0 0 281 0 0 0 25 0 9 0 655768846 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=201960 CPUtime=2.99 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 3982 1261 0 0 299 0 0 0 25 0 9 0 655768846 206807040 11711 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.57
Current children cumulated vsize (KiB) 201960

[startup+6.30047 s]
/proc/loadavg: 7.56 7.90 7.97 9/290 12504
/proc/meminfo: memFree=29148396/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=226020 CPUtime=49.13 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 61967 1261 0 0 4900 13 0 0 18 0 9 0 655768836 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 56505 17735 180 55 0 54391 0
[pid=12490/tid=12497] ppid=12488 vsize=226020 CPUtime=6.13 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 8473 1261 0 0 612 1 0 0 25 0 9 0 655768845 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364657 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=226020 CPUtime=6.19 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 7111 1261 0 0 618 1 0 0 25 0 9 0 655768845 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=226020 CPUtime=6.19 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 10161 1261 0 0 618 1 0 0 25 0 9 0 655768845 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=226020 CPUtime=6.2 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 4511 1261 0 0 619 1 0 0 25 0 9 0 655768845 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=226020 CPUtime=6.09 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 5618 1261 0 0 609 0 0 0 25 0 9 0 655768845 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=226020 CPUtime=5.95 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 4476 1261 0 0 594 1 0 0 25 0 9 0 655768846 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=226020 CPUtime=6.02 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 8486 1261 0 0 600 2 0 0 25 0 9 0 655768846 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=226020 CPUtime=6.19 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 8080 1261 0 0 618 1 0 0 25 0 9 0 655768846 231444480 17735 33554432000 4194304 4417452 140735602319424 18446744073709551615 4234260 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.13
Current children cumulated vsize (KiB) 226020

[startup+12.7013 s]
/proc/loadavg: 7.59 7.90 7.97 9/290 12505
/proc/meminfo: memFree=29126600/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=243432 CPUtime=100.28 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 110634 1261 0 0 10003 25 0 0 18 0 9 0 655768836 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 60858 22089 180 55 0 58744 0
[pid=12490/tid=12497] ppid=12488 vsize=243432 CPUtime=12.47 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 16387 1261 0 0 1244 3 0 0 25 0 9 0 655768845 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4354450 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=243432 CPUtime=12.6 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 13404 1261 0 0 1257 3 0 0 25 0 9 0 655768845 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=243432 CPUtime=12.6 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 17587 1261 0 0 1257 3 0 0 25 0 9 0 655768845 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=243432 CPUtime=12.6 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 7952 1261 0 0 1258 2 0 0 25 0 9 0 655768845 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=243432 CPUtime=12.5 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 11964 1261 0 0 1248 2 0 0 25 0 9 0 655768845 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=243432 CPUtime=12.36 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 7075 1261 0 0 1234 2 0 0 25 0 9 0 655768846 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=243432 CPUtime=12.42 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 16331 1261 0 0 1238 4 0 0 25 0 9 0 655768846 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4246315 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=243432 CPUtime=12.59 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 14883 1261 0 0 1257 2 0 0 25 0 9 0 655768846 249274368 22089 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.28
Current children cumulated vsize (KiB) 243432

[startup+25.501 s]
/proc/loadavg: 7.68 7.91 7.97 9/290 12505
/proc/meminfo: memFree=29110648/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=271092 CPUtime=202.56 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 185546 1261 0 0 20216 40 0 0 18 0 9 0 655768836 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 67773 29017 180 55 0 65659 0
[pid=12490/tid=12497] ppid=12488 vsize=271092 CPUtime=25.15 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 32053 1261 0 0 2509 6 0 0 25 0 9 0 655768845 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=271092 CPUtime=25.4 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 23289 1261 0 0 2534 6 0 0 25 0 9 0 655768845 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=271092 CPUtime=25.4 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 28696 1261 0 0 2535 5 0 0 25 0 9 0 655768845 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=271092 CPUtime=25.4 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 13425 1261 0 0 2537 3 0 0 25 0 9 0 655768845 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=271092 CPUtime=25.3 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 16547 1261 0 0 2527 3 0 0 25 0 9 0 655768845 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=271092 CPUtime=25.15 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 14410 1261 0 0 2512 3 0 0 25 0 9 0 655768846 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365106 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=271092 CPUtime=25.21 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 24443 1261 0 0 2516 5 0 0 25 0 9 0 655768846 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4245774 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=271092 CPUtime=25.39 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 27632 1261 0 0 2534 5 0 0 25 0 9 0 655768846 277598208 29017 33554432000 4194304 4417452 140735602319424 18446744073709551615 4357949 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.56
Current children cumulated vsize (KiB) 271092

[startup+51.1004 s]
/proc/loadavg: 7.79 7.91 7.97 9/290 12506
/proc/meminfo: memFree=29078176/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=309844 CPUtime=407.12 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 288396 1261 0 0 40650 62 0 0 18 0 9 0 655768836 317280256 38705 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 77461 38705 180 55 0 75347 0
[pid=12490/tid=12497] ppid=12488 vsize=309844 CPUtime=50.52 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 48444 1261 0 0 5042 10 0 0 25 0 9 0 655768845 317280256 38705 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=309844 CPUtime=51 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 35044 1261 0 0 5091 9 0 0 25 0 9 0 655768845 317280256 38705 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=309844 CPUtime=51 cores=0-7

################
# More data... #
################

/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 262446 1261 0 0 472192 40 0 0 25 0 9 0 655768845 1647656960 363504 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1609040 CPUtime=4722.21 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 297508 1261 0 0 472174 47 0 0 25 0 9 0 655768845 1647656960 363504 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364752 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1609040 CPUtime=4721.29 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 416697 1261 0 0 472044 85 0 0 25 0 9 0 655768846 1647656960 363504 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1609040 CPUtime=4722 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 506145 1261 0 0 472114 86 0 0 25 0 9 0 655768846 1647656960 363504 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1609040 CPUtime=4721.66 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 587103 1261 0 0 472075 91 0 0 25 0 9 0 655768846 1647656960 363504 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37733.7
Current children cumulated vsize (KiB) 1609040

[startup+4782.3 s]
/proc/loadavg: 8.07 8.04 8.00 9/290 12653
/proc/meminfo: memFree=27701620/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=1670468 CPUtime=38213.1 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 3540403 1261 0 0 3820687 618 0 0 18 0 9 0 655768836 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 417617 378861 180 55 0 415503 0
[pid=12490/tid=12497] ppid=12488 vsize=1670468 CPUtime=4738.83 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 584868 1261 0 0 473779 104 0 0 25 0 9 0 655768845 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365047 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=1670468 CPUtime=4782.33 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 409510 1261 0 0 478167 66 0 0 25 0 9 0 655768845 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=1670468 CPUtime=4782.32 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 429438 1261 0 0 478149 83 0 0 25 0 9 0 655768845 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=1670468 CPUtime=4782.32 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 268110 1261 0 0 478190 42 0 0 25 0 9 0 655768845 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4366400 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1670468 CPUtime=4782.2 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 299940 1261 0 0 478172 48 0 0 25 0 9 0 655768845 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1670468 CPUtime=4781.28 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 433081 1261 0 0 478038 90 0 0 25 0 9 0 655768846 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1670468 CPUtime=4781.98 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 515991 1261 0 0 478109 89 0 0 25 0 9 0 655768846 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1670468 CPUtime=4781.64 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 594414 1261 0 0 478072 92 0 0 25 0 9 0 655768846 1710559232 378861 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38213.1
Current children cumulated vsize (KiB) 1670468

[startup+4842.3 s]
/proc/loadavg: 8.07 8.04 8.00 9/290 12654
/proc/meminfo: memFree=27506404/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=1865796 CPUtime=38692.5 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 3589235 1261 0 0 3868620 631 0 0 18 0 9 0 655768836 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 466449 427693 180 55 0 464335 0
[pid=12490/tid=12497] ppid=12488 vsize=1865796 CPUtime=4798.28 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 590804 1261 0 0 479723 105 0 0 25 0 9 0 655768845 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=1865796 CPUtime=4842.33 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 409510 1261 0 0 484167 66 0 0 25 0 9 0 655768845 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=1865796 CPUtime=4842.32 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 429438 1261 0 0 484149 83 0 0 25 0 9 0 655768845 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=1865796 CPUtime=4842.32 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 272334 1261 0 0 484190 42 0 0 25 0 9 0 655768845 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1865796 CPUtime=4842.21 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 304975 1261 0 0 484171 50 0 0 25 0 9 0 655768845 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364984 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1865796 CPUtime=4841.28 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 456513 1261 0 0 484030 98 0 0 25 0 9 0 655768846 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1865796 CPUtime=4841.98 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 519204 1261 0 0 484108 90 0 0 25 0 9 0 655768846 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364893 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1865796 CPUtime=4841.63 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 601406 1261 0 0 484070 93 0 0 25 0 9 0 655768846 1910575104 427693 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38692.5
Current children cumulated vsize (KiB) 1865796

[startup+4902.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/290 12656
/proc/meminfo: memFree=27776076/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=1596416 CPUtime=39172 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 3628009 1261 0 0 3916555 643 0 0 18 0 9 0 655768836 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 399104 360348 180 55 0 396990 0
[pid=12490/tid=12497] ppid=12488 vsize=1596416 CPUtime=4857.74 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 596784 1261 0 0 485667 107 0 0 25 0 9 0 655768845 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=1596416 CPUtime=4902.32 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 414206 1261 0 0 490165 67 0 0 25 0 9 0 655768845 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=1596416 CPUtime=4902.32 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 436862 1261 0 0 490147 85 0 0 25 0 9 0 655768845 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=1596416 CPUtime=4902.32 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 272334 1261 0 0 490190 42 0 0 25 0 9 0 655768845 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1596416 CPUtime=4902.21 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 310373 1261 0 0 490170 51 0 0 25 0 9 0 655768845 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364958 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1596416 CPUtime=4901.29 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 456872 1261 0 0 490030 99 0 0 25 0 9 0 655768846 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1596416 CPUtime=4901.97 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 532382 1261 0 0 490104 93 0 0 25 0 9 0 655768846 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4357708 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1596416 CPUtime=4901.63 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 603145 1261 0 0 490070 93 0 0 25 0 9 0 655768846 1634729984 360348 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39172
Current children cumulated vsize (KiB) 1596416

[startup+4962.3 s]
/proc/loadavg: 8.14 8.07 8.01 10/291 12661
/proc/meminfo: memFree=27858592/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=1496176 CPUtime=39651.1 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 3645838 1261 0 0 3964467 645 0 0 18 0 9 0 655768836 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 374044 335288 180 55 0 371930 0
[pid=12490/tid=12497] ppid=12488 vsize=1496176 CPUtime=4917.19 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 604766 1261 0 0 491611 108 0 0 25 0 9 0 655768845 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=1496176 CPUtime=4962.33 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 417033 1261 0 0 496165 68 0 0 25 0 9 0 655768845 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=1496176 CPUtime=4962.32 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 436862 1261 0 0 496147 85 0 0 25 0 9 0 655768845 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=1496176 CPUtime=4962.32 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 272334 1261 0 0 496190 42 0 0 25 0 9 0 655768845 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1496176 CPUtime=4962.21 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 310628 1261 0 0 496170 51 0 0 25 0 9 0 655768845 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4250882 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1496176 CPUtime=4961 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 460083 1261 0 0 496001 99 0 0 25 0 9 0 655768846 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1496176 CPUtime=4961.98 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 534539 1261 0 0 496104 94 0 0 25 0 9 0 655768846 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1496176 CPUtime=4961.6 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 604542 1261 0 0 496067 93 0 0 25 0 9 0 655768846 1532084224 335288 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365042 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39651.1
Current children cumulated vsize (KiB) 1496176



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.11 s]
/proc/loadavg: 8.20 8.10 8.02 9/290 12662
/proc/meminfo: memFree=27765972/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=1605784 CPUtime=40000.7 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 3673240 1261 0 0 3999424 650 0 0 18 0 9 0 655768836 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 401446 362690 180 55 0 399332 0
[pid=12490/tid=12497] ppid=12488 vsize=1605784 CPUtime=4960.61 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 617205 1261 0 0 495950 111 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4366126 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=1605784 CPUtime=5006.15 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 418738 1261 0 0 500547 68 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=1605784 CPUtime=5006.15 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 436862 1261 0 0 500530 85 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364805 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=1605784 CPUtime=5006.15 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 276430 1261 0 0 500572 43 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1605784 CPUtime=5006.03 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 311327 1261 0 0 500552 51 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365996 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1605784 CPUtime=5004.31 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 462919 1261 0 0 500332 99 0 0 25 0 9 0 655768846 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1605784 CPUtime=5005.8 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 537372 1261 0 0 500486 94 0 0 25 0 9 0 655768846 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1605784 CPUtime=5005.41 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 607336 1261 0 0 500448 93 0 0 25 0 9 0 655768846 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 1605784

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5006.11 s]
/proc/loadavg: 8.20 8.10 8.02 9/290 12662
/proc/meminfo: memFree=27765972/32951124 swapFree=54427468/67111528
[pid=12490] ppid=12488 vsize=1605784 CPUtime=40000.7 cores=0-7
/proc/12490/stat : 12490 (plingeling) S 12488 12490 12278 0 -1 4202496 3673240 1261 0 0 3999424 650 0 0 18 0 9 0 655768836 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/12490/statm: 401446 362690 180 55 0 399332 0
[pid=12490/tid=12497] ppid=12488 vsize=1605784 CPUtime=4960.61 cores=0-7
/proc/12490/task/12497/stat : 12497 (plingeling) R 12488 12490 12278 0 -1 4202560 617205 1261 0 0 495950 111 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4366126 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=12490/tid=12498] ppid=12488 vsize=1605784 CPUtime=5006.15 cores=0-7
/proc/12490/task/12498/stat : 12498 (plingeling) R 12488 12490 12278 0 -1 4202560 418738 1261 0 0 500547 68 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12490/tid=12499] ppid=12488 vsize=1605784 CPUtime=5006.15 cores=0-7
/proc/12490/task/12499/stat : 12499 (plingeling) R 12488 12490 12278 0 -1 4202560 436862 1261 0 0 500530 85 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364805 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12490/tid=12500] ppid=12488 vsize=1605784 CPUtime=5006.15 cores=0-7
/proc/12490/task/12500/stat : 12500 (plingeling) R 12488 12490 12278 0 -1 4202560 276430 1261 0 0 500572 43 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12490/tid=12501] ppid=12488 vsize=1605784 CPUtime=5006.03 cores=0-7
/proc/12490/task/12501/stat : 12501 (plingeling) R 12488 12490 12278 0 -1 4202560 311327 1261 0 0 500552 51 0 0 25 0 9 0 655768845 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4365996 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12490/tid=12502] ppid=12488 vsize=1605784 CPUtime=5004.31 cores=0-7
/proc/12490/task/12502/stat : 12502 (plingeling) R 12488 12490 12278 0 -1 4202560 462919 1261 0 0 500332 99 0 0 25 0 9 0 655768846 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12490/tid=12503] ppid=12488 vsize=1605784 CPUtime=5005.8 cores=0-7
/proc/12490/task/12503/stat : 12503 (plingeling) R 12488 12490 12278 0 -1 4202560 537372 1261 0 0 500486 94 0 0 25 0 9 0 655768846 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12490/tid=12504] ppid=12488 vsize=1605784 CPUtime=5005.41 cores=0-7
/proc/12490/task/12504/stat : 12504 (plingeling) R 12488 12490 12278 0 -1 4202560 607336 1261 0 0 500448 93 0 0 25 0 9 0 655768846 1644322816 362690 33554432000 4194304 4417452 140735602319424 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 1605784

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5006.18
CPU time (s): 40000.9
CPU user time (s): 39994.3
CPU system time (s): 6.562
CPU usage (%): 799.03
Max. virtual memory (cumulated for all children) (KiB): 1912648

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39994.3
system time used= 6.562
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3674501
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 355
involuntary context switches= 191955

runsolver used 12.4891 second user time and 33.043 second system time

The end

Launcher Data 

Begin job on node138 at 2011-05-15 08:41:49
IDJOB=3372731
IDBENCH=83408
IDSOLVER=1851
FILE ID=node138/3372731-1305441709
RUNJOBID= node138-1305436700-12292
PBS_JOBID= 13324755
Free space on /tmp= 73612 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_3_keyfind_1.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372731-1305441709/watcher-3372731-1305441709 -o /tmp/evaluation-result-3372731-1305441709/solver-3372731-1305441709 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372731-1305441709.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ff59ad0e88befc51bfa041e45143dc53
RANDOM SEED=1454368632

node138.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29218828 kB
Buffers:        264072 kB
Cached:        3229656 kB
SwapCached:      18496 kB
Active:         908700 kB
Inactive:      2607540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29218828 kB
SwapTotal:    67111528 kB
SwapFree:     54427468 kB
Dirty:            1204 kB
Writeback:           0 kB
AnonPages:       16128 kB
Mapped:           8800 kB
Slab:           111800 kB
PageTables:      39612 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 39468116 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73612 MiB
End job on node138 at 2011-05-15 10:05:17