Trace number 3394804

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40001 5011.77

General information on the benchmark

NameSAT04/handmade/marijn/equilarge-l/
equilarge_l4.used-as.sat04-896.cnf
MD5SUMfc993d9544ccd8eaeaa4882599387947
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark821.55
Satisfiable
(Un)Satisfiability was proved
Number of variables4499
Number of clauses18603
Sum of the clauses size54454
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21355
Number of clauses of size 317248
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394804-1306106647/watcher-3394804-1306106647 -o /tmp/evaluation-result-3394804-1306106647/solver-3394804-1306106647 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394804-1306106647.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.44 7.93 7.97 2/543 22889
/proc/meminfo: memFree=16258092/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=13216 CPUtime=0.01 cores=0-7
/proc/22889/stat : 22889 (plingeling) R 22887 22889 22863 0 -1 4202496 1366 1253 0 0 1 0 0 0 25 0 1 0 722265103 13533184 1296 33554432000 4194304 4417452 140736811571088 18446744073709551615 253640859392 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22889/statm: 3304 1296 149 55 0 1190 0

[startup+0.0390391 s]
/proc/loadavg: 7.44 7.93 7.97 2/543 22889
/proc/meminfo: memFree=16258092/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=18480 CPUtime=0.03 cores=0-7
/proc/22889/stat : 22889 (plingeling) R 22887 22889 22863 0 -1 4202496 2641 1253 0 0 3 0 0 0 25 0 1 0 722265103 18923520 2572 33554432000 4194304 4417452 140736811571088 18446744073709551615 253640553453 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/22889/statm: 4620 2604 149 55 0 2506 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 18480

[startup+0.255933 s]
/proc/loadavg: 7.44 7.93 7.97 2/543 22889
/proc/meminfo: memFree=16258092/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=242400 CPUtime=7.33 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 8783 1253 0 0 732 1 0 0 18 0 9 0 722265103 248217600 5546 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 60600 5546 180 55 0 58486 0
Current children cumulated CPU time (s) 7.33
Current children cumulated vsize (KiB) 242400

[startup+1.0678 s]
/proc/loadavg: 7.44 7.93 7.97 2/543 22889
/proc/meminfo: memFree=16258092/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=243648 CPUtime=8.13 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 9095 1253 0 0 812 1 0 0 18 0 9 0 722265103 249495552 5858 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 60912 5858 180 55 0 58798 0
Current children cumulated CPU time (s) 8.13
Current children cumulated vsize (KiB) 243648

[startup+1.16778 s]
/proc/loadavg: 7.44 7.93 7.97 2/543 22889
/proc/meminfo: memFree=16258092/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=243824 CPUtime=8.93 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 9139 1253 0 0 892 1 0 0 18 0 9 0 722265103 249675776 5902 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 60956 5902 180 55 0 58842 0
Current children cumulated CPU time (s) 8.93
Current children cumulated vsize (KiB) 243824

[startup+1.56771 s]
/proc/loadavg: 7.44 7.93 7.97 2/543 22889
/proc/meminfo: memFree=16258092/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=248028 CPUtime=12.13 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 10421 1253 0 0 1212 1 0 0 18 0 9 0 722265103 253980672 6953 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 62007 6953 180 55 0 59893 0
Current children cumulated CPU time (s) 12.13
Current children cumulated vsize (KiB) 248028

[startup+3.10043 s]
/proc/loadavg: 7.48 7.93 7.97 9/552 22904
/proc/meminfo: memFree=16218108/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=261132 CPUtime=24.35 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 17467 1253 0 0 2433 2 0 0 18 0 9 0 722265103 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 65283 10229 180 55 0 63169 0
[pid=22889/tid=22896] ppid=22887 vsize=261132 CPUtime=3.04 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 2389 1253 0 0 304 0 0 0 25 0 9 0 722265107 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=261132 CPUtime=3 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 2080 1253 0 0 300 0 0 0 24 0 9 0 722265107 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4250786 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=261132 CPUtime=3.02 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 511 1253 0 0 302 0 0 0 25 0 9 0 722265107 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236941 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=261132 CPUtime=3.03 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1769 1253 0 0 303 0 0 0 25 0 9 0 722265107 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4250945 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=261132 CPUtime=3.04 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1883 1253 0 0 304 0 0 0 25 0 9 0 722265107 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=261132 CPUtime=3.04 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 2112 1253 0 0 304 0 0 0 25 0 9 0 722265107 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4353615 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=261132 CPUtime=3.04 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1273 1253 0 0 304 0 0 0 25 0 9 0 722265108 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4365103 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=261132 CPUtime=3.04 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1883 1253 0 0 304 0 0 0 25 0 9 0 722265108 267399168 10229 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.35
Current children cumulated vsize (KiB) 261132

[startup+6.32084 s]
/proc/loadavg: 7.48 7.93 7.97 9/552 22904
/proc/meminfo: memFree=16209328/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=269528 CPUtime=50.05 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 30867 1253 0 0 5001 4 0 0 18 0 9 0 722265103 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 67382 12328 180 55 0 65268 0
[pid=22889/tid=22896] ppid=22887 vsize=269528 CPUtime=6.26 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 4548 1253 0 0 626 0 0 0 25 0 9 0 722265107 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=269528 CPUtime=6.16 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 4393 1253 0 0 616 0 0 0 25 0 9 0 722265107 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4353636 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=269528 CPUtime=6.24 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 2199 1253 0 0 624 0 0 0 25 0 9 0 722265107 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4357269 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=269528 CPUtime=6.25 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 3379 1253 0 0 625 0 0 0 25 0 9 0 722265107 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=269528 CPUtime=6.26 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 2825 1253 0 0 626 0 0 0 25 0 9 0 722265107 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4366169 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=269528 CPUtime=6.26 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 4106 1253 0 0 626 0 0 0 25 0 9 0 722265107 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4246384 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=269528 CPUtime=6.26 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 2321 1253 0 0 626 0 0 0 25 0 9 0 722265108 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=269528 CPUtime=6.25 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 3529 1253 0 0 625 0 0 0 25 0 9 0 722265108 275996672 12328 33554432000 4194304 4417452 140736811571088 18446744073709551615 4353727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 50.05
Current children cumulated vsize (KiB) 269528

[startup+12.7007 s]
/proc/loadavg: 7.52 7.93 7.97 9/552 22904
/proc/meminfo: memFree=16199936/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=283596 CPUtime=100.99 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 63222 1253 0 0 10090 9 0 0 18 0 9 0 722265103 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 70899 15851 180 55 0 68785 0
[pid=22889/tid=22896] ppid=22887 vsize=283596 CPUtime=12.63 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 8960 1253 0 0 1263 0 0 0 25 0 9 0 722265107 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=283596 CPUtime=12.44 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 9680 1253 0 0 1243 1 0 0 25 0 9 0 722265107 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4365118 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=283596 CPUtime=12.62 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 6411 1253 0 0 1262 0 0 0 25 0 9 0 722265107 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=283596 CPUtime=12.63 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 6274 1253 0 0 1262 1 0 0 25 0 9 0 722265107 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4250977 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=283596 CPUtime=12.64 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 5625 1253 0 0 1263 1 0 0 25 0 9 0 722265107 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=283596 CPUtime=12.63 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 9345 1253 0 0 1263 0 0 0 25 0 9 0 722265107 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236812 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=283596 CPUtime=12.63 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 5886 1253 0 0 1263 0 0 0 25 0 9 0 722265108 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=283596 CPUtime=12.64 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 7474 1253 0 0 1263 1 0 0 25 0 9 0 722265108 290402304 15851 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 100.99
Current children cumulated vsize (KiB) 283596

[startup+25.5013 s]
/proc/loadavg: 7.63 7.94 7.97 9/552 22904
/proc/meminfo: memFree=16175696/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=309456 CPUtime=203.16 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 118880 1253 0 0 20296 20 0 0 18 0 9 0 722265103 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 77364 22316 180 55 0 75250 0
[pid=22889/tid=22896] ppid=22887 vsize=309456 CPUtime=25.44 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 17179 1253 0 0 2542 2 0 0 25 0 9 0 722265107 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4321395 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=309456 CPUtime=25 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 17804 1253 0 0 2497 3 0 0 25 0 9 0 722265107 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4237326 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=309456 CPUtime=25.42 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 12299 1253 0 0 2540 2 0 0 25 0 9 0 722265107 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=309456 CPUtime=25.43 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 10262 1253 0 0 2541 2 0 0 25 0 9 0 722265107 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236689 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=309456 CPUtime=25.44 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 11662 1253 0 0 2542 2 0 0 25 0 9 0 722265107 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=309456 CPUtime=25.44 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 19171 1253 0 0 2542 2 0 0 25 0 9 0 722265107 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4365004 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=309456 CPUtime=25.44 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 10573 1253 0 0 2543 1 0 0 25 0 9 0 722265108 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4365004 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=309456 CPUtime=25.43 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 16363 1253 0 0 2541 2 0 0 25 0 9 0 722265108 316882944 22316 33554432000 4194304 4417452 140736811571088 18446744073709551615 4251045 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 203.16
Current children cumulated vsize (KiB) 309456

[startup+51.1007 s]
/proc/loadavg: 7.82 7.96 7.97 9/552 22905
/proc/meminfo: memFree=16145724/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=335948 CPUtime=407.49 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 241377 1253 0 0 40706 43 0 0 18 0 9 0 722265103 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 83987 28939 180 55 0 81873 0
[pid=22889/tid=22896] ppid=22887 vsize=335948 CPUtime=51.04 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 34845 1253 0 0 5098 6 0 0 25 0 9 0 722265107 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=335948 CPUtime=50.14 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 30855 1253 0 0 5009 5 0 0 25 0 9 0 722265107 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364716 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=335948 CPUtime=51.02 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 34286 1253 0 0 5097 5 0 0 25 0 9 0 722265107 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4252379 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=335948 CPUtime=51.04 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 18683 1253 0 0 5100 4 0 0 25 0 9 0 722265107 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=335948 CPUtime=51.04 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 26693 1253 0 0 5099 5 0 0 25 0 9 0 722265107 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=335948 CPUtime=51.04 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 39790 1253 0 0 5098 6 0 0 25 0 9 0 722265107 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=335948 CPUtime=51.04 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 20697 1253 0 0 5101 3 0 0 25 0 9 0 722265108 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=335948 CPUtime=51.03 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 31961 1253 0 0 5098 5 0 0 25 0 9 0 722265108 344010752 28939 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 407.49
Current children cumulated vsize (KiB) 335948

[startup+102.301 s]
/proc/loadavg: 7.99 7.98 7.98 9/552 22906

################
# More data... #
################

/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 996343 1253 0 0 472107 129 0 0 25 0 9 0 722265107 1072627712 206830 33554432000 4194304 4417452 140736811571088 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1047488 CPUtime=4722.26 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1143511 1253 0 0 472045 181 0 0 25 0 9 0 722265107 1072627712 206830 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364809 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1047488 CPUtime=4722.37 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1386231 1253 0 0 472016 221 0 0 25 0 9 0 722265107 1072627712 206830 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1047488 CPUtime=4722.38 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1829618 1253 0 0 471977 261 0 0 25 0 9 0 722265108 1072627712 206830 33554432000 4194304 4417452 140736811571088 18446744073709551615 4237237 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1047488 CPUtime=4720.97 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1538880 1253 0 0 471872 225 0 0 25 0 9 0 722265108 1072627712 206830 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37690.8
Current children cumulated vsize (KiB) 1047488

[startup+4782.3 s]
/proc/loadavg: 8.06 8.08 8.06 9/552 23043
/proc/meminfo: memFree=15366284/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=1114280 CPUtime=38169.8 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 10801644 1253 0 0 3815357 1620 0 0 18 0 9 0 722265103 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 278570 223528 180 55 0 276456 0
[pid=22889/tid=22896] ppid=22887 vsize=1114280 CPUtime=4782.35 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 971729 1253 0 0 478082 153 0 0 25 0 9 0 722265107 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=1114280 CPUtime=4694.63 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 1614820 1253 0 0 469236 227 0 0 25 0 9 0 722265107 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236713 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=1114280 CPUtime=4782.34 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 1227086 1253 0 0 478033 201 0 0 25 0 9 0 722265107 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=1114280 CPUtime=4782.36 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1005025 1253 0 0 478106 130 0 0 25 0 9 0 722265107 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1114280 CPUtime=4782.26 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1163109 1253 0 0 478042 184 0 0 25 0 9 0 722265107 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1114280 CPUtime=4782.37 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1394028 1253 0 0 478014 223 0 0 25 0 9 0 722265107 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1114280 CPUtime=4782.38 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1850785 1253 0 0 477972 266 0 0 25 0 9 0 722265108 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364495 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1114280 CPUtime=4780.96 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1571495 1253 0 0 477864 232 0 0 25 0 9 0 722265108 1141022720 223528 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38169.8
Current children cumulated vsize (KiB) 1114280

[startup+4842.3 s]
/proc/loadavg: 8.06 8.08 8.06 9/552 23045
/proc/meminfo: memFree=15447880/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=1031264 CPUtime=38648.7 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 10918749 1253 0 0 3863230 1637 0 0 18 0 9 0 722265103 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 257816 202773 180 55 0 255702 0
[pid=22889/tid=22896] ppid=22887 vsize=1031264 CPUtime=4842.36 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 981330 1253 0 0 484080 156 0 0 25 0 9 0 722265107 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=1031264 CPUtime=4753.53 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 1629312 1253 0 0 475124 229 0 0 25 0 9 0 722265107 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=1031264 CPUtime=4842.35 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 1238217 1253 0 0 484031 204 0 0 25 0 9 0 722265107 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=1031264 CPUtime=4842.36 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1016872 1253 0 0 484104 132 0 0 25 0 9 0 722265107 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1031264 CPUtime=4842.26 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1179993 1253 0 0 484040 186 0 0 25 0 9 0 722265107 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1031264 CPUtime=4842.37 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1408795 1253 0 0 484013 224 0 0 25 0 9 0 722265107 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1031264 CPUtime=4842.37 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1867792 1253 0 0 483970 267 0 0 25 0 9 0 722265108 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1031264 CPUtime=4840.95 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1592871 1253 0 0 483861 234 0 0 25 0 9 0 722265108 1056014336 202773 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38648.7
Current children cumulated vsize (KiB) 1031264

[startup+4902.3 s]
/proc/loadavg: 8.24 8.12 8.07 9/552 23046
/proc/meminfo: memFree=15390348/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=1088536 CPUtime=39127.7 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 11043337 1253 0 0 3911115 1656 0 0 18 0 9 0 722265103 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 272134 217092 180 55 0 270020 0
[pid=22889/tid=22896] ppid=22887 vsize=1088536 CPUtime=4902.38 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 989974 1253 0 0 490081 157 0 0 25 0 9 0 722265107 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=1088536 CPUtime=4812.45 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 1643948 1253 0 0 481015 230 0 0 25 0 9 0 722265107 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=1088536 CPUtime=4902.37 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 1252834 1253 0 0 490031 206 0 0 25 0 9 0 722265107 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=1088536 CPUtime=4902.38 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1027587 1253 0 0 490104 134 0 0 25 0 9 0 722265107 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1088536 CPUtime=4902.27 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1203514 1253 0 0 490037 190 0 0 25 0 9 0 722265107 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1088536 CPUtime=4902.4 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1423718 1253 0 0 490013 227 0 0 25 0 9 0 722265107 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1088536 CPUtime=4902.4 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1884892 1253 0 0 489970 270 0 0 25 0 9 0 722265108 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1088536 CPUtime=4900.95 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1613303 1253 0 0 489857 238 0 0 25 0 9 0 722265108 1114660864 217092 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39127.7
Current children cumulated vsize (KiB) 1088536

[startup+4962.3 s]
/proc/loadavg: 8.09 8.10 8.07 9/552 23048
/proc/meminfo: memFree=15411804/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=1067964 CPUtime=39606.5 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 11155680 1253 0 0 3958974 1674 0 0 18 0 9 0 722265103 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 266991 211949 180 55 0 264877 0
[pid=22889/tid=22896] ppid=22887 vsize=1067964 CPUtime=4962.37 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 995504 1253 0 0 496079 158 0 0 25 0 9 0 722265107 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=1067964 CPUtime=4871.36 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 1658866 1253 0 0 486902 234 0 0 25 0 9 0 722265107 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=1067964 CPUtime=4962.35 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 1266683 1253 0 0 496027 208 0 0 25 0 9 0 722265107 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=1067964 CPUtime=4962.36 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1036460 1253 0 0 496100 136 0 0 25 0 9 0 722265107 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1067964 CPUtime=4962.26 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1223043 1253 0 0 496033 193 0 0 25 0 9 0 722265107 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1067964 CPUtime=4962.38 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1438805 1253 0 0 496008 230 0 0 25 0 9 0 722265107 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4236812 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1067964 CPUtime=4962.38 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1901947 1253 0 0 495966 272 0 0 25 0 9 0 722265108 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1067964 CPUtime=4960.92 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1630805 1253 0 0 495852 240 0 0 25 0 9 0 722265108 1093595136 211949 33554432000 4194304 4417452 140736811571088 18446744073709551615 4237002 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39606.5
Current children cumulated vsize (KiB) 1067964



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.7 s]
/proc/loadavg: 8.04 8.08 8.06 9/552 23049
/proc/meminfo: memFree=15328100/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=1151012 CPUtime=40000.9 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 11253177 1253 0 0 3998403 1688 0 0 18 0 9 0 722265103 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 287753 232711 180 55 0 285639 0
[pid=22889/tid=22896] ppid=22887 vsize=1151012 CPUtime=5011.78 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 999765 1253 0 0 501020 158 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364716 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=1151012 CPUtime=4919.86 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 1666814 1253 0 0 491751 235 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=1151012 CPUtime=5011.77 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 1278631 1253 0 0 500968 209 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=1151012 CPUtime=5011.79 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1045391 1253 0 0 501042 137 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4365962 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1151012 CPUtime=5011.68 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1240594 1253 0 0 500972 196 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1151012 CPUtime=5011.8 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1450426 1253 0 0 500947 233 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1151012 CPUtime=5011.8 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1920421 1253 0 0 500907 273 0 0 25 0 9 0 722265108 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1151012 CPUtime=5010.32 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1647568 1253 0 0 500789 243 0 0 25 0 9 0 722265108 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 1151012

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011.7 s]
/proc/loadavg: 8.04 8.08 8.06 9/552 23049
/proc/meminfo: memFree=15328100/32951124 swapFree=54323904/67111528
[pid=22889] ppid=22887 vsize=1151012 CPUtime=40000.9 cores=0-7
/proc/22889/stat : 22889 (plingeling) S 22887 22889 22863 0 -1 4202496 11253177 1253 0 0 3998403 1688 0 0 18 0 9 0 722265103 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/22889/statm: 287753 232711 180 55 0 285639 0
[pid=22889/tid=22896] ppid=22887 vsize=1151012 CPUtime=5011.78 cores=0-7
/proc/22889/task/22896/stat : 22896 (plingeling) R 22887 22889 22863 0 -1 4202560 999765 1253 0 0 501020 158 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364716 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=22889/tid=22897] ppid=22887 vsize=1151012 CPUtime=4919.86 cores=0-7
/proc/22889/task/22897/stat : 22897 (plingeling) R 22887 22889 22863 0 -1 4202560 1666814 1253 0 0 491751 235 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22889/tid=22898] ppid=22887 vsize=1151012 CPUtime=5011.77 cores=0-7
/proc/22889/task/22898/stat : 22898 (plingeling) R 22887 22889 22863 0 -1 4202560 1278631 1253 0 0 500968 209 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22889/tid=22899] ppid=22887 vsize=1151012 CPUtime=5011.79 cores=0-7
/proc/22889/task/22899/stat : 22899 (plingeling) R 22887 22889 22863 0 -1 4202560 1045391 1253 0 0 501042 137 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4365962 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=22889/tid=22900] ppid=22887 vsize=1151012 CPUtime=5011.68 cores=0-7
/proc/22889/task/22900/stat : 22900 (plingeling) R 22887 22889 22863 0 -1 4202560 1240594 1253 0 0 500972 196 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=22889/tid=22901] ppid=22887 vsize=1151012 CPUtime=5011.8 cores=0-7
/proc/22889/task/22901/stat : 22901 (plingeling) R 22887 22889 22863 0 -1 4202560 1450426 1253 0 0 500947 233 0 0 25 0 9 0 722265107 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=22889/tid=22902] ppid=22887 vsize=1151012 CPUtime=5011.8 cores=0-7
/proc/22889/task/22902/stat : 22902 (plingeling) R 22887 22889 22863 0 -1 4202560 1920421 1253 0 0 500907 273 0 0 25 0 9 0 722265108 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22889/tid=22903] ppid=22887 vsize=1151012 CPUtime=5010.32 cores=0-7
/proc/22889/task/22903/stat : 22903 (plingeling) R 22887 22889 22863 0 -1 4202560 1647568 1253 0 0 500789 243 0 0 25 0 9 0 722265108 1178636288 232711 33554432000 4194304 4417452 140736811571088 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 1151012

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5011.77
CPU time (s): 40001
CPU user time (s): 39984.1
CPU system time (s): 16.9324
CPU usage (%): 798.141
Max. virtual memory (cumulated for all children) (KiB): 1226748

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39984.1
system time used= 16.9324
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 11254431
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 465
involuntary context switches= 140281

runsolver used 25.2802 second user time and 65.871 second system time

The end

Launcher Data 

Begin job on node106 at 2011-05-23 01:24:07
IDJOB=3394804
IDBENCH=84762
IDSOLVER=1851
FILE ID=node106/3394804-1306106647
RUNJOBID= node106-1306106647-22877
PBS_JOBID= 13388028
Free space on /tmp= 73260 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT04/handmade/marijn/equilarge-l/equilarge_l4.used-as.sat04-896.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394804-1306106647/watcher-3394804-1306106647 -o /tmp/evaluation-result-3394804-1306106647/solver-3394804-1306106647 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394804-1306106647.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= fc993d9544ccd8eaeaa4882599387947
RANDOM SEED=859948293

node106.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16258372 kB
Buffers:        343320 kB
Cached:       10389824 kB
SwapCached:      24016 kB
Active:        6115684 kB
Inactive:     10266972 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16258372 kB
SwapTotal:    67111528 kB
SwapFree:     54323904 kB
Dirty:             396 kB
Writeback:           0 kB
AnonPages:     5634908 kB
Mapped:          25560 kB
Slab:           158192 kB
PageTables:      77744 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 167516468 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73260 MiB
End job on node106 at 2011-05-23 02:47:41