Trace number 3372128

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.9 5007.58

General information on the benchmark

Nameindustrial/palacios/hard/
sortnet-7-ipc5-h15-unsat.cnf
MD5SUM0683489092a5eb0c881dd17c2c45d08e
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables105623
Number of clauses374254
Sum of the clauses size911890
Maximum clause length43
Minimum clause length1
Number of clauses of size 1128
Number of clauses of size 2269133
Number of clauses of size 399944
Number of clauses of size 4407
Number of clauses of size 5309
Number of clauses of size over 54333

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372128-1305435500/watcher-3372128-1305435500 -o /tmp/evaluation-result-3372128-1305435500/solver-3372128-1305435500 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372128-1305435500.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.00 7.88 7.89 1/300 30098
/proc/meminfo: memFree=17989504/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=11760 CPUtime=0 cores=0-7
/proc/30098/stat : 30098 (plingeling) R 30096 30098 30072 0 -1 4202496 994 1252 0 0 0 0 0 0 18 0 1 0 655149672 12042240 924 33554432000 4194304 4417452 140737458956128 18446744073709551615 274473664697 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/30098/statm: 2940 932 148 55 0 826 0

[startup+0.0497379 s]
/proc/loadavg: 7.00 7.88 7.89 1/300 30098
/proc/meminfo: memFree=17989504/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=19584 CPUtime=0.04 cores=0-7
/proc/30098/stat : 30098 (plingeling) R 30096 30098 30072 0 -1 4202496 2990 1252 0 0 4 0 0 0 18 0 1 0 655149672 20054016 2920 33554432000 4194304 4417452 140737458956128 18446744073709551615 4264911 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/30098/statm: 4896 2920 149 55 0 2782 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 19584

[startup+0.100728 s]
/proc/loadavg: 7.00 7.88 7.89 1/300 30098
/proc/meminfo: memFree=17989504/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=28444 CPUtime=0.09 cores=0-7
/proc/30098/stat : 30098 (plingeling) R 30096 30098 30072 0 -1 4202496 5205 1252 0 0 9 0 0 0 18 0 1 0 655149672 29126656 5135 33554432000 4194304 4417452 140737458956128 18446744073709551615 4260083 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/30098/statm: 7111 5135 149 55 0 4997 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28444

[startup+0.300756 s]
/proc/loadavg: 7.00 7.88 7.89 1/300 30098
/proc/meminfo: memFree=17989504/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=67016 CPUtime=0.29 cores=0-7
/proc/30098/stat : 30098 (plingeling) R 30096 30098 30072 0 -1 4202496 14892 1252 0 0 28 1 0 0 19 0 1 0 655149672 68624384 14778 33554432000 4194304 4417452 140737458956128 18446744073709551615 274473967360 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/30098/statm: 16754 14778 149 55 0 14640 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 67016

[startup+0.700622 s]
/proc/loadavg: 7.00 7.88 7.89 1/300 30098
/proc/meminfo: memFree=17989504/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=155860 CPUtime=0.69 cores=0-7
/proc/30098/stat : 30098 (plingeling) R 30096 30098 30072 0 -1 4202496 37103 1252 0 0 65 4 0 0 19 0 1 0 655149672 159600640 36989 33554432000 4194304 4417452 140737458956128 18446744073709551615 4264940 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/30098/statm: 38965 36989 149 55 0 36851 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 155860

[startup+1.50049 s]
/proc/loadavg: 7.00 7.88 7.89 9/309 30112
/proc/meminfo: memFree=17842512/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=302824 CPUtime=5.08 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 64604 1252 0 0 499 9 0 0 18 0 9 0 655149672 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 75706 44724 174 55 0 73592 0
[pid=30098/tid=30105] ppid=30096 vsize=302824 CPUtime=0.57 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 2891 1252 0 0 57 0 0 0 23 0 9 0 655149763 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4286414 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=302824 CPUtime=0.58 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 2800 1252 0 0 58 0 0 0 18 0 9 0 655149763 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4284514 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=302824 CPUtime=0.58 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 5733 1252 0 0 57 1 0 0 18 0 9 0 655149763 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4284784 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=302824 CPUtime=0.58 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 2371 1252 0 0 58 0 0 0 20 0 9 0 655149763 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4365092 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=302824 CPUtime=0.1 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 2268 1252 0 0 10 0 0 0 19 0 9 0 655149763 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4286255 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=302824 CPUtime=0.57 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 2782 1252 0 0 57 0 0 0 18 0 9 0 655149764 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4284788 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=302824 CPUtime=0.56 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 5755 1252 0 0 55 1 0 0 20 0 9 0 655149764 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4284741 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=302824 CPUtime=0.57 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 2885 1252 0 0 57 0 0 0 20 0 9 0 655149764 310091776 44724 33554432000 4194304 4417452 140737458956128 18446744073709551615 4284514 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5.08
Current children cumulated vsize (KiB) 302824

[startup+3.10121 s]
/proc/loadavg: 7.00 7.88 7.89 9/309 30112
/proc/meminfo: memFree=17812008/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=377680 CPUtime=17.88 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 90330 1252 0 0 1774 14 0 0 18 0 9 0 655149672 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 94420 59789 176 55 0 92306 0
[pid=30098/tid=30105] ppid=30096 vsize=377680 CPUtime=2.18 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 6721 1252 0 0 217 1 0 0 25 0 9 0 655149763 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4277495 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=377680 CPUtime=2.18 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 6013 1252 0 0 217 1 0 0 21 0 9 0 655149763 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4234289 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=377680 CPUtime=2.18 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 9726 1252 0 0 216 2 0 0 21 0 9 0 655149763 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 274473661492 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=377680 CPUtime=2.18 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 2371 1252 0 0 218 0 0 0 25 0 9 0 655149763 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4357260 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=377680 CPUtime=1.7 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 7087 1252 0 0 169 1 0 0 25 0 9 0 655149763 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4275762 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=377680 CPUtime=2.18 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 4830 1252 0 0 217 1 0 0 21 0 9 0 655149764 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4232479 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=377680 CPUtime=2.15 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 10365 1252 0 0 213 2 0 0 25 0 9 0 655149764 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4274888 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=377680 CPUtime=2.17 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 6098 1252 0 0 217 0 0 0 25 0 9 0 655149764 386744320 59789 33554432000 4194304 4417452 140737458956128 18446744073709551615 4275748 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 17.88
Current children cumulated vsize (KiB) 377680

[startup+6.30064 s]
/proc/loadavg: 7.08 7.88 7.89 9/309 30112
/proc/meminfo: memFree=17728100/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=397228 CPUtime=43.44 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 116872 1252 0 0 4326 18 0 0 18 0 9 0 655149672 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 99307 65956 180 55 0 97193 0
[pid=30098/tid=30105] ppid=30096 vsize=397228 CPUtime=5.37 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 10552 1252 0 0 536 1 0 0 25 0 9 0 655149763 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4252053 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=397228 CPUtime=5.38 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 14528 1252 0 0 536 2 0 0 25 0 9 0 655149763 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4240208 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=397228 CPUtime=5.38 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 13423 1252 0 0 536 2 0 0 25 0 9 0 655149763 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=397228 CPUtime=5.38 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 2371 1252 0 0 538 0 0 0 25 0 9 0 655149763 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=397228 CPUtime=4.9 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 10728 1252 0 0 488 2 0 0 25 0 9 0 655149763 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4253320 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=397228 CPUtime=5.38 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 5374 1252 0 0 537 1 0 0 25 0 9 0 655149764 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4253152 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=397228 CPUtime=5.32 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 14393 1252 0 0 530 2 0 0 25 0 9 0 655149764 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=397228 CPUtime=5.37 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 8384 1252 0 0 536 1 0 0 25 0 9 0 655149764 406761472 65956 33554432000 4194304 4417452 140737458956128 18446744073709551615 4239777 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 43.44
Current children cumulated vsize (KiB) 397228

[startup+12.7005 s]
/proc/loadavg: 7.15 7.89 7.89 9/309 30112
/proc/meminfo: memFree=17621360/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=496960 CPUtime=94.56 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 142832 1252 0 0 9432 24 0 0 18 0 9 0 655149672 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 124240 91913 180 55 0 122126 0
[pid=30098/tid=30105] ppid=30096 vsize=496960 CPUtime=11.77 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 15570 1252 0 0 1175 2 0 0 25 0 9 0 655149763 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4258241 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=496960 CPUtime=11.78 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 19546 1252 0 0 1175 3 0 0 25 0 9 0 655149763 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=496960 CPUtime=11.78 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 13423 1252 0 0 1176 2 0 0 25 0 9 0 655149763 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4237232 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=496960 CPUtime=11.78 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 2371 1252 0 0 1178 0 0 0 25 0 9 0 655149763 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4365936 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=496960 CPUtime=11.3 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 16923 1252 0 0 1127 3 0 0 25 0 9 0 655149763 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=496960 CPUtime=11.77 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 10085 1252 0 0 1175 2 0 0 25 0 9 0 655149764 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236224 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=496960 CPUtime=11.65 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 14393 1252 0 0 1163 2 0 0 25 0 9 0 655149764 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364589 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=496960 CPUtime=11.76 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 13402 1252 0 0 1175 1 0 0 25 0 9 0 655149764 508887040 91913 33554432000 4194304 4417452 140737458956128 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 94.56
Current children cumulated vsize (KiB) 496960

[startup+25.5013 s]
/proc/loadavg: 7.41 7.91 7.89 9/309 30113
/proc/meminfo: memFree=17712116/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=405636 CPUtime=196.85 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 142920 1252 0 0 19660 25 0 0 18 0 9 0 655149672 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 101409 69082 180 55 0 99295 0
[pid=30098/tid=30105] ppid=30096 vsize=405636 CPUtime=24.57 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 15570 1252 0 0 2455 2 0 0 25 0 9 0 655149763 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=405636 CPUtime=24.58 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 19546 1252 0 0 2455 3 0 0 25 0 9 0 655149763 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364583 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=405636 CPUtime=24.58 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 13423 1252 0 0 2456 2 0 0 25 0 9 0 655149763 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=405636 CPUtime=24.58 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 2371 1252 0 0 2458 0 0 0 25 0 9 0 655149763 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=405636 CPUtime=24.1 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 16923 1252 0 0 2407 3 0 0 25 0 9 0 655149763 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=405636 CPUtime=24.57 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 10085 1252 0 0 2455 2 0 0 25 0 9 0 655149764 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=405636 CPUtime=24.32 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 14481 1252 0 0 2430 2 0 0 25 0 9 0 655149764 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=405636 CPUtime=24.56 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 13402 1252 0 0 2455 1 0 0 25 0 9 0 655149764 415371264 69082 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236812 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 196.85
Current children cumulated vsize (KiB) 405636

[startup+51.1008 s]
/proc/loadavg: 7.61 7.91 7.90 9/309 30113
/proc/meminfo: memFree=17706976/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=407396 CPUtime=401.38 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 157359 1252 0 0 40111 27 0 0 18 0 9 0 655149672 417173504 70546 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 101849 70546 180 55 0 99735 0
[pid=30098/tid=30105] ppid=30096 vsize=407396 CPUtime=50.17 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 15962 1252 0 0 5015 2 0 0 25 0 9 0 655149763 417173504 70546 33554432000 4194304 4417452 140737458956128 18446744073709551615 4307441 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=407396 CPUtime=50.18 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 22759 1252 0 0 5014 4 0 0 25 0 9 0 655149763 417173504 70546 33554432000 4194304 4417452 140737458956128 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=407396 CPUtime=50.18 cores=0-7

################
# More data... #
################


[startup+4722.31 s]
/proc/loadavg: 8.13 8.07 8.02 9/309 30262
/proc/meminfo: memFree=14863832/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=3228404 CPUtime=37722.7 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 2621866 1252 0 0 3771737 532 0 0 18 0 9 0 655149672 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 807101 779938 180 55 0 804987 0
[pid=30098/tid=30105] ppid=30096 vsize=3228404 CPUtime=4721.18 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 336586 1252 0 0 472049 69 0 0 25 0 9 0 655149763 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=3228404 CPUtime=4721.5 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 408786 1252 0 0 472075 75 0 0 25 0 9 0 655149763 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=3228404 CPUtime=4721.49 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 361649 1252 0 0 472081 68 0 0 25 0 9 0 655149763 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=3228404 CPUtime=4721.45 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 49460 1252 0 0 472133 12 0 0 25 0 9 0 655149763 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=3228404 CPUtime=4719.96 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 402925 1252 0 0 471908 88 0 0 25 0 9 0 655149763 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=3228404 CPUtime=4721.49 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 367046 1252 0 0 472073 76 0 0 25 0 9 0 655149764 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=3228404 CPUtime=4674.28 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 270760 1252 0 0 467374 54 0 0 25 0 9 0 655149764 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=3228404 CPUtime=4720.36 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 387535 1252 0 0 471955 81 0 0 25 0 9 0 655149764 3305885696 779938 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37722.7
Current children cumulated vsize (KiB) 3228404

[startup+4782.3 s]
/proc/loadavg: 8.04 8.06 8.01 9/309 30264
/proc/meminfo: memFree=14758752/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=3333288 CPUtime=38202 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 2648087 1252 0 0 3819664 538 0 0 18 0 9 0 655149672 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 833322 806159 180 55 0 831208 0
[pid=30098/tid=30105] ppid=30096 vsize=3333288 CPUtime=4781.17 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 336586 1252 0 0 478048 69 0 0 25 0 9 0 655149763 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=3333288 CPUtime=4781.49 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 408786 1252 0 0 478074 75 0 0 25 0 9 0 655149763 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=3333288 CPUtime=4781.48 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 371486 1252 0 0 478076 72 0 0 25 0 9 0 655149763 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=3333288 CPUtime=4781.44 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 49460 1252 0 0 478132 12 0 0 25 0 9 0 655149763 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=3333288 CPUtime=4779.94 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 402925 1252 0 0 477906 88 0 0 25 0 9 0 655149763 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=3333288 CPUtime=4781.47 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 383430 1252 0 0 478069 78 0 0 25 0 9 0 655149764 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=3333288 CPUtime=4733.68 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 270760 1252 0 0 473314 54 0 0 25 0 9 0 655149764 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=3333288 CPUtime=4780.35 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 387535 1252 0 0 477954 81 0 0 25 0 9 0 655149764 3413286912 806159 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38202
Current children cumulated vsize (KiB) 3333288

[startup+4842.3 s]
/proc/loadavg: 8.08 8.06 8.01 9/309 30265
/proc/meminfo: memFree=14758712/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=3333324 CPUtime=38681.4 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 2648096 1252 0 0 3867604 538 0 0 18 0 9 0 655149672 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 833331 806168 180 55 0 831217 0
[pid=30098/tid=30105] ppid=30096 vsize=3333324 CPUtime=4841.17 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 336586 1252 0 0 484048 69 0 0 25 0 9 0 655149763 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4263136 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=3333324 CPUtime=4841.5 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 408786 1252 0 0 484075 75 0 0 25 0 9 0 655149763 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=3333324 CPUtime=4841.49 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 371486 1252 0 0 484077 72 0 0 25 0 9 0 655149763 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=3333324 CPUtime=4841.44 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 49460 1252 0 0 484132 12 0 0 25 0 9 0 655149763 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=3333324 CPUtime=4839.93 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 402925 1252 0 0 483905 88 0 0 25 0 9 0 655149763 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=3333324 CPUtime=4841.48 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 383439 1252 0 0 484070 78 0 0 25 0 9 0 655149764 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=3333324 CPUtime=4793.07 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 270760 1252 0 0 479253 54 0 0 25 0 9 0 655149764 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=3333324 CPUtime=4840.35 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 387535 1252 0 0 483954 81 0 0 25 0 9 0 655149764 3413323776 806168 33554432000 4194304 4417452 140737458956128 18446744073709551615 4357902 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38681.4
Current children cumulated vsize (KiB) 3333324

[startup+4902.3 s]
/proc/loadavg: 8.10 8.06 8.01 9/309 30267
/proc/meminfo: memFree=14937728/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=3220844 CPUtime=39160.8 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 2693676 1252 0 0 3915534 548 0 0 18 0 9 0 655149672 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 805211 778048 180 55 0 803097 0
[pid=30098/tid=30105] ppid=30096 vsize=3220844 CPUtime=4901.17 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 336586 1252 0 0 490048 69 0 0 25 0 9 0 655149763 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364637 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=3220844 CPUtime=4901.5 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 408786 1252 0 0 490075 75 0 0 25 0 9 0 655149763 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4365027 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=3220844 CPUtime=4901.49 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 387870 1252 0 0 490072 77 0 0 25 0 9 0 655149763 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4277563 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=3220844 CPUtime=4901.44 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 49460 1252 0 0 490132 12 0 0 25 0 9 0 655149763 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=3220844 CPUtime=4899.92 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 418564 1252 0 0 489901 91 0 0 25 0 9 0 655149763 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=3220844 CPUtime=4901.48 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 383439 1252 0 0 490070 78 0 0 25 0 9 0 655149764 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=3220844 CPUtime=4852.48 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 284317 1252 0 0 485192 56 0 0 25 0 9 0 655149764 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=3220844 CPUtime=4900.36 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 387535 1252 0 0 489955 81 0 0 25 0 9 0 655149764 3298144256 778048 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39160.8
Current children cumulated vsize (KiB) 3220844

[startup+4962.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/309 30268
/proc/meminfo: memFree=14871840/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=3220880 CPUtime=39640.2 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 2693685 1252 0 0 3963473 548 0 0 18 0 9 0 655149672 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 805220 778057 180 55 0 803106 0
[pid=30098/tid=30105] ppid=30096 vsize=3220880 CPUtime=4961.17 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 336586 1252 0 0 496048 69 0 0 25 0 9 0 655149763 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=3220880 CPUtime=4961.5 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 408786 1252 0 0 496075 75 0 0 25 0 9 0 655149763 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=3220880 CPUtime=4961.49 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 387870 1252 0 0 496072 77 0 0 25 0 9 0 655149763 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=3220880 CPUtime=4961.44 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 49460 1252 0 0 496132 12 0 0 25 0 9 0 655149763 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4237290 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=3220880 CPUtime=4959.92 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 418573 1252 0 0 495901 91 0 0 25 0 9 0 655149763 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=3220880 CPUtime=4961.48 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 383439 1252 0 0 496070 78 0 0 25 0 9 0 655149764 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=3220880 CPUtime=4911.88 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 284317 1252 0 0 491132 56 0 0 25 0 9 0 655149764 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=3220880 CPUtime=4960.36 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 387535 1252 0 0 495955 81 0 0 25 0 9 0 655149764 3298181120 778057 33554432000 4194304 4417452 140737458956128 18446744073709551615 4281230 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39640.2
Current children cumulated vsize (KiB) 3220880



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.4 s]
/proc/loadavg: 8.10 8.06 8.01 9/309 30270
/proc/meminfo: memFree=14827536/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=3265700 CPUtime=40000.6 cores=0-7
/proc/30098/stat : 30098 (plingeling) S 30096 30098 30072 0 -1 4202496 2735542 1252 0 0 3999504 558 0 0 18 0 9 0 655149672 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 816425 789262 180 55 0 814311 0
[pid=30098/tid=30105] ppid=30096 vsize=3265700 CPUtime=5006.28 cores=0-7
/proc/30098/task/30105/stat : 30105 (plingeling) R 30096 30098 30072 0 -1 4202560 336586 1252 0 0 500559 69 0 0 25 0 9 0 655149763 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=30098/tid=30106] ppid=30096 vsize=3265700 CPUtime=5006.61 cores=0-7
/proc/30098/task/30106/stat : 30106 (plingeling) R 30096 30098 30072 0 -1 4202560 408786 1252 0 0 500586 75 0 0 25 0 9 0 655149763 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=30098/tid=30107] ppid=30096 vsize=3265700 CPUtime=5006.6 cores=0-7
/proc/30098/task/30107/stat : 30107 (plingeling) R 30096 30098 30072 0 -1 4202560 387870 1252 0 0 500583 77 0 0 25 0 9 0 655149763 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=30098/tid=30108] ppid=30096 vsize=3265700 CPUtime=5006.55 cores=0-7
/proc/30098/task/30108/stat : 30108 (plingeling) R 30096 30098 30072 0 -1 4202560 49460 1252 0 0 500643 12 0 0 25 0 9 0 655149763 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=30098/tid=30109] ppid=30096 vsize=3265700 CPUtime=5005.02 cores=0-7
/proc/30098/task/30109/stat : 30109 (plingeling) R 30096 30098 30072 0 -1 4202560 422549 1252 0 0 500410 92 0 0 25 0 9 0 655149763 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4199435 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=30098/tid=30110] ppid=30096 vsize=3265700 CPUtime=5006.59 cores=0-7
/proc/30098/task/30110/stat : 30110 (plingeling) R 30096 30098 30072 0 -1 4202560 383439 1252 0 0 500581 78 0 0 25 0 9 0 655149764 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=3265700 CPUtime=4956.53 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4202560 284317 1252 0 0 495597 56 0 0 25 0 9 0 655149764 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=30098/tid=30112] ppid=30096 vsize=3265700 CPUtime=5005.47 cores=0-7
/proc/30098/task/30112/stat : 30112 (plingeling) R 30096 30098 30072 0 -1 4202560 425416 1252 0 0 500457 90 0 0 25 0 9 0 655149764 3344076800 789262 33554432000 4194304 4417452 140737458956128 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 3265700

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.52 s]
/proc/loadavg: 8.10 8.06 8.01 9/309 30270
/proc/meminfo: memFree=14827536/32950928 swapFree=67111348/67111528
[pid=30098] ppid=30096 vsize=0 CPUtime=40000.8 cores=0-7
/proc/30098/stat : 30098 (plingeling) Z 30096 30098 30072 0 -1 4203548 2735543 1252 0 0 3999510 568 0 0 18 0 2 0 655149672 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30098/statm: 0 0 0 0 0 0 0
[pid=30098/tid=30111] ppid=30096 vsize=0 CPUtime=4956.63 cores=0-7
/proc/30098/task/30111/stat : 30111 (plingeling) R 30096 30098 30072 0 -1 4203588 284317 1252 0 0 495597 66 0 0 25 0 2 0 655149764 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5007.58
CPU time (s): 40000.9
CPU user time (s): 39995.1
CPU system time (s): 5.74613
CPU usage (%): 798.806
Max. virtual memory (cumulated for all children) (KiB): 3333324

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.1
system time used= 5.74613
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2736795
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 297
involuntary context switches= 194193

runsolver used 14.0689 second user time and 36.0085 second system time

The end

Launcher Data 

Begin job on node125 at 2011-05-15 06:58:20
IDJOB=3372128
IDBENCH=20406
IDSOLVER=1851
FILE ID=node125/3372128-1305435500
RUNJOBID= node125-1305435500-30086
PBS_JOBID= 13324741
Free space on /tmp= 73224 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT07/industrial/palacios/hard/sortnet-7-ipc5-h15-unsat.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372128-1305435500/watcher-3372128-1305435500 -o /tmp/evaluation-result-3372128-1305435500/solver-3372128-1305435500 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372128-1305435500.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 0683489092a5eb0c881dd17c2c45d08e
RANDOM SEED=2002472446

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      17989536 kB
Buffers:        268564 kB
Cached:        4262760 kB
SwapCached:          0 kB
Active:       11177196 kB
Inactive:      3408820 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      17989536 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:            6784 kB
Writeback:           0 kB
AnonPages:    10055204 kB
Mapped:          28996 kB
Slab:           274440 kB
PageTables:      35028 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 50984724 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73224 MiB
End job on node125 at 2011-05-15 08:21:50