Trace number 3309804

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 40001 5004.16

General information on the benchmark

Nameapplication/fuhs/slp-synthesis-AES/
slp-synthesis-aes-top21.cnf
MD5SUM9fab5edd49fb904ff76ff7abf98e0d14
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark18199.9
Satisfiable
(Un)Satisfiability was proved
Number of variables51138
Number of clauses162526
Sum of the clauses size397144
Maximum clause length2101
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 299113
Number of clauses of size 362300
Number of clauses of size 4230
Number of clauses of size 519
Number of clauses of size over 5863

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3309804-1304009274/watcher-3309804-1304009274 -o /tmp/evaluation-result-3309804-1304009274/solver-3309804-1304009274 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3309804-1304009274.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.03 8.04 8.00 1/170 12996
/proc/meminfo: memFree=18510416/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=15496 CPUtime=0 cores=0-7
/proc/12996/stat : 12996 (plingeling) R 12994 12996 12970 0 -1 4202496 1793 1257 0 0 0 0 0 0 25 0 1 0 591158048 15867904 1722 33554432000 4194304 4417332 140733852504128 18446744073709551615 224284619869 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/12996/statm: 3874 1735 140 55 0 1760 0

[startup+0.0741509 s]
/proc/loadavg: 8.03 8.04 8.00 1/170 12996
/proc/meminfo: memFree=18510416/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=30040 CPUtime=0.06 cores=0-7
/proc/12996/stat : 12996 (plingeling) R 12994 12996 12970 0 -1 4202496 5637 1257 0 0 6 0 0 0 25 0 1 0 591158048 30760960 5534 33554432000 4194304 4417332 140733852504128 18446744073709551615 4232654 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/12996/statm: 7510 5534 148 55 0 5396 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 30040

[startup+0.101125 s]
/proc/loadavg: 8.03 8.04 8.00 1/170 12996
/proc/meminfo: memFree=18510416/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=41864 CPUtime=0.09 cores=0-7
/proc/12996/stat : 12996 (plingeling) R 12994 12996 12970 0 -1 4202496 8593 1257 0 0 8 1 0 0 25 0 1 0 591158048 42868736 8490 33554432000 4194304 4417332 140733852504128 18446744073709551615 4259186 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/12996/statm: 10466 8490 148 55 0 8352 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 41864

[startup+0.30111 s]
/proc/loadavg: 8.03 8.04 8.00 1/170 12996
/proc/meminfo: memFree=18510416/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=81044 CPUtime=0.29 cores=0-7
/proc/12996/stat : 12996 (plingeling) R 12994 12996 12970 0 -1 4202496 18388 1257 0 0 27 2 0 0 25 0 1 0 591158048 82989056 18285 33554432000 4194304 4417332 140733852504128 18446744073709551615 4259140 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/12996/statm: 20261 18285 148 55 0 18147 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 81044

[startup+1.38569 s]
/proc/loadavg: 8.03 8.04 8.00 1/170 12996
/proc/meminfo: memFree=18510416/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=288260 CPUtime=8.1 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 46709 1257 0 0 804 6 0 0 18 0 9 0 591158048 295178240 22383 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 72065 22383 175 55 0 69951 0
Current children cumulated CPU time (s) 8.1
Current children cumulated vsize (KiB) 288260

[startup+1.50086 s]
/proc/loadavg: 8.03 8.04 8.00 1/170 12996
/proc/meminfo: memFree=18510416/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=288260 CPUtime=9.02 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 48125 1257 0 0 896 6 0 0 18 0 9 0 591158048 295178240 22170 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 72065 22170 175 55 0 69951 0
Current children cumulated CPU time (s) 9.02
Current children cumulated vsize (KiB) 288260

[startup+3.10059 s]
/proc/loadavg: 8.11 8.06 8.01 9/179 13010
/proc/meminfo: memFree=18424644/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=298796 CPUtime=21.8 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 63335 1257 0 0 2172 8 0 0 18 0 9 0 591158048 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 74699 22594 181 55 0 72585 0
[pid=12996/tid=13003] ppid=12994 vsize=298796 CPUtime=2.7 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 13391 1257 0 0 268 2 0 0 22 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4255239 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=298796 CPUtime=2.68 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 3340 1257 0 0 268 0 0 0 22 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4253179 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=298796 CPUtime=2.71 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 2313 1257 0 0 271 0 0 0 23 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4334556 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=298796 CPUtime=2.68 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 951 1257 0 0 268 0 0 0 22 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=298796 CPUtime=2.7 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 8200 1257 0 0 269 1 0 0 23 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4254099 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=298796 CPUtime=2.7 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 3650 1257 0 0 270 0 0 0 23 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4320455 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=298796 CPUtime=2.61 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 5377 1257 0 0 261 0 0 0 22 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4378059 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=298796 CPUtime=2.59 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 7661 1257 0 0 259 0 0 0 22 0 9 0 591158086 305967104 22594 33554432000 4194304 4417332 140733852504128 18446744073709551615 4320576 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 21.8
Current children cumulated vsize (KiB) 298796

[startup+6.30105 s]
/proc/loadavg: 8.10 8.06 8.00 9/179 13010
/proc/meminfo: memFree=18399348/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=316644 CPUtime=47.39 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 68361 1257 0 0 4730 9 0 0 18 0 9 0 591158048 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 79161 27592 181 55 0 77047 0
[pid=12996/tid=13003] ppid=12994 vsize=316644 CPUtime=5.9 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 14800 1257 0 0 588 2 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4250022 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=316644 CPUtime=5.88 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 3933 1257 0 0 588 0 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=316644 CPUtime=5.91 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 2313 1257 0 0 591 0 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=316644 CPUtime=5.88 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 951 1257 0 0 588 0 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4365894 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=316644 CPUtime=5.9 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 8924 1257 0 0 589 1 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4252757 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=316644 CPUtime=5.9 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 4544 1257 0 0 590 0 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4236224 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=316644 CPUtime=5.81 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 5377 1257 0 0 581 0 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4236766 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=316644 CPUtime=5.78 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 9067 1257 0 0 577 1 0 0 25 0 9 0 591158086 324243456 27592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4248897 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 47.39
Current children cumulated vsize (KiB) 316644

[startup+12.701 s]
/proc/loadavg: 8.09 8.06 8.00 9/179 13012
/proc/meminfo: memFree=18397288/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=318140 CPUtime=98.59 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 73691 1257 0 0 9850 9 0 0 18 0 9 0 591158048 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 79535 28167 181 55 0 77421 0
[pid=12996/tid=13003] ppid=12994 vsize=318140 CPUtime=12.3 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 16534 1257 0 0 1228 2 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=318140 CPUtime=12.28 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 4581 1257 0 0 1228 0 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=318140 CPUtime=12.32 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 2394 1257 0 0 1232 0 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4264976 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=318140 CPUtime=12.29 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 1227 1257 0 0 1229 0 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364417 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=318140 CPUtime=12.31 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 9444 1257 0 0 1230 1 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=318140 CPUtime=12.3 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 5568 1257 0 0 1230 0 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4265437 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=318140 CPUtime=12.22 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 5393 1257 0 0 1222 0 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364491 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=318140 CPUtime=12.15 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 10098 1257 0 0 1214 1 0 0 25 0 9 0 591158086 325775360 28167 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 98.59
Current children cumulated vsize (KiB) 318140

[startup+25.5009 s]
/proc/loadavg: 8.08 8.06 8.00 9/179 13012
/proc/meminfo: memFree=18359956/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=363464 CPUtime=200.88 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 95637 1257 0 0 20076 12 0 0 18 0 9 0 591158048 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 90866 38811 181 55 0 88752 0
[pid=12996/tid=13003] ppid=12994 vsize=363464 CPUtime=25.1 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 22441 1257 0 0 2507 3 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=363464 CPUtime=25.08 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 7528 1257 0 0 2507 1 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4358118 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=363464 CPUtime=25.11 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 6244 1257 0 0 2511 0 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4275873 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=363464 CPUtime=25.08 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 2341 1257 0 0 2508 0 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=363464 CPUtime=25.1 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 11281 1257 0 0 2509 1 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=363464 CPUtime=25.1 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 8280 1257 0 0 2510 0 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4236906 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=363464 CPUtime=25.01 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 6949 1257 0 0 2501 0 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4365033 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=363464 CPUtime=24.88 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 12121 1257 0 0 2487 1 0 0 25 0 9 0 591158086 372187136 38811 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 200.88
Current children cumulated vsize (KiB) 363464

[startup+51.1006 s]
/proc/loadavg: 8.05 8.05 8.00 9/179 13012
/proc/meminfo: memFree=18323680/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=389144 CPUtime=405.55 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 140164 1257 0 0 40536 19 0 0 18 0 9 0 591158048 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 97286 46592 181 55 0 95172 0
[pid=12996/tid=13003] ppid=12994 vsize=389144 CPUtime=50.7 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 35731 1257 0 0 5065 5 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=389144 CPUtime=50.68 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 13742 1257 0 0 5066 2 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4366297 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=389144 CPUtime=50.71 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 9972 1257 0 0 5070 1 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=389144 CPUtime=50.68 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 3075 1257 0 0 5068 0 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4352306 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=389144 CPUtime=50.7 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 16627 1257 0 0 5068 2 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=389144 CPUtime=50.7 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 12798 1257 0 0 5069 1 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4353472 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=389144 CPUtime=50.61 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 10057 1257 0 0 5060 1 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364491 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=389144 CPUtime=50.34 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 19710 1257 0 0 5032 2 0 0 25 0 9 0 591158086 398483456 46592 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364722 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 405.55
Current children cumulated vsize (KiB) 389144

[startup+102.301 s]
/proc/loadavg: 8.06 8.05 8.00 9/179 13014

################
# More data... #
################


[startup+4722.3 s]
/proc/loadavg: 8.09 8.06 8.00 9/179 13157
/proc/meminfo: memFree=15308008/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=3409796 CPUtime=37748.7 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 5444248 1257 0 0 3773871 996 0 0 18 0 9 0 591158048 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 852449 799886 181 55 0 850335 0
[pid=12996/tid=13003] ppid=12994 vsize=3409796 CPUtime=4721.93 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 789201 1257 0 0 472045 148 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=3409796 CPUtime=4721.94 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 698311 1257 0 0 472063 131 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=3409796 CPUtime=4721.94 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 688171 1257 0 0 472063 131 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=3409796 CPUtime=4721.95 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 543014 1257 0 0 472114 81 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364463 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=3409796 CPUtime=4721.97 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 627459 1257 0 0 472074 123 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=3409796 CPUtime=4721.11 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 748506 1257 0 0 471970 141 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=3409796 CPUtime=4721.42 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 661927 1257 0 0 472032 110 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364502 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=3409796 CPUtime=4695.98 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 669207 1257 0 0 469472 126 0 0 25 0 9 0 591158086 3491631104 799886 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37748.7
Current children cumulated vsize (KiB) 3409796

[startup+4782.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/179 13160
/proc/meminfo: memFree=15504264/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=3213860 CPUtime=38228.4 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 5470307 1257 0 0 3821838 1000 0 0 18 0 9 0 591158048 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 803465 750902 181 55 0 801351 0
[pid=12996/tid=13003] ppid=12994 vsize=3213860 CPUtime=4781.94 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 789201 1257 0 0 478046 148 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=3213860 CPUtime=4781.95 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 698311 1257 0 0 478064 131 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=3213860 CPUtime=4781.95 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 709178 1257 0 0 478060 135 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=3213860 CPUtime=4781.95 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 548061 1257 0 0 478114 81 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=3213860 CPUtime=4781.97 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 627464 1257 0 0 478074 123 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364683 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=3213860 CPUtime=4781.12 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 748506 1257 0 0 477971 141 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=3213860 CPUtime=4781.42 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 661927 1257 0 0 478032 110 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=3213860 CPUtime=4755.65 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 669207 1257 0 0 475439 126 0 0 25 0 9 0 591158086 3290992640 750902 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38228.4
Current children cumulated vsize (KiB) 3213860

[startup+4842.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/179 13162
/proc/meminfo: memFree=15362256/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=3355532 CPUtime=38708 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 5548766 1257 0 0 3869785 1017 0 0 18 0 9 0 591158048 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 838883 786320 181 55 0 836769 0
[pid=12996/tid=13003] ppid=12994 vsize=3355532 CPUtime=4841.93 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 789201 1257 0 0 484045 148 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=3355532 CPUtime=4841.94 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 736097 1257 0 0 484058 136 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364482 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=3355532 CPUtime=4841.94 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 741946 1257 0 0 484049 145 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=3355532 CPUtime=4841.94 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 555966 1257 0 0 484112 82 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=3355532 CPUtime=4841.97 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 627464 1257 0 0 484074 123 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=3355532 CPUtime=4841.12 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 748506 1257 0 0 483971 141 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=3355532 CPUtime=4841.41 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 661927 1257 0 0 484031 110 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=3355532 CPUtime=4815.32 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 669207 1257 0 0 481406 126 0 0 25 0 9 0 591158086 3436064768 786320 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38708
Current children cumulated vsize (KiB) 3355532

[startup+4902.31 s]
/proc/loadavg: 8.07 8.06 8.00 9/179 13164
/proc/meminfo: memFree=15349556/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=3368136 CPUtime=39187.7 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 5551917 1257 0 0 3917756 1017 0 0 18 0 9 0 591158048 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 842034 789471 181 55 0 839920 0
[pid=12996/tid=13003] ppid=12994 vsize=3368136 CPUtime=4901.94 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 789201 1257 0 0 490046 148 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364465 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=3368136 CPUtime=4901.95 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 736097 1257 0 0 490059 136 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4365846 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=3368136 CPUtime=4901.95 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 741946 1257 0 0 490050 145 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=3368136 CPUtime=4901.95 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 559117 1257 0 0 490113 82 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=3368136 CPUtime=4901.97 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 627464 1257 0 0 490074 123 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=3368136 CPUtime=4901.12 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 748506 1257 0 0 489971 141 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=3368136 CPUtime=4901.42 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 661927 1257 0 0 490032 110 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=3368136 CPUtime=4875 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 669207 1257 0 0 487374 126 0 0 25 0 9 0 591158086 3448971264 789471 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39187.7
Current children cumulated vsize (KiB) 3368136

[startup+4962.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/179 13164
/proc/meminfo: memFree=15227756/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=3490012 CPUtime=39667.4 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 5596267 1257 0 0 3965708 1028 0 0 18 0 9 0 591158048 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 872503 819940 181 55 0 870389 0
[pid=12996/tid=13003] ppid=12994 vsize=3490012 CPUtime=4961.94 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 821969 1257 0 0 496036 158 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=3490012 CPUtime=4961.94 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 736097 1257 0 0 496058 136 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4365018 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=3490012 CPUtime=4961.94 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 741946 1257 0 0 496049 145 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=3490012 CPUtime=4961.95 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 570699 1257 0 0 496111 84 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=3490012 CPUtime=4961.97 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 627464 1257 0 0 496074 123 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=3490012 CPUtime=4961.12 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 748506 1257 0 0 495971 141 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364502 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=3490012 CPUtime=4961.4 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 661927 1257 0 0 496030 110 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=3490012 CPUtime=4934.67 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 669207 1257 0 0 493341 126 0 0 25 0 9 0 591158086 3573772288 819940 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39667.4
Current children cumulated vsize (KiB) 3490012



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5004 s]
/proc/loadavg: 8.08 8.07 8.00 9/179 13166
/proc/meminfo: memFree=15214452/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=3503144 CPUtime=40000.8 cores=0-7
/proc/12996/stat : 12996 (plingeling) S 12994 12996 12970 0 -1 4202496 5599550 1257 0 0 3999049 1029 0 0 18 0 9 0 591158048 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 875786 823223 181 55 0 873672 0
[pid=12996/tid=13003] ppid=12994 vsize=3503144 CPUtime=5003.64 cores=0-7
/proc/12996/task/13003/stat : 13003 (plingeling) R 12994 12996 12970 0 -1 4202560 821969 1257 0 0 500206 158 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364680 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=12996/tid=13004] ppid=12994 vsize=3503144 CPUtime=5003.65 cores=0-7
/proc/12996/task/13004/stat : 13004 (plingeling) R 12994 12996 12970 0 -1 4202560 736097 1257 0 0 500229 136 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4354227 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=12996/tid=13005] ppid=12994 vsize=3503144 CPUtime=5003.65 cores=0-7
/proc/12996/task/13005/stat : 13005 (plingeling) R 12994 12996 12970 0 -1 4202560 741946 1257 0 0 500220 145 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=12996/tid=13006] ppid=12994 vsize=3503144 CPUtime=5003.65 cores=0-7
/proc/12996/task/13006/stat : 13006 (plingeling) R 12994 12996 12970 0 -1 4202560 573982 1257 0 0 500281 84 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=12996/tid=13007] ppid=12994 vsize=3503144 CPUtime=5003.68 cores=0-7
/proc/12996/task/13007/stat : 13007 (plingeling) R 12994 12996 12970 0 -1 4202560 627464 1257 0 0 500245 123 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=12996/tid=13008] ppid=12994 vsize=3503144 CPUtime=5002.82 cores=0-7
/proc/12996/task/13008/stat : 13008 (plingeling) R 12994 12996 12970 0 -1 4202560 748506 1257 0 0 500141 141 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=12996/tid=13009] ppid=12994 vsize=3503144 CPUtime=5003.11 cores=0-7
/proc/12996/task/13009/stat : 13009 (plingeling) R 12994 12996 12970 0 -1 4202560 661927 1257 0 0 500201 110 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=3503144 CPUtime=4976.14 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4202560 669207 1257 0 0 497488 126 0 0 25 0 9 0 591158086 3587219456 823223 33554432000 4194304 4417332 140733852504128 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 3503144

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5004.11 s]
/proc/loadavg: 8.08 8.07 8.00 9/179 13166
/proc/meminfo: memFree=15214452/32951124 swapFree=67110396/67111528
[pid=12996] ppid=12994 vsize=0 CPUtime=40000.9 cores=0-7
/proc/12996/stat : 12996 (plingeling) Z 12994 12996 12970 0 -1 4203548 5599550 1257 0 0 3999053 1039 0 0 18 0 2 0 591158048 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 7 0 0 0
/proc/12996/statm: 0 0 0 0 0 0 0
[pid=12996/tid=13010] ppid=12994 vsize=0 CPUtime=4976.23 cores=0-7
/proc/12996/task/13010/stat : 13010 (plingeling) R 12994 12996 12970 0 -1 4203588 669207 1257 0 0 497488 135 0 0 25 0 2 0 591158086 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5004.16
CPU time (s): 40001
CPU user time (s): 39990.5
CPU system time (s): 10.4364
CPU usage (%): 799.355
Max. virtual memory (cumulated for all children) (KiB): 3591504

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39990.5
system time used= 10.4364
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 5600807
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 537
involuntary context switches= 192285

runsolver used 8.02578 second user time and 19.2691 second system time

The end

Launcher Data 

Begin job on node118 at 2011-04-28 18:47:54
IDJOB=3309804
IDBENCH=83115
IDSOLVER=1695
FILE ID=node118/3309804-1304009274
RUNJOBID= node118-1304009274-12984
PBS_JOBID= 13168849
Free space on /tmp= 71616 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/application/fuhs/slp-synthesis-AES/slp-synthesis-aes-top21.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3309804-1304009274/watcher-3309804-1304009274 -o /tmp/evaluation-result-3309804-1304009274/solver-3309804-1304009274 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3309804-1304009274.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 9fab5edd49fb904ff76ff7abf98e0d14
RANDOM SEED=1005580862

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18510696 kB
Buffers:       1895236 kB
Cached:       11943324 kB
SwapCached:        600 kB
Active:        9363548 kB
Inactive:      4539092 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18510696 kB
SwapTotal:    67111528 kB
SwapFree:     67110396 kB
Dirty:            2748 kB
Writeback:           0 kB
AnonPages:       63252 kB
Mapped:          14028 kB
Slab:           471164 kB
PageTables:       4256 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183892 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71616 MiB
End job on node118 at 2011-04-28 20:11:20