Trace number 3322502

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 35674.7 5100.41

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_10_keyfind_1.cnf
MD5SUMf35f3aee72237c1bf3d673feb4883756
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables8080
Number of clauses96704
Sum of the clauses size539496
Maximum clause length8
Minimum clause length1
Number of clauses of size 1336
Number of clauses of size 21152
Number of clauses of size 315040
Number of clauses of size 42944
Number of clauses of size 518432
Number of clauses of size over 558800

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...387047/instance-3322502-1304387047.cnf
0.19/0.23	c Solving...
35674.28/5100.03	
35674.28/5100.03	*** INTERRUPTED! ***
35674.28/5100.03	*** Shutting down threads...
35674.69/5100.41	s UNKNOWN
35674.69/5100.41	
35674.69/5100.41	c INTERRUPTED   : 1
35674.69/5100.41	c Models        : 0+    
35674.69/5100.41	c Time          : 5100.032s (Solving: 5099.80s 1st Model: 0.00s Unsat: 0.00s)
35674.69/5100.41	c CPU Time      : 35674.320s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322502-1304387047/watcher-3322502-1304387047 -o /tmp/evaluation-result-3322502-1304387047/solver-3322502-1304387047 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322502-1304387047.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.44 8.95 8.97 3/345 20791
/proc/meminfo: memFree=18896628/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=24364 CPUtime=0.01 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 1523 0 0 0 1 0 0 0 25 0 1 0 628935374 24948736 1300 33554432000 4194304 4964704 140733966956608 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/20791/statm: 6091 1301 389 189 0 965 0

[startup+0.0927651 s]
/proc/loadavg: 8.44 8.95 8.97 3/345 20791
/proc/meminfo: memFree=18896628/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=33052 CPUtime=0.08 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 4037 0 0 0 8 0 0 0 25 0 1 0 628935374 33845248 3353 33554432000 4194304 4964704 140733966956608 18446744073709551615 4607472 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/20791/statm: 8263 3353 403 189 0 3137 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 33052

[startup+0.100762 s]
/proc/loadavg: 8.44 8.95 8.97 3/345 20791
/proc/meminfo: memFree=18896628/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=33052 CPUtime=0.09 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 4037 0 0 0 9 0 0 0 25 0 1 0 628935374 33845248 3353 33554432000 4194304 4964704 140733966956608 18446744073709551615 4607549 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/20791/statm: 8263 3353 403 189 0 3137 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 33052

[startup+0.300721 s]
/proc/loadavg: 8.44 8.95 8.97 3/345 20791
/proc/meminfo: memFree=18896628/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=534132 CPUtime=0.32 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 14856 0 0 0 29 3 0 0 25 0 8 0 628935374 546951168 13606 33554432000 4194304 4964704 140733966956608 18446744073709551615 4651984 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 133533 13608 454 189 0 127882 0
Current children cumulated CPU time (s) 0.32
Current children cumulated vsize (KiB) 534132

[startup+0.802433 s]
/proc/loadavg: 8.44 8.95 8.97 3/345 20791
/proc/meminfo: memFree=18896628/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=534488 CPUtime=3.8 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 18649 0 0 0 374 6 0 0 18 0 8 0 628935374 547315712 17399 33554432000 4194304 4964704 140733966956608 18446744073709551615 4658370 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 133622 17399 464 189 0 127971 0
Current children cumulated CPU time (s) 3.8
Current children cumulated vsize (KiB) 534488

[startup+1.51344 s]
/proc/loadavg: 8.44 8.95 8.97 10/353 20799
/proc/meminfo: memFree=18817208/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=536076 CPUtime=8.84 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 22487 0 0 0 877 7 0 0 18 0 8 0 628935374 548941824 21237 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 134019 21238 464 189 0 128368 0
[pid=20791/tid=20793] ppid=20789 vsize=536076 CPUtime=0.75 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 1276 0 0 0 75 0 0 0 18 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 4370229 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=536076 CPUtime=1.18 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 3023 0 0 0 118 0 0 0 19 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415058906 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=536076 CPUtime=1.2 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 2869 0 0 0 119 1 0 0 19 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 4320897 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=536076 CPUtime=1.22 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 1978 0 0 0 122 0 0 0 19 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=536076 CPUtime=1.2 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) S 20789 20791 20765 0 -1 4202560 2227 0 0 0 120 0 0 0 18 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415496686 0 0 4096 18946 18446604442867258432 0 0 -1 3 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=536076 CPUtime=0.94 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 3091 0 0 0 94 0 0 0 18 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415052833 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=536076 CPUtime=0.84 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 2570 0 0 0 84 0 0 0 18 0 8 0 628935397 548941824 21238 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.84
Current children cumulated vsize (KiB) 536076

[startup+3.10113 s]
/proc/loadavg: 8.44 8.95 8.97 10/353 20799
/proc/meminfo: memFree=18806916/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=538332 CPUtime=20.53 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 25996 0 0 0 2044 9 0 0 20 0 8 0 628935374 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 134583 24746 464 189 0 128932 0
[pid=20791/tid=20793] ppid=20789 vsize=538332 CPUtime=2.33 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 1426 0 0 0 233 0 0 0 20 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4668673 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=538332 CPUtime=2.76 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 3675 0 0 0 275 1 0 0 22 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4321023 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=538332 CPUtime=2.09 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 3003 0 0 0 208 1 0 0 20 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=538332 CPUtime=2.8 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 2420 0 0 0 280 0 0 0 23 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4660874 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=538332 CPUtime=2.78 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 2753 0 0 0 278 0 0 0 20 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=538332 CPUtime=2.53 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 3774 0 0 0 252 1 0 0 20 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=538332 CPUtime=2.44 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 3372 0 0 0 243 1 0 0 21 0 8 0 628935397 551251968 24746 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 20.53
Current children cumulated vsize (KiB) 538332

[startup+6.30053 s]
/proc/loadavg: 8.49 8.95 8.97 10/353 20799
/proc/meminfo: memFree=18780496/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=541108 CPUtime=42.84 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 31687 0 0 0 4272 12 0 0 19 0 8 0 628935374 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 135277 30437 466 189 0 129626 0
[pid=20791/tid=20793] ppid=20789 vsize=541108 CPUtime=5.28 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 1720 0 0 0 527 1 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4370644 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=541108 CPUtime=4.54 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 4359 0 0 0 453 1 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=541108 CPUtime=5.04 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 4296 0 0 0 502 2 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=541108 CPUtime=5.69 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 2624 0 0 0 569 0 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670565 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=541108 CPUtime=5.89 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 3562 0 0 0 588 1 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4334250 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=541108 CPUtime=5.47 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 4366 0 0 0 546 1 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=541108 CPUtime=5.03 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 5034 0 0 0 502 1 0 0 25 0 8 0 628935397 554094592 30437 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 42.84
Current children cumulated vsize (KiB) 541108

[startup+12.7013 s]
/proc/loadavg: 8.53 8.95 8.97 10/353 20799
/proc/meminfo: memFree=18750724/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=544376 CPUtime=87.53 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 39803 0 0 0 8735 18 0 0 25 0 8 0 628935374 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4370224 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/20791/statm: 136094 38553 466 189 0 130443 0
[pid=20791/tid=20793] ppid=20789 vsize=544376 CPUtime=11.68 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 2217 0 0 0 1167 1 0 0 25 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4338800 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=544376 CPUtime=10.88 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 5637 0 0 0 1086 2 0 0 25 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415496735 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=544376 CPUtime=10.63 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 5317 0 0 0 1059 4 0 0 25 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4342167 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=544376 CPUtime=12.09 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 3276 0 0 0 1208 1 0 0 25 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=544376 CPUtime=9.25 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 4242 0 0 0 924 1 0 0 19 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=544376 CPUtime=11.86 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 5998 0 0 0 1184 2 0 0 25 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=544376 CPUtime=11.42 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 7154 0 0 0 1139 3 0 0 25 0 8 0 628935397 557441024 38553 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 87.53
Current children cumulated vsize (KiB) 544376

[startup+25.5009 s]
/proc/loadavg: 8.70 8.97 8.98 10/353 20800
/proc/meminfo: memFree=18703612/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=551172 CPUtime=176.91 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 51486 0 0 0 17666 25 0 0 25 0 8 0 628935374 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4321079 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/20791/statm: 137793 50236 466 189 0 132142 0
[pid=20791/tid=20793] ppid=20789 vsize=551172 CPUtime=21.36 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 3290 0 0 0 2134 2 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=551172 CPUtime=21.07 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 7051 0 0 0 2104 3 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415496686 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=551172 CPUtime=22.84 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 6564 0 0 0 2280 4 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=551172 CPUtime=22.17 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 4112 0 0 0 2216 1 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=551172 CPUtime=20.57 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 6177 0 0 0 2055 2 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=551172 CPUtime=24.66 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 7144 0 0 0 2463 3 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4653272 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=551172 CPUtime=21.71 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 10323 0 0 0 2166 5 0 0 25 0 8 0 628935397 564400128 50236 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670283 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 176.91
Current children cumulated vsize (KiB) 551172

[startup+51.1011 s]
/proc/loadavg: 8.88 8.99 8.98 10/353 20800
/proc/meminfo: memFree=18641920/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=558704 CPUtime=355.77 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 66377 0 0 0 35541 36 0 0 25 0 8 0 628935374 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4668632 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/20791/statm: 139676 65104 466 189 0 134025 0
[pid=20791/tid=20793] ppid=20789 vsize=558704 CPUtime=44.56 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 3977 0 0 0 4454 2 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=558704 CPUtime=44.93 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 8299 0 0 0 4489 4 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=558704 CPUtime=46.18 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 7987 0 0 0 4613 5 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=558704 CPUtime=46.49 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 5370 0 0 0 4646 3 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670260 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=558704 CPUtime=42.02 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 7641 0 0 0 4198 4 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=558704 CPUtime=45.29 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 11075 0 0 0 4524 5 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4660068 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=558704 CPUtime=38.22 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 13265 0 0 0 3815 7 0 0 25 0 8 0 628935397 572112896 65104 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 355.77

################
# More data... #
################

/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59253 0 0 0 438357 59 0 0 25 0 8 0 628935397 2217013248 472667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4668632 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2165052 CPUtime=3914 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 124189 0 0 0 391263 137 0 0 25 0 8 0 628935397 2217013248 472667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 33454
Current children cumulated vsize (KiB) 2165052

[startup+4842.3 s]
/proc/loadavg: 9.02 9.03 9.00 10/353 21447
/proc/meminfo: memFree=17032996/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2165052 CPUtime=33873.3 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 483123 0 0 0 3386798 534 0 0 25 0 8 0 628935374 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4684027 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/20791/statm: 541263 474681 466 189 0 535612 0
[pid=20791/tid=20793] ppid=20789 vsize=2165052 CPUtime=4311.17 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 22808 0 0 0 431096 21 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=2165052 CPUtime=4340.59 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 35955 0 0 0 434031 28 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670241 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=2165052 CPUtime=4008.65 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 37292 0 0 0 400828 37 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=2165052 CPUtime=4508.56 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 52865 0 0 0 450793 63 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=2165052 CPUtime=4480.95 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 52757 0 0 0 448022 73 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=2165052 CPUtime=4438.06 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59253 0 0 0 443747 59 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2165052 CPUtime=3972.2 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 124230 0 0 0 397083 137 0 0 25 0 8 0 628935397 2217013248 474681 33554432000 4194304 4964704 140733966956608 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 33873.3
Current children cumulated vsize (KiB) 2165052

[startup+4902.3 s]
/proc/loadavg: 9.10 9.05 9.01 10/353 21449
/proc/meminfo: memFree=16993004/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2230588 CPUtime=34292.6 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 493310 0 0 0 3428712 546 0 0 25 0 8 0 628935374 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 557647 484667 466 189 0 551996 0
[pid=20791/tid=20793] ppid=20789 vsize=2230588 CPUtime=4354.33 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 23074 0 0 0 435411 22 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=2230588 CPUtime=4400.58 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 35955 0 0 0 440030 28 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=2230588 CPUtime=4068.63 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 37294 0 0 0 406826 37 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=2230588 CPUtime=4554.93 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 52866 0 0 0 455430 63 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=2230588 CPUtime=4540.95 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 52764 0 0 0 454022 73 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=2230588 CPUtime=4498.06 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59592 0 0 0 449747 59 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2230588 CPUtime=4032.2 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 131622 0 0 0 403075 145 0 0 25 0 8 0 628935397 2284122112 484667 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 34292.6
Current children cumulated vsize (KiB) 2230588

[startup+4962.3 s]
/proc/loadavg: 9.14 9.07 9.01 10/353 21450
/proc/meminfo: memFree=16990252/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2230588 CPUtime=34711.9 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 494125 0 0 0 3470640 548 0 0 25 0 8 0 628935374 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 557647 485482 466 189 0 551996 0
[pid=20791/tid=20793] ppid=20789 vsize=2230588 CPUtime=4384.92 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 23083 0 0 0 438470 22 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=2230588 CPUtime=4460.58 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 35955 0 0 0 446030 28 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=2230588 CPUtime=4128.62 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 37395 0 0 0 412825 37 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=2230588 CPUtime=4614.24 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 53124 0 0 0 461361 63 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=2230588 CPUtime=4589.8 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 52764 0 0 0 458907 73 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=2230588 CPUtime=4558.07 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59594 0 0 0 455747 60 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2230588 CPUtime=4092.19 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 131623 0 0 0 409074 145 0 0 25 0 8 0 628935397 2284122112 485482 33554432000 4194304 4964704 140733966956608 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 34711.9
Current children cumulated vsize (KiB) 2230588

[startup+5022.3 s]
/proc/loadavg: 9.04 9.05 9.01 10/353 21452
/proc/meminfo: memFree=16988852/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2230588 CPUtime=35131.1 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 494482 0 0 0 3512566 549 0 0 25 0 8 0 628935374 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 557647 485802 466 189 0 551996 0
[pid=20791/tid=20793] ppid=20789 vsize=2230588 CPUtime=4442.32 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 23084 0 0 0 444210 22 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670327 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=2230588 CPUtime=4513.97 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 35955 0 0 0 451369 28 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=2230588 CPUtime=4168.35 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 37395 0 0 0 416798 37 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=2230588 CPUtime=4673.52 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 53124 0 0 0 467289 63 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=2230588 CPUtime=4633.03 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 53109 0 0 0 463230 73 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=2230588 CPUtime=4606.09 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59594 0 0 0 460549 60 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670281 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2230588 CPUtime=4150.39 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 131624 0 0 0 414894 145 0 0 25 0 8 0 628935397 2284122112 485802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 35131.1
Current children cumulated vsize (KiB) 2230588

[startup+5082.3 s]
/proc/loadavg: 9.06 9.06 9.01 10/353 21454
/proc/meminfo: memFree=16984832/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2230588 CPUtime=35550.5 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 495692 0 0 0 3554498 552 0 0 25 0 8 0 628935374 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 557647 486801 466 189 0 551996 0
[pid=20791/tid=20793] ppid=20789 vsize=2230588 CPUtime=4481.55 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 23311 0 0 0 448133 22 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=2230588 CPUtime=4568.39 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 35955 0 0 0 456811 28 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=2230588 CPUtime=4203.85 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 37440 0 0 0 420348 37 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=2230588 CPUtime=4732.82 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 53124 0 0 0 473219 63 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=2230588 CPUtime=4693.03 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 53127 0 0 0 469230 73 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=2230588 CPUtime=4656.98 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59697 0 0 0 465638 60 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2230588 CPUtime=4210.41 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 131626 0 0 0 420895 146 0 0 25 0 8 0 628935397 2284122112 486801 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 35550.5
Current children cumulated vsize (KiB) 2230588



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100.01 s]
/proc/loadavg: 9.04 9.05 9.00 10/353 21454
/proc/meminfo: memFree=16984804/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2230588 CPUtime=35674.3 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 495693 0 0 0 3566876 552 0 0 25 0 8 0 628935374 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/20791/statm: 557647 486802 466 189 0 551996 0
[pid=20791/tid=20793] ppid=20789 vsize=2230588 CPUtime=4497.04 cores=0-7
/proc/20791/task/20793/stat : 20793 (claspmt) R 20789 20791 20765 0 -1 4202560 23311 0 0 0 449682 22 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20791/tid=20794] ppid=20789 vsize=2230588 CPUtime=4579.11 cores=0-7
/proc/20791/task/20794/stat : 20794 (claspmt) R 20789 20791 20765 0 -1 4202560 35955 0 0 0 457883 28 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20795] ppid=20789 vsize=2230588 CPUtime=4221.56 cores=0-7
/proc/20791/task/20795/stat : 20795 (claspmt) R 20789 20791 20765 0 -1 4202560 37440 0 0 0 422119 37 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=20791/tid=20796] ppid=20789 vsize=2230588 CPUtime=4750.31 cores=0-7
/proc/20791/task/20796/stat : 20796 (claspmt) R 20789 20791 20765 0 -1 4202560 53124 0 0 0 474968 63 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20791/tid=20797] ppid=20789 vsize=2230588 CPUtime=4702.25 cores=0-7
/proc/20791/task/20797/stat : 20797 (claspmt) R 20789 20791 20765 0 -1 4202560 53127 0 0 0 470152 73 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20791/tid=20798] ppid=20789 vsize=2230588 CPUtime=4674.69 cores=0-7
/proc/20791/task/20798/stat : 20798 (claspmt) R 20789 20791 20765 0 -1 4202560 59698 0 0 0 467409 60 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20791/tid=20799] ppid=20789 vsize=2230588 CPUtime=4228.12 cores=0-7
/proc/20791/task/20799/stat : 20799 (claspmt) R 20789 20791 20765 0 -1 4202560 131626 0 0 0 422666 146 0 0 25 0 8 0 628935397 2284122112 486802 33554432000 4194304 4964704 140733966956608 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 35674.3
Current children cumulated vsize (KiB) 2230588

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.2 s]
/proc/loadavg: 9.04 9.05 9.00 10/353 21454
/proc/meminfo: memFree=16984804/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=2224712 CPUtime=35674.5 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 495703 0 0 0 3566897 552 0 0 25 0 1 0 628935374 2278105088 485248 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415041100 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/20791/statm: 556178 485248 476 189 0 550527 0
Current children cumulated CPU time (s) 35674.5
Current children cumulated vsize (KiB) 2224712

[startup+5100.4 s]
/proc/loadavg: 9.04 9.05 9.00 10/353 21454
/proc/meminfo: memFree=16984804/32950928 swapFree=67111364/67111528
[pid=20791] ppid=20789 vsize=721236 CPUtime=35674.7 cores=0-7
/proc/20791/stat : 20791 (claspmt) R 20789 20791 20765 0 -1 4202496 495703 0 0 0 3566910 559 0 0 25 0 1 0 628935374 738545664 70704 33554432000 4194304 4964704 140733966956608 18446744073709551615 256415434887 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/20791/statm: 180309 70704 476 189 0 174658 0
Current children cumulated CPU time (s) 35674.7
Current children cumulated vsize (KiB) 721236

Child status: 0
Real time (s): 5100.41
CPU time (s): 35674.7
CPU user time (s): 35669.1
CPU system time (s): 5.60415
CPU usage (%): 699.447
Max. virtual memory (cumulated for all children) (KiB): 2230588

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35669.1
system time used= 5.60415
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 495704
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 61507
involuntary context switches= 246031

runsolver used 16.0456 second user time and 42.1106 second system time

The end

Launcher Data 

Begin job on node115 at 2011-05-03 03:44:07
IDJOB=3322502
IDBENCH=83413
IDSOLVER=1711
FILE ID=node115/3322502-1304387047
RUNJOBID= node115-1304387047-20779
PBS_JOBID= 13181814
Free space on /tmp= 73472 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_10_keyfind_1.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322502-1304387047/watcher-3322502-1304387047 -o /tmp/evaluation-result-3322502-1304387047/solver-3322502-1304387047 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322502-1304387047.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f35f3aee72237c1bf3d673feb4883756
RANDOM SEED=1393250064

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      18898884 kB
Buffers:        909224 kB
Cached:        2619080 kB
SwapCached:          0 kB
Active:       10536624 kB
Inactive:      3036248 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      18898884 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:            4040 kB
Writeback:           0 kB
AnonPages:    10043660 kB
Mapped:          27332 kB
Slab:           372880 kB
PageTables:      40988 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 69921368 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73572 MiB
End job on node115 at 2011-05-03 05:09:09