Trace number 3381101

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.7 5007.13

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_3_keyfind_1.cnf
MD5SUMff59ad0e88befc51bfa041e45143dc53
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2808
Number of clauses30432
Sum of the clauses size167020
Maximum clause length8
Minimum clause length1
Number of clauses of size 1280
Number of clauses of size 2384
Number of clauses of size 35024
Number of clauses of size 4960
Number of clauses of size 56144
Number of clauses of size over 517640

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...504174/instance-3381101-1305504174.cnf
0.02/0.07	c Solving...
40000.43/5006.91	
40000.43/5006.91	*** INTERRUPTED! ***
40000.43/5006.91	*** Shutting down threads...
40000.68/5007.12	s UNKNOWN
40000.68/5007.12	
40000.68/5007.12	c INTERRUPTED   : 1
40000.68/5007.12	c Models        : 0+    
40000.68/5007.12	c Time          : 5006.915s (Solving: 5006.84s 1st Model: 0.00s Unsat: 0.00s)
40000.68/5007.12	c CPU Time      : 40000.480s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381101-1305504174/watcher-3381101-1305504174 -o /tmp/evaluation-result-3381101-1305504174/solver-3381101-1305504174 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381101-1305504174.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 8.05 8.01 2/295 15228
/proc/meminfo: memFree=26126820/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=22352 CPUtime=0 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 910 0 0 0 0 0 0 0 24 0 1 0 740648018 22888448 820 33554432000 4194304 4964800 140734797360944 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/15228/statm: 5588 820 389 189 0 462 0

[startup+0.0233429 s]
/proc/loadavg: 8.11 8.05 8.01 2/295 15228
/proc/meminfo: memFree=26126820/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=24308 CPUtime=0.02 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 1472 0 0 0 2 0 0 0 24 0 1 0 740648018 24891392 1266 33554432000 4194304 4964800 140734797360944 18446744073709551615 4615951 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/15228/statm: 6077 1266 399 189 0 951 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 24308

[startup+0.100342 s]
/proc/loadavg: 8.11 8.05 8.01 2/295 15228
/proc/meminfo: memFree=26126820/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=452592 CPUtime=0.11 cores=0-7
/proc/15228/stat : 15228 (claspmt) S 15226 15228 15045 0 -1 4202496 5059 0 0 0 10 1 0 0 19 0 8 0 740648018 463454208 4663 33554432000 4194304 4964800 140734797360944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/15228/statm: 113148 4663 438 189 0 107497 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 452592

[startup+0.30025 s]
/proc/loadavg: 8.11 8.05 8.01 2/295 15228
/proc/meminfo: memFree=26126820/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=453452 CPUtime=1.64 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 7726 0 0 0 160 4 0 0 20 0 8 0 740648018 464334848 7329 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670282 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 113363 7330 464 189 0 107712 0
Current children cumulated CPU time (s) 1.64
Current children cumulated vsize (KiB) 453452

[startup+0.700178 s]
/proc/loadavg: 8.11 8.05 8.01 2/295 15228
/proc/meminfo: memFree=26126820/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=454640 CPUtime=4.85 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 9989 0 0 0 479 6 0 0 20 0 8 0 740648018 465551360 9592 33554432000 4194304 4964800 140734797360944 18446744073709551615 4338784 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 113660 9592 464 189 0 108009 0
Current children cumulated CPU time (s) 4.85
Current children cumulated vsize (KiB) 454640

[startup+1.50103 s]
/proc/loadavg: 8.11 8.05 8.01 9/303 15236
/proc/meminfo: memFree=26086580/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=456372 CPUtime=11.22 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 12785 0 0 0 1115 7 0 0 22 0 8 0 740648018 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 4364717 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 114093 12388 464 189 0 108442 0
[pid=15228/tid=15230] ppid=15226 vsize=456372 CPUtime=1.39 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 899 0 0 0 139 0 0 0 18 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 4684805 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=456372 CPUtime=1.4 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) S 15226 15228 15045 0 -1 4202560 1734 0 0 0 139 1 0 0 19 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 260020500974 0 0 4096 18946 18446604452120543296 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=456372 CPUtime=1.38 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 1474 0 0 0 138 0 0 0 19 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=456372 CPUtime=1.39 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 1193 0 0 0 138 1 0 0 20 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 4321069 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=456372 CPUtime=1.38 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 1422 0 0 0 137 1 0 0 20 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 260020051147 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=456372 CPUtime=1.38 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 1579 0 0 0 138 0 0 0 19 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 4342228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=456372 CPUtime=1.39 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 1897 0 0 0 139 0 0 0 19 0 8 0 740648026 467324928 12388 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.22
Current children cumulated vsize (KiB) 456372

[startup+3.10074 s]
/proc/loadavg: 8.11 8.05 8.01 9/303 15236
/proc/meminfo: memFree=26072752/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=459124 CPUtime=24 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 16348 0 0 0 2391 9 0 0 25 0 8 0 740648018 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 114781 15951 466 189 0 109130 0
[pid=15228/tid=15230] ppid=15226 vsize=459124 CPUtime=2.99 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 1145 0 0 0 299 0 0 0 20 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4660303 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=459124 CPUtime=2.99 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 2339 0 0 0 298 1 0 0 23 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=459124 CPUtime=2.98 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 1840 0 0 0 298 0 0 0 23 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4627519 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=459124 CPUtime=2.99 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 1533 0 0 0 298 1 0 0 18 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=459124 CPUtime=2.97 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 1970 0 0 0 296 1 0 0 25 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=459124 CPUtime=2.98 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 2063 0 0 0 297 1 0 0 24 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4325338 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=459124 CPUtime=3 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 2487 0 0 0 299 1 0 0 25 0 8 0 740648026 470142976 15951 33554432000 4194304 4964800 140734797360944 18446744073709551615 4627606 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24
Current children cumulated vsize (KiB) 459124

[startup+6.30115 s]
/proc/loadavg: 8.10 8.05 8.00 9/303 15236
/proc/meminfo: memFree=26050392/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=461368 CPUtime=49.56 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 21038 0 0 0 4943 13 0 0 25 0 8 0 740648018 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4364587 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 115342 20520 466 189 0 109691 0
[pid=15228/tid=15230] ppid=15226 vsize=461368 CPUtime=6.19 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 1615 0 0 0 618 1 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=461368 CPUtime=6.19 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 2987 0 0 0 618 1 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=461368 CPUtime=6.18 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 2302 0 0 0 617 1 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670313 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=461368 CPUtime=6.19 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 2055 0 0 0 617 2 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320921 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=461368 CPUtime=6.14 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 2680 0 0 0 613 1 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4669272 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=461368 CPUtime=6.17 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 2526 0 0 0 616 1 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4669184 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=461368 CPUtime=6.19 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 3223 0 0 0 618 1 0 0 25 0 8 0 740648026 472440832 20520 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 49.56
Current children cumulated vsize (KiB) 461368

[startup+12.701 s]
/proc/loadavg: 8.09 8.05 8.00 9/303 15237
/proc/meminfo: memFree=26018572/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=465084 CPUtime=100.67 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 28671 0 0 0 10049 18 0 0 25 0 8 0 740648018 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4654540 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 116271 27972 466 189 0 110620 0
[pid=15228/tid=15230] ppid=15226 vsize=465084 CPUtime=12.58 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 2274 0 0 0 1257 1 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4632278 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=465084 CPUtime=12.6 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 3791 0 0 0 1258 2 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=465084 CPUtime=12.58 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 2890 0 0 0 1256 2 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320902 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=465084 CPUtime=12.58 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 2915 0 0 0 1256 2 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=465084 CPUtime=12.48 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 3541 0 0 0 1246 2 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4364621 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=465084 CPUtime=12.57 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 3477 0 0 0 1255 2 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4363459 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=465084 CPUtime=12.59 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 5060 0 0 0 1257 2 0 0 25 0 8 0 740648026 476246016 27972 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 100.67
Current children cumulated vsize (KiB) 465084

[startup+25.5006 s]
/proc/loadavg: 8.07 8.05 8.00 9/303 15237
/proc/meminfo: memFree=25981036/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=471048 CPUtime=202.93 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 38556 0 0 0 20268 25 0 0 25 0 8 0 740648018 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670407 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 117762 37850 466 189 0 112111 0
[pid=15228/tid=15230] ppid=15226 vsize=471048 CPUtime=25.38 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 3464 0 0 0 2536 2 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4659391 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=471048 CPUtime=25.39 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 4638 0 0 0 2537 2 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=471048 CPUtime=25.38 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 3784 0 0 0 2535 3 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=471048 CPUtime=25.38 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 4260 0 0 0 2535 3 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=471048 CPUtime=25.15 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 4999 0 0 0 2512 3 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=471048 CPUtime=25.36 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 4463 0 0 0 2534 2 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=471048 CPUtime=25.39 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 6771 0 0 0 2535 4 0 0 25 0 8 0 740648026 482353152 37850 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670434 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 202.93
Current children cumulated vsize (KiB) 471048

[startup+51.1009 s]
/proc/loadavg: 8.04 8.04 8.00 9/303 15238
/proc/meminfo: memFree=25909064/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=480944 CPUtime=407.45 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 56490 0 0 0 40707 38 0 0 25 0 8 0 740648018 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/15228/statm: 120236 55680 466 189 0 114585 0
[pid=15228/tid=15230] ppid=15226 vsize=480944 CPUtime=50.97 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 5293 0 0 0 5093 4 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=480944 CPUtime=50.98 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 5644 0 0 0 5095 3 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 260020054936 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=480944 CPUtime=50.97 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 5054 0 0 0 5093 4 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320890 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=480944 CPUtime=50.98 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 7619 0 0 0 5093 5 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670640 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=480944 CPUtime=50.65 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 6880 0 0 0 5060 5 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 260020051067 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=480944 CPUtime=50.95 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 6749 0 0 0 5091 4 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 4627208 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=480944 CPUtime=50.85 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 10365 0 0 0 5079 6 0 0 25 0 8 0 740648026 492486656 55680 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.45

################
# More data... #
################

/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 40570 0 0 0 466032 56 0 0 25 0 8 0 740648026 1456484352 300571 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1422348 CPUtime=4655.17 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49667 0 0 0 465443 74 0 0 25 0 8 0 740648026 1456484352 300571 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37247.2
Current children cumulated vsize (KiB) 1422348

[startup+4722.31 s]
/proc/loadavg: 8.02 8.06 8.00 9/303 15374
/proc/meminfo: memFree=24918952/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1422348 CPUtime=37726.6 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 311421 0 0 0 3772175 489 0 0 25 0 8 0 740648018 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670346 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/15228/statm: 355587 301948 466 189 0 349936 0
[pid=15228/tid=15230] ppid=15226 vsize=1422348 CPUtime=4702.51 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 37173 0 0 0 470180 71 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=1422348 CPUtime=4721.37 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 28571 0 0 0 472095 42 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=1422348 CPUtime=4721.69 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 33648 0 0 0 472118 51 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=1422348 CPUtime=4721.74 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 35778 0 0 0 472125 49 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=1422348 CPUtime=4717 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 35566 0 0 0 471637 63 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=1422348 CPUtime=4720.88 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 40841 0 0 0 472032 56 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1422348 CPUtime=4715.17 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49694 0 0 0 471443 74 0 0 25 0 8 0 740648026 1456484352 301948 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37726.6
Current children cumulated vsize (KiB) 1422348

[startup+4782.3 s]
/proc/loadavg: 8.01 8.05 8.00 9/303 15376
/proc/meminfo: memFree=24910968/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1423408 CPUtime=38205.9 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 313652 0 0 0 3820101 493 0 0 25 0 8 0 740648018 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/15228/statm: 355852 304100 466 189 0 350201 0
[pid=15228/tid=15230] ppid=15226 vsize=1423408 CPUtime=4761.92 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 37502 0 0 0 476120 72 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4644896 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=1423408 CPUtime=4781.35 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 28652 0 0 0 478093 42 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=1423408 CPUtime=4781.67 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 33787 0 0 0 478116 51 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=1423408 CPUtime=4781.72 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 35978 0 0 0 478123 49 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670321 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=1423408 CPUtime=4776.98 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 36595 0 0 0 477634 64 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=1423408 CPUtime=4780.84 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 40871 0 0 0 478028 56 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1423408 CPUtime=4775.15 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49819 0 0 0 477441 74 0 0 25 0 8 0 740648026 1457569792 304100 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38205.9
Current children cumulated vsize (KiB) 1423408

[startup+4842.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/303 15378
/proc/meminfo: memFree=24908184/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1423892 CPUtime=38685.3 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 314325 0 0 0 3868035 495 0 0 25 0 8 0 740648018 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/15228/statm: 355973 304660 466 189 0 350322 0
[pid=15228/tid=15230] ppid=15226 vsize=1423892 CPUtime=4821.33 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 37595 0 0 0 482061 72 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=1423892 CPUtime=4841.35 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 28771 0 0 0 484093 42 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320905 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=1423892 CPUtime=4841.67 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 33838 0 0 0 484116 51 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=1423892 CPUtime=4841.72 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 36028 0 0 0 484123 49 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=1423892 CPUtime=4836.97 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 36596 0 0 0 483633 64 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=1423892 CPUtime=4840.84 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 41021 0 0 0 484027 57 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1423892 CPUtime=4835.15 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49897 0 0 0 483440 75 0 0 25 0 8 0 740648026 1458065408 304660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38685.3
Current children cumulated vsize (KiB) 1423892

[startup+4902.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/303 15379
/proc/meminfo: memFree=24906776/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1424176 CPUtime=39164.7 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 314691 0 0 0 3915972 497 0 0 25 0 8 0 740648018 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/15228/statm: 356044 304989 466 189 0 350393 0
[pid=15228/tid=15230] ppid=15226 vsize=1424176 CPUtime=4881.21 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 37686 0 0 0 488048 73 0 0 25 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=1424176 CPUtime=4901.34 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 28890 0 0 0 490091 43 0 0 25 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4668794 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=1424176 CPUtime=4901.67 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 33840 0 0 0 490115 52 0 0 25 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=1424176 CPUtime=4901.72 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 36028 0 0 0 490123 49 0 0 25 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=1424176 CPUtime=4896.51 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 36639 0 0 0 489586 65 0 0 19 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=1424176 CPUtime=4900.83 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 41061 0 0 0 490026 57 0 0 25 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1424176 CPUtime=4895.15 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49924 0 0 0 489440 75 0 0 25 0 8 0 740648026 1458356224 304989 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39164.7
Current children cumulated vsize (KiB) 1424176

[startup+4962.3 s]
/proc/loadavg: 8.10 8.05 8.00 9/303 15381
/proc/meminfo: memFree=24904104/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1425336 CPUtime=39644 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 315427 0 0 0 3963903 499 0 0 25 0 8 0 740648018 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/15228/statm: 356334 305672 466 189 0 350683 0
[pid=15228/tid=15230] ppid=15226 vsize=1425336 CPUtime=4940.6 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 37797 0 0 0 493987 73 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=1425336 CPUtime=4961.33 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 28890 0 0 0 496090 43 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=1425336 CPUtime=4961.65 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 33861 0 0 0 496113 52 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 260020086844 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=1425336 CPUtime=4961.72 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 36274 0 0 0 496122 50 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=1425336 CPUtime=4956.51 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 36710 0 0 0 495586 65 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=1425336 CPUtime=4960.81 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 41087 0 0 0 496024 57 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1425336 CPUtime=4955.14 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49925 0 0 0 495439 75 0 0 25 0 8 0 740648026 1459544064 305672 33554432000 4194304 4964800 140734797360944 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39644
Current children cumulated vsize (KiB) 1425336



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.9 s]
/proc/loadavg: 8.05 8.04 8.00 9/303 15382
/proc/meminfo: memFree=24900480/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1425804 CPUtime=40000.4 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 316510 0 0 0 3999543 500 0 0 25 0 8 0 740648018 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4684040 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/15228/statm: 356451 306660 466 189 0 350800 0
[pid=15228/tid=15230] ppid=15226 vsize=1425804 CPUtime=4984.78 cores=0-7
/proc/15228/task/15230/stat : 15230 (claspmt) R 15226 15228 15045 0 -1 4202560 38069 0 0 0 498404 74 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15228/tid=15231] ppid=15226 vsize=1425804 CPUtime=5005.93 cores=0-7
/proc/15228/task/15231/stat : 15231 (claspmt) R 15226 15228 15045 0 -1 4202560 29083 0 0 0 500550 43 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15228/tid=15232] ppid=15226 vsize=1425804 CPUtime=5006.26 cores=0-7
/proc/15228/task/15232/stat : 15232 (claspmt) R 15226 15228 15045 0 -1 4202560 33864 0 0 0 500574 52 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15228/tid=15233] ppid=15226 vsize=1425804 CPUtime=5006.33 cores=0-7
/proc/15228/task/15233/stat : 15233 (claspmt) R 15226 15228 15045 0 -1 4202560 36317 0 0 0 500583 50 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15228/tid=15234] ppid=15226 vsize=1425804 CPUtime=5001.11 cores=0-7
/proc/15228/task/15234/stat : 15234 (claspmt) R 15226 15228 15045 0 -1 4202560 37022 0 0 0 500046 65 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4342213 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15228/tid=15235] ppid=15226 vsize=1425804 CPUtime=5005.41 cores=0-7
/proc/15228/task/15235/stat : 15235 (claspmt) R 15226 15228 15045 0 -1 4202560 41093 0 0 0 500484 57 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15228/tid=15236] ppid=15226 vsize=1425804 CPUtime=4999.75 cores=0-7
/proc/15228/task/15236/stat : 15236 (claspmt) R 15226 15228 15045 0 -1 4202560 49925 0 0 0 499900 75 0 0 25 0 8 0 740648026 1460023296 306660 33554432000 4194304 4964800 140734797360944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 1425804

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.02 s]
/proc/loadavg: 8.05 8.04 8.00 9/303 15382
/proc/meminfo: memFree=24900480/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=1422200 CPUtime=40000.6 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 316520 0 0 0 3999558 500 0 0 25 0 1 0 740648018 1456332800 305871 33554432000 4194304 4964800 140734797360944 18446744073709551615 4313376 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/15228/statm: 355550 305871 476 189 0 349899 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 1422200

[startup+5007.1 s]
/proc/loadavg: 8.05 8.04 8.00 2/297 15383
/proc/meminfo: memFree=25715608/32951124 swapFree=67083432/67111528
[pid=15228] ppid=15226 vsize=631500 CPUtime=40000.7 cores=0-7
/proc/15228/stat : 15228 (claspmt) R 15226 15228 15045 0 -1 4202496 316520 0 0 0 3999562 506 0 0 25 0 1 0 740648018 646656000 55642 33554432000 4194304 4964800 140734797360944 18446744073709551615 260020439399 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/15228/statm: 157875 55459 476 189 0 152224 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 631500

Child status: 0
Real time (s): 5007.13
CPU time (s): 40000.7
CPU user time (s): 39995.6
CPU system time (s): 5.07623
CPU usage (%): 798.875
Max. virtual memory (cumulated for all children) (KiB): 1425804

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.6
system time used= 5.07623
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 316521
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 295374
involuntary context switches= 190524

runsolver used 13.241 second user time and 33.7309 second system time

The end

Launcher Data 

Begin job on node110 at 2011-05-16 02:02:54
IDJOB=3381101
IDBENCH=83408
IDSOLVER=1865
FILE ID=node110/3381101-1305504174
RUNJOBID= node110-1305503705-15059
PBS_JOBID= 13325472
Free space on /tmp= 73612 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_3_keyfind_1.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381101-1305504174/watcher-3381101-1305504174 -o /tmp/evaluation-result-3381101-1305504174/solver-3381101-1305504174 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381101-1305504174.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ff59ad0e88befc51bfa041e45143dc53
RANDOM SEED=1598860569

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      26127216 kB
Buffers:        267600 kB
Cached:        2827132 kB
SwapCached:       1416 kB
Active:        3496028 kB
Inactive:      3002868 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      26127216 kB
SwapTotal:    67111528 kB
SwapFree:     67083432 kB
Dirty:            1976 kB
Writeback:           0 kB
AnonPages:     3403048 kB
Mapped:          26336 kB
Slab:           236012 kB
PageTables:      23684 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 63399336 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73612 MiB
End job on node110 at 2011-05-16 03:26:23