Trace number 3321415

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40001.6 5007.03

General information on the benchmark

NameCRAFTED/edgematching/
compact/em_12_2_4_cmp.cnf
MD5SUM7ea97a36d5ab6c8a5aa99830d5253591
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark61.2837
Satisfiable
(Un)Satisfiability was proved
Number of variables12584
Number of clauses155712
Sum of the clauses size523920
Maximum clause length100
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 217404
Number of clauses of size 3103200
Number of clauses of size 417828
Number of clauses of size 517000
Number of clauses of size over 5280

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...339336/instance-3321415-1304339336.cnf
0.18/0.20	c Solving...
40000.78/5006.31	
40000.78/5006.31	*** INTERRUPTED! ***
40000.78/5006.31	*** Shutting down threads...
40001.52/5007.01	s UNKNOWN
40001.52/5007.01	
40001.52/5007.01	c INTERRUPTED   : 1
40001.52/5007.01	c Models        : 0+    
40001.52/5007.01	c Time          : 5006.316s (Solving: 5006.11s 1st Model: 0.00s Unsat: 0.00s)
40001.52/5007.01	c CPU Time      : 40000.840s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321415-1304339336/watcher-3321415-1304339336 -o /tmp/evaluation-result-3321415-1304339336/solver-3321415-1304339336 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321415-1304339336.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.19 8.15 8.00 1/265 11011
/proc/meminfo: memFree=19260480/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=24004 CPUtime=0 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1359 0 0 0 0 0 0 0 20 0 1 0 526349244 24580096 1191 33554432000 4194304 4964704 140733820338160 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11011/statm: 6001 1191 386 189 0 875 0

[startup+0.067364 s]
/proc/loadavg: 8.19 8.15 8.00 1/265 11011
/proc/meminfo: memFree=19260480/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=34636 CPUtime=0.05 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 4764 0 0 0 5 0 0 0 20 0 1 0 526349244 35467264 3730 33554432000 4194304 4964704 140733820338160 18446744073709551615 4615454 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11011/statm: 8659 3731 397 189 0 3533 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 34636

[startup+0.100371 s]
/proc/loadavg: 8.19 8.15 8.00 1/265 11011
/proc/meminfo: memFree=19260480/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=37652 CPUtime=0.08 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 5490 0 0 0 8 0 0 0 20 0 1 0 526349244 38555648 4456 33554432000 4194304 4964704 140733820338160 18446744073709551615 4607554 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11011/statm: 9413 4456 399 189 0 4287 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 37652

[startup+0.301998 s]
/proc/loadavg: 8.19 8.15 8.00 1/265 11011
/proc/meminfo: memFree=19260480/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=463432 CPUtime=0.49 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 12102 0 0 0 45 4 0 0 18 0 8 0 526349244 474554368 10357 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668709 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/11011/statm: 115858 10357 464 189 0 110207 0
Current children cumulated CPU time (s) 0.49
Current children cumulated vsize (KiB) 463432

[startup+0.700208 s]
/proc/loadavg: 8.19 8.15 8.00 1/265 11011
/proc/meminfo: memFree=19260480/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=467008 CPUtime=3.56 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 20362 0 0 0 346 10 0 0 18 0 8 0 526349244 478216192 18617 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320944 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 116752 18618 464 189 0 111101 0
Current children cumulated CPU time (s) 3.56
Current children cumulated vsize (KiB) 467008

[startup+1.50106 s]
/proc/loadavg: 8.19 8.15 8.00 9/273 11019
/proc/meminfo: memFree=19171632/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=472824 CPUtime=9.94 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 31095 0 0 0 977 17 0 0 19 0 8 0 526349244 484171776 29350 33554432000 4194304 4964704 140733820338160 18446744073709551615 4362195 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 118206 29350 464 189 0 112555 0
[pid=11011/tid=11013] ppid=11009 vsize=472824 CPUtime=1.22 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 3970 0 0 0 120 2 0 0 19 0 8 0 526349265 484171776 29350 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=472824 CPUtime=1.23 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 2603 0 0 0 122 1 0 0 19 0 8 0 526349265 484171776 29350 33554432000 4194304 4964704 140733820338160 18446744073709551615 4362864 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=472824 CPUtime=1.2 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 3509 0 0 0 118 2 0 0 19 0 8 0 526349265 484171776 29351 33554432000 4194304 4964704 140733820338160 18446744073709551615 4363438 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=472824 CPUtime=1.2 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 4744 0 0 0 117 3 0 0 19 0 8 0 526349265 484171776 29352 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=472824 CPUtime=1.26 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 1138 0 0 0 126 0 0 0 19 0 8 0 526349265 484171776 29352 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668693 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=472824 CPUtime=1.17 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 3115 0 0 0 116 1 0 0 19 0 8 0 526349265 484171776 29353 33554432000 4194304 4964704 140733820338160 18446744073709551615 4654456 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=472824 CPUtime=1.22 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 3065 0 0 0 121 1 0 0 19 0 8 0 526349265 484171776 29353 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 9.94
Current children cumulated vsize (KiB) 472824

[startup+3.10076 s]
/proc/loadavg: 8.18 8.15 8.00 9/273 11019
/proc/meminfo: memFree=19124760/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=480452 CPUtime=22.71 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 45785 0 0 0 2247 24 0 0 23 0 8 0 526349244 491982848 44040 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 120113 44041 464 189 0 114462 0
[pid=11011/tid=11013] ppid=11009 vsize=480452 CPUtime=2.82 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 6295 0 0 0 278 4 0 0 22 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 264726066209 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=480452 CPUtime=2.82 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 2715 0 0 0 281 1 0 0 23 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=480452 CPUtime=2.8 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 5437 0 0 0 277 3 0 0 22 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=480452 CPUtime=2.81 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 7356 0 0 0 276 5 0 0 22 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=480452 CPUtime=2.86 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 3704 0 0 0 284 2 0 0 24 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=480452 CPUtime=2.77 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 4953 0 0 0 275 2 0 0 22 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 47294234275792 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=480452 CPUtime=2.82 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 3416 0 0 0 281 1 0 0 22 0 8 0 526349265 491982848 44042 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670288 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 22.71
Current children cumulated vsize (KiB) 480452

[startup+6.30115 s]
/proc/loadavg: 8.18 8.15 8.00 9/273 11019
/proc/meminfo: memFree=19003860/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=497676 CPUtime=48.28 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 71960 0 0 0 4791 37 0 0 25 0 8 0 526349244 509620224 70215 33554432000 4194304 4964704 140733820338160 18446744073709551615 4325370 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 124419 70216 466 189 0 118768 0
[pid=11011/tid=11013] ppid=11009 vsize=497676 CPUtime=6.02 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 11402 0 0 0 595 7 0 0 25 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 264726066209 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=497676 CPUtime=6.02 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 2913 0 0 0 601 1 0 0 25 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668648 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=497676 CPUtime=5.97 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 8527 0 0 0 593 4 0 0 25 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=497676 CPUtime=6 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 11207 0 0 0 594 6 0 0 25 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=497676 CPUtime=6.06 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 8113 0 0 0 603 3 0 0 25 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668907 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=497676 CPUtime=5.97 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 8124 0 0 0 593 4 0 0 19 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=497676 CPUtime=6.01 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 3804 0 0 0 600 1 0 0 25 0 8 0 526349265 509620224 70216 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668676 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 48.28
Current children cumulated vsize (KiB) 497676

[startup+12.7009 s]
/proc/loadavg: 8.15 8.14 8.00 9/273 11019
/proc/meminfo: memFree=18934692/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=500048 CPUtime=99.39 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 86281 0 0 0 9894 45 0 0 25 0 8 0 526349244 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4321079 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 125012 84536 466 189 0 119361 0
[pid=11011/tid=11013] ppid=11009 vsize=500048 CPUtime=12.41 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 14111 0 0 0 1233 8 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=500048 CPUtime=12.43 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 3130 0 0 0 1241 2 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=500048 CPUtime=12.31 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 10093 0 0 0 1226 5 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=500048 CPUtime=12.4 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 12771 0 0 0 1232 8 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=500048 CPUtime=12.45 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 10277 0 0 0 1241 4 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4654508 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=500048 CPUtime=12.37 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 10548 0 0 0 1232 5 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668907 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=500048 CPUtime=12.42 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 3909 0 0 0 1240 2 0 0 25 0 8 0 526349265 512049152 84536 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.39
Current children cumulated vsize (KiB) 500048

[startup+25.5005 s]
/proc/loadavg: 8.13 8.14 8.00 9/273 11019
/proc/meminfo: memFree=18822428/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=584892 CPUtime=201.66 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 112783 0 0 0 20107 59 0 0 25 0 8 0 526349244 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4321010 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 146223 111027 466 189 0 140572 0
[pid=11011/tid=11013] ppid=11009 vsize=584892 CPUtime=25.21 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 17036 0 0 0 2511 10 0 0 25 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=584892 CPUtime=25.22 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 3584 0 0 0 2520 2 0 0 25 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=584892 CPUtime=24.99 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 14985 0 0 0 2491 8 0 0 22 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=584892 CPUtime=25.2 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 16386 0 0 0 2510 10 0 0 25 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4631193 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=584892 CPUtime=25.24 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 14116 0 0 0 2518 6 0 0 25 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=584892 CPUtime=25.17 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 14245 0 0 0 2509 8 0 0 25 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4669568 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=584892 CPUtime=25.22 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 4408 0 0 0 2520 2 0 0 25 0 8 0 526349265 598929408 111027 33554432000 4194304 4964704 140733820338160 18446744073709551615 4660304 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 201.66
Current children cumulated vsize (KiB) 584892

[startup+51.1007 s]
/proc/loadavg: 8.08 8.13 8.00 9/273 11020
/proc/meminfo: memFree=18711696/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=935988 CPUtime=406.2 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 140376 0 0 0 40545 75 0 0 25 0 8 0 526349244 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11011/statm: 233997 138615 466 189 0 228346 0
[pid=11011/tid=11013] ppid=11009 vsize=935988 CPUtime=50.79 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 22285 0 0 0 5066 13 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=935988 CPUtime=50.82 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 4211 0 0 0 5080 2 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=935988 CPUtime=50.38 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 17156 0 0 0 5028 10 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=935988 CPUtime=50.8 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 21172 0 0 0 5068 12 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=935988 CPUtime=50.82 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 18690 0 0 0 5074 8 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=935988 CPUtime=50.77 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 16024 0 0 0 5067 10 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=935988 CPUtime=50.81 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 5027 0 0 0 5079 2 0 0 25 0 8 0 526349265 958451712 138615 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.2

################
# More data... #
################

/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 155963 0 0 0 465902 149 0 0 25 0 8 0 526349265 4553134080 1018994 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4446420 CPUtime=4650.5 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 58357 0 0 0 464991 59 0 0 25 0 8 0 526349265 4553134080 1018994 33554432000 4194304 4964704 140733820338160 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37252.1
Current children cumulated vsize (KiB) 4446420

[startup+4722.31 s]
/proc/loadavg: 8.06 8.05 8.00 9/273 11157
/proc/meminfo: memFree=15180016/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4446420 CPUtime=37731.6 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1032135 0 0 0 3772194 965 0 0 25 0 8 0 526349244 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11011/statm: 1111605 1019265 466 189 0 1105954 0
[pid=11011/tid=11013] ppid=11009 vsize=4446420 CPUtime=4710.73 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 82318 0 0 0 470985 88 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4653233 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=4446420 CPUtime=4720.59 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 41525 0 0 0 472015 44 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=4446420 CPUtime=4718.58 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 141144 0 0 0 471712 146 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=4446420 CPUtime=4721.48 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 118825 0 0 0 472036 112 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320961 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=4446420 CPUtime=4720.75 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 92767 0 0 0 471999 76 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=4446420 CPUtime=4720.5 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 156162 0 0 0 471901 149 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4446420 CPUtime=4710.51 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 58357 0 0 0 470992 59 0 0 25 0 8 0 526349265 4553134080 1019265 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37731.6
Current children cumulated vsize (KiB) 4446420

[startup+4782.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/273 11158
/proc/meminfo: memFree=15099236/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4577492 CPUtime=38211 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1053145 0 0 0 3820110 990 0 0 25 0 8 0 526349244 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4332507 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11011/statm: 1144373 1040275 466 189 0 1138722 0
[pid=11011/tid=11013] ppid=11009 vsize=4577492 CPUtime=4770.72 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 83300 0 0 0 476982 90 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4668512 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=4577492 CPUtime=4780.57 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 41771 0 0 0 478013 44 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=4577492 CPUtime=4778.56 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 147838 0 0 0 477702 154 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=4577492 CPUtime=4781.47 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 118988 0 0 0 478035 112 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=4577492 CPUtime=4780.73 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 93888 0 0 0 477997 76 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=4577492 CPUtime=4780.5 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 156291 0 0 0 477900 150 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4577492 CPUtime=4770.5 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 64807 0 0 0 476983 67 0 0 25 0 8 0 526349265 4687351808 1040275 33554432000 4194304 4964704 140733820338160 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38211
Current children cumulated vsize (KiB) 4577492

[startup+4842.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/273 11160
/proc/meminfo: memFree=15074252/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4577492 CPUtime=38690.5 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1058935 0 0 0 3868053 996 0 0 25 0 8 0 526349244 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11011/statm: 1144373 1046065 466 189 0 1138722 0
[pid=11011/tid=11013] ppid=11009 vsize=4577492 CPUtime=4830.72 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 83352 0 0 0 482982 90 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=4577492 CPUtime=4840.58 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 41771 0 0 0 484014 44 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=4577492 CPUtime=4838.54 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 151137 0 0 0 483695 159 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=4577492 CPUtime=4841.48 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 119000 0 0 0 484036 112 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=4577492 CPUtime=4840.72 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 95558 0 0 0 483996 76 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=4577492 CPUtime=4840.49 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 156403 0 0 0 483899 150 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4577492 CPUtime=4830.5 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 64895 0 0 0 482983 67 0 0 25 0 8 0 526349265 4687351808 1046065 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38690.5
Current children cumulated vsize (KiB) 4577492

[startup+4902.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/273 11161
/proc/meminfo: memFree=14935820/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4664292 CPUtime=39170 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1093509 0 0 0 3915969 1029 0 0 25 0 8 0 526349244 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11011/statm: 1166073 1080639 466 189 0 1160422 0
[pid=11011/tid=11013] ppid=11009 vsize=4664292 CPUtime=4890.73 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 85943 0 0 0 488978 95 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4660034 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=4664292 CPUtime=4900.58 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 42923 0 0 0 490012 46 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320921 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=4664292 CPUtime=4898.53 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 151768 0 0 0 489693 160 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670241 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=4664292 CPUtime=4901.48 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 119399 0 0 0 490035 113 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=4664292 CPUtime=4900.71 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 97876 0 0 0 489994 77 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 264726102626 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=4664292 CPUtime=4900.48 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 158661 0 0 0 489897 151 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4664292 CPUtime=4890.49 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 82875 0 0 0 488967 82 0 0 25 0 8 0 526349265 4776235008 1080639 33554432000 4194304 4964704 140733820338160 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39170
Current children cumulated vsize (KiB) 4664292

[startup+4962.3 s]
/proc/loadavg: 8.11 8.05 8.01 9/273 11163
/proc/meminfo: memFree=14934756/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4665272 CPUtime=39649.2 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1093736 0 0 0 3963883 1033 0 0 25 0 8 0 526349244 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320993 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/11011/statm: 1166318 1080789 466 189 0 1160667 0
[pid=11011/tid=11013] ppid=11009 vsize=4665272 CPUtime=4950.33 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 85946 0 0 0 494938 95 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=4665272 CPUtime=4960.34 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 42923 0 0 0 495988 46 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4379082 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=4665272 CPUtime=4958.53 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 151773 0 0 0 495693 160 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=4665272 CPUtime=4961.41 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 119399 0 0 0 496028 113 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=4665272 CPUtime=4960.67 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 98062 0 0 0 495989 78 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=4665272 CPUtime=4960.48 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 158694 0 0 0 495897 151 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4665272 CPUtime=4950.49 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 82875 0 0 0 494966 83 0 0 25 0 8 0 526349265 4777238528 1080789 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39649.2
Current children cumulated vsize (KiB) 4665272



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.3 s]
/proc/loadavg: 8.05 8.04 8.00 9/273 11164
/proc/meminfo: memFree=14934828/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4665272 CPUtime=40000.8 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1093755 0 0 0 3999044 1034 0 0 25 0 8 0 526349244 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/11011/statm: 1166318 1080664 466 189 0 1160667 0
[pid=11011/tid=11013] ppid=11009 vsize=4665272 CPUtime=4994.34 cores=0-7
/proc/11011/task/11013/stat : 11013 (claspmt) R 11009 11011 10846 0 -1 4202560 85946 0 0 0 499339 95 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11011/tid=11014] ppid=11009 vsize=4665272 CPUtime=5003.99 cores=0-7
/proc/11011/task/11014/stat : 11014 (claspmt) R 11009 11011 10846 0 -1 4202560 42923 0 0 0 500352 47 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11011/tid=11015] ppid=11009 vsize=4665272 CPUtime=5002.5 cores=0-7
/proc/11011/task/11015/stat : 11015 (claspmt) R 11009 11011 10846 0 -1 4202560 151773 0 0 0 500089 161 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11011/tid=11016] ppid=11009 vsize=4665272 CPUtime=5005.42 cores=0-7
/proc/11011/task/11016/stat : 11016 (claspmt) R 11009 11011 10846 0 -1 4202560 119400 0 0 0 500429 113 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11011/tid=11017] ppid=11009 vsize=4665272 CPUtime=5004.68 cores=0-7
/proc/11011/task/11017/stat : 11017 (claspmt) R 11009 11011 10846 0 -1 4202560 98062 0 0 0 500390 78 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11011/tid=11018] ppid=11009 vsize=4665272 CPUtime=5004.46 cores=0-7
/proc/11011/task/11018/stat : 11018 (claspmt) R 11009 11011 10846 0 -1 4202560 158706 0 0 0 500294 152 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11011/tid=11019] ppid=11009 vsize=4665272 CPUtime=4994.5 cores=0-7
/proc/11011/task/11019/stat : 11019 (claspmt) R 11009 11011 10846 0 -1 4202560 82875 0 0 0 499367 83 0 0 25 0 8 0 526349265 4777238528 1080664 33554432000 4194304 4964704 140733820338160 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 4665272

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5006.6 s]
/proc/loadavg: 8.05 8.04 8.00 9/273 11164
/proc/meminfo: memFree=14934828/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=4658224 CPUtime=40001.1 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1093766 0 0 0 3999078 1034 0 0 25 0 1 0 526349244 4770021376 1078172 33554432000 4194304 4964704 140733820338160 18446744073709551615 4646575 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/11011/statm: 1164556 1078172 476 189 0 1158905 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 4658224

[startup+5007 s]
/proc/loadavg: 8.05 8.04 8.00 9/273 11164
/proc/meminfo: memFree=14934828/32951124 swapFree=67111524/67111528
[pid=11011] ppid=11009 vsize=656876 CPUtime=40001.5 cores=0-7
/proc/11011/stat : 11011 (claspmt) R 11009 11011 10846 0 -1 4202496 1093766 0 0 0 3999100 1052 0 0 25 0 1 0 526349244 672641024 93976 33554432000 4194304 4964704 140733820338160 18446744073709551615 264726448487 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/11011/statm: 164219 93976 476 189 0 158568 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 656876

Child status: 0
Real time (s): 5007.03
CPU time (s): 40001.6
CPU user time (s): 39991
CPU system time (s): 10.5454
CPU usage (%): 798.908
Max. virtual memory (cumulated for all children) (KiB): 4665272

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39991
system time used= 10.5454
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1093767
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 213795
involuntary context switches= 189485

runsolver used 10.8284 second user time and 30.1714 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-02 14:28:56
IDJOB=3321415
IDBENCH=70908
IDSOLVER=1711
FILE ID=node141/3321415-1304339336
RUNJOBID= node141-1304337558-10860
PBS_JOBID= 13178040
Free space on /tmp= 73004 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT09/CRAFTED/edgematching/compact/em_12_2_4_cmp.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321415-1304339336/watcher-3321415-1304339336 -o /tmp/evaluation-result-3321415-1304339336/solver-3321415-1304339336 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321415-1304339336.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 7ea97a36d5ab6c8a5aa99830d5253591
RANDOM SEED=1732318888

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19260876 kB
Buffers:       1429440 kB
Cached:        8728692 kB
SwapCached:          4 kB
Active:        5975108 kB
Inactive:      7111296 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19260876 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            4500 kB
Writeback:           0 kB
AnonPages:     2928952 kB
Mapped:          22008 kB
Slab:           518008 kB
PageTables:      21284 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 44521768 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73000 MiB
End job on node141 at 2011-05-02 15:52:25