Trace number 3381385

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.4 5011.98

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m47-q841.5-n45-p15-s16393788.cnf
MD5SUM683c9f75b69575e6451b8563d12f176b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.196969
Satisfiable
(Un)Satisfiability was proved
Number of variables1981
Number of clauses333838
Sum of the clauses size669567
Maximum clause length47
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2333793
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...514567/instance-3381385-1305514567.cnf
1.89/1.97	c Solving...
40000.37/5011.02	
40000.37/5011.02	*** INTERRUPTED! ***
40000.37/5011.02	*** Shutting down threads...
40001.30/5011.94	s UNKNOWN
40001.30/5011.94	
40001.30/5011.94	c INTERRUPTED   : 1
40001.30/5011.94	c Models        : 0+    
40001.30/5011.94	c Time          : 5011.041s (Solving: 5009.07s 1st Model: 0.00s Unsat: 0.00s)
40001.30/5011.94	c CPU Time      : 40000.440s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381385-1305514567/watcher-3381385-1305514567 -o /tmp/evaluation-result-3381385-1305514567/solver-3381385-1305514567 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381385-1305514567.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 7.65 6.48 2/418 30008
/proc/meminfo: memFree=13538652/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=23416 CPUtime=0.01 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1322 0 0 0 1 0 0 0 25 0 1 0 741687361 23977984 1067 33554432000 4194304 4964800 140737384317280 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30008/statm: 5854 1067 386 189 0 728 0

[startup+0.0756029 s]
/proc/loadavg: 8.11 7.65 6.48 2/418 30008
/proc/meminfo: memFree=13538652/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=35808 CPUtime=0.07 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 5777 0 0 0 7 0 0 0 25 0 1 0 741687361 36667392 3917 33554432000 4194304 4964800 140737384317280 18446744073709551615 4617797 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30008/statm: 8952 3917 397 189 0 3826 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 35808

[startup+0.100595 s]
/proc/loadavg: 8.11 7.65 6.48 2/418 30008
/proc/meminfo: memFree=13538652/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=41020 CPUtime=0.09 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 7022 0 0 0 9 0 0 0 25 0 1 0 741687361 42004480 5162 33554432000 4194304 4964800 140737384317280 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30008/statm: 10255 5162 397 189 0 5129 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 41020

[startup+0.300564 s]
/proc/loadavg: 8.11 7.65 6.48 2/418 30008
/proc/meminfo: memFree=13538652/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=43532 CPUtime=0.29 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 7577 0 0 0 29 0 0 0 25 0 1 0 741687361 44576768 5717 33554432000 4194304 4964800 140737384317280 18446744073709551615 4607562 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30008/statm: 10883 5717 399 189 0 5757 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 43532

[startup+0.700493 s]
/proc/loadavg: 8.11 7.65 6.48 2/418 30008
/proc/meminfo: memFree=13538652/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=43532 CPUtime=0.69 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 7577 0 0 0 69 0 0 0 25 0 1 0 741687361 44576768 5717 33554432000 4194304 4964800 140737384317280 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30008/statm: 10883 5717 399 189 0 5757 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 43532

[startup+1.50035 s]
/proc/loadavg: 7.54 7.54 6.45 2/419 30009
/proc/meminfo: memFree=13517200/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=43532 CPUtime=1.49 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 7578 0 0 0 149 0 0 0 25 0 1 0 741687361 44576768 5718 33554432000 4194304 4964800 140737384317280 18446744073709551615 4600372 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30008/statm: 10883 5718 400 189 0 5757 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 43532

[startup+3.10104 s]
/proc/loadavg: 7.54 7.54 6.45 2/426 30016
/proc/meminfo: memFree=13513796/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=466580 CPUtime=9.86 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 21831 0 0 0 976 10 0 0 17 0 8 0 741687361 477777920 19317 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/30008/statm: 116645 19317 464 189 0 110994 0
[pid=30008/tid=30010] ppid=30006 vsize=466580 CPUtime=0.97 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 1639 0 0 0 97 0 0 0 17 0 8 0 741687558 477777920 19318 33554432000 4194304 4964800 140737384317280 18446744073709551615 4334248 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=466580 CPUtime=0.98 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 1531 0 0 0 97 1 0 0 17 0 8 0 741687558 477777920 19318 33554432000 4194304 4964800 140737384317280 18446744073709551615 4627951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=466580 CPUtime=0.98 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 1547 0 0 0 97 1 0 0 20 0 8 0 741687558 477777920 19318 33554432000 4194304 4964800 140737384317280 18446744073709551615 4660157 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=466580 CPUtime=0.97 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 2593 0 0 0 96 1 0 0 19 0 8 0 741687558 477777920 19319 33554432000 4194304 4964800 140737384317280 18446744073709551615 4669136 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=466580 CPUtime=1 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 545 0 0 0 100 0 0 0 19 0 8 0 741687558 477777920 19319 33554432000 4194304 4964800 140737384317280 18446744073709551615 4343376 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=466580 CPUtime=0.97 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 1663 0 0 0 97 0 0 0 17 0 8 0 741687558 477777920 19319 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=466580 CPUtime=0.98 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 2575 0 0 0 97 1 0 0 17 0 8 0 741687558 477777920 19319 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 9.86
Current children cumulated vsize (KiB) 466580

[startup+6.30048 s]
/proc/loadavg: 7.57 7.55 6.46 9/426 30016
/proc/meminfo: memFree=13371092/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=483572 CPUtime=35.38 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 51837 0 0 0 3514 24 0 0 19 0 8 0 741687361 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/30008/statm: 120893 49323 466 189 0 115242 0
[pid=30008/tid=30010] ppid=30006 vsize=483572 CPUtime=4.16 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 5796 0 0 0 413 3 0 0 19 0 8 0 741687558 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=483572 CPUtime=4.17 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 3925 0 0 0 416 1 0 0 25 0 8 0 741687558 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=483572 CPUtime=4.17 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 4682 0 0 0 415 2 0 0 25 0 8 0 741687558 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4654463 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=483572 CPUtime=4.16 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 7111 0 0 0 414 2 0 0 25 0 8 0 741687558 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=483572 CPUtime=4.19 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 4002 0 0 0 418 1 0 0 21 0 8 0 741687558 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320961 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=483572 CPUtime=4.12 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 5856 0 0 0 410 2 0 0 19 0 8 0 741687558 495177728 49323 33554432000 4194304 4964800 140737384317280 18446744073709551615 4654540 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=483572 CPUtime=4.18 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 6951 0 0 0 414 4 0 0 19 0 8 0 741687558 495177728 49325 33554432000 4194304 4964800 140737384317280 18446744073709551615 216734839841 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35.38
Current children cumulated vsize (KiB) 483572

[startup+12.7004 s]
/proc/loadavg: 7.61 7.56 6.47 9/426 30016
/proc/meminfo: memFree=13195892/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=509652 CPUtime=86.46 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 93782 0 0 0 8599 47 0 0 25 0 8 0 741687361 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/30008/statm: 127413 91268 466 189 0 121762 0
[pid=30008/tid=30010] ppid=30006 vsize=509652 CPUtime=10.55 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 12165 0 0 0 1049 6 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4654668 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=509652 CPUtime=10.57 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 6500 0 0 0 1055 2 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320963 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=509652 CPUtime=10.57 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 8656 0 0 0 1052 5 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=509652 CPUtime=10.56 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 13961 0 0 0 1050 6 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=509652 CPUtime=10.59 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 9217 0 0 0 1055 4 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=509652 CPUtime=10.43 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 10031 0 0 0 1038 5 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=509652 CPUtime=10.58 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 13735 0 0 0 1051 7 0 0 25 0 8 0 741687558 521883648 91268 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 86.46
Current children cumulated vsize (KiB) 509652

[startup+25.5011 s]
/proc/loadavg: 7.67 7.57 6.48 9/426 30016
/proc/meminfo: memFree=12992436/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=923068 CPUtime=188.67 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 139579 0 0 0 18793 74 0 0 25 0 8 0 741687361 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4342144 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/30008/statm: 230767 137065 466 189 0 225116 0
[pid=30008/tid=30010] ppid=30006 vsize=923068 CPUtime=23.35 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 18574 0 0 0 2325 10 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=923068 CPUtime=23.38 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 8874 0 0 0 2335 3 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=923068 CPUtime=23.37 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 14238 0 0 0 2330 7 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=923068 CPUtime=23.35 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 18955 0 0 0 2327 8 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=923068 CPUtime=23.4 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 15881 0 0 0 2330 10 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=923068 CPUtime=23.05 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 13321 0 0 0 2299 6 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=923068 CPUtime=23.37 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 23736 0 0 0 2324 13 0 0 25 0 8 0 741687558 945221632 137065 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 188.67
Current children cumulated vsize (KiB) 923068

[startup+51.1007 s]
/proc/loadavg: 7.80 7.61 6.53 9/426 30017
/proc/meminfo: memFree=12849692/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=939524 CPUtime=393.06 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 175598 0 0 0 39209 97 0 0 25 0 8 0 741687361 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320944 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/30008/statm: 234881 173076 466 189 0 229230 0
[pid=30008/tid=30010] ppid=30006 vsize=939524 CPUtime=48.93 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 24045 0 0 0 4879 14 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=939524 CPUtime=48.96 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 10796 0 0 0 4893 3 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=939524 CPUtime=48.96 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 17948 0 0 0 4885 11 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4654529 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=939524 CPUtime=48.95 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 23068 0 0 0 4884 11 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=939524 CPUtime=48.98 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 20277 0 0 0 4886 12 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=939524 CPUtime=48.3 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 15253 0 0 0 4823 7 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4654540 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=939524 CPUtime=48.97 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 29056 0 0 0 4881 16 0 0 25 0 8 0 741687558 962072576 173076 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 393.06
Current children cumulated vsize (KiB) 939524

[startup+102.317 s]
/proc/loadavg: 7.91 7.67 6.61 9/426 30018
/proc/meminfo: memFree=12570672/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=1256500 CPUtime=802.01 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 244669 0 0 0 80054 147 0 0 25 0 8 0 741687361 1286656000 242120 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670332 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30008/statm: 314125 242120 466 189 0 308474 0
[pid=30008/tid=30010] ppid=30006 vsize=1256500 CPUtime=100.12 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 38416 0 0 0 9987 25 0 0 25 0 8 0 741687558 1286656000 242120 33554432000 4194304 4964800 140737384317280 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=1256500 CPUtime=100.17 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 13680 0 0 0 10011 6 0 0 25 0 8 0 741687558 1286656000 242120 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=1256500 CPUtime=100.18 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 22718 0 0 0 10002 16 0 0 25 0 8 0 741687558 1286656000 242120 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0

################
# More data... #
################

[pid=30008] ppid=30006 vsize=5543064 CPUtime=37695 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1323138 0 0 0 3768137 1365 0 0 25 0 8 0 741687361 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4321016 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30008/statm: 1385766 1308071 466 189 0 1380115 0
[pid=30008/tid=30010] ppid=30006 vsize=5543064 CPUtime=4698.47 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 146352 0 0 0 469693 154 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=5543064 CPUtime=4719.06 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 74816 0 0 0 471874 32 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=5543064 CPUtime=4719.81 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 96910 0 0 0 471882 99 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=5543064 CPUtime=4719.43 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 66562 0 0 0 471887 56 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=5543064 CPUtime=4720.04 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 80816 0 0 0 471934 70 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=5543064 CPUtime=4706.72 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 99853 0 0 0 470554 118 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=5543064 CPUtime=4710.71 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 371238 0 0 0 470650 421 0 0 25 0 8 0 741687558 5676097536 1308071 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37695
Current children cumulated vsize (KiB) 5543064

[startup+4782.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/426 30164
/proc/meminfo: memFree=8294048/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=5539884 CPUtime=38174 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1325349 0 0 0 3816031 1367 0 0 25 0 8 0 741687361 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30008/statm: 1384971 1308914 466 189 0 1379320 0
[pid=30008/tid=30010] ppid=30006 vsize=5539884 CPUtime=4757.65 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 146732 0 0 0 475611 154 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=5539884 CPUtime=4779.03 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 75063 0 0 0 477871 32 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=5539884 CPUtime=4779.75 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 96918 0 0 0 477875 100 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=5539884 CPUtime=4779.42 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 67952 0 0 0 477886 56 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=5539884 CPUtime=4780.04 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 80819 0 0 0 477933 71 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=5539884 CPUtime=4766.65 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 100019 0 0 0 476546 119 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=5539884 CPUtime=4770.69 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 371247 0 0 0 476648 421 0 0 25 0 8 0 741687558 5672841216 1308914 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38174
Current children cumulated vsize (KiB) 5539884

[startup+4842.3 s]
/proc/loadavg: 8.09 8.06 8.00 9/426 30166
/proc/meminfo: memFree=8293024/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=5539884 CPUtime=38653.1 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1325777 0 0 0 3863946 1368 0 0 25 0 8 0 741687361 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30008/statm: 1384971 1309302 466 189 0 1379320 0
[pid=30008/tid=30010] ppid=30006 vsize=5539884 CPUtime=4816.83 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 146740 0 0 0 481529 154 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670351 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=5539884 CPUtime=4839.02 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 75171 0 0 0 483870 32 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=5539884 CPUtime=4839.75 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 96925 0 0 0 483875 100 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=5539884 CPUtime=4839.42 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 68080 0 0 0 483886 56 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=5539884 CPUtime=4840.04 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 80966 0 0 0 483933 71 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=5539884 CPUtime=4826.65 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 100048 0 0 0 482546 119 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=5539884 CPUtime=4830.68 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 371248 0 0 0 482647 421 0 0 25 0 8 0 741687558 5672841216 1309302 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38653.1
Current children cumulated vsize (KiB) 5539884

[startup+4902.3 s]
/proc/loadavg: 8.09 8.06 8.00 9/426 30167
/proc/meminfo: memFree=8291384/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=5539884 CPUtime=39132.3 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1326230 0 0 0 3911863 1368 0 0 25 0 8 0 741687361 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670328 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30008/statm: 1384971 1309755 466 189 0 1379320 0
[pid=30008/tid=30010] ppid=30006 vsize=5539884 CPUtime=4876.01 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 146741 0 0 0 487447 154 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4627281 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=5539884 CPUtime=4899 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 75548 0 0 0 489868 32 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=5539884 CPUtime=4899.75 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 96925 0 0 0 489875 100 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=5539884 CPUtime=4899.42 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 68111 0 0 0 489886 56 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=5539884 CPUtime=4900.04 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 80975 0 0 0 489933 71 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=5539884 CPUtime=4886.65 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 100064 0 0 0 488546 119 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=5539884 CPUtime=4890.68 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 371263 0 0 0 488647 421 0 0 25 0 8 0 741687558 5672841216 1309755 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39132.3
Current children cumulated vsize (KiB) 5539884

[startup+4962.3 s]
/proc/loadavg: 8.14 8.08 8.01 9/426 30169
/proc/meminfo: memFree=8289620/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=5539884 CPUtime=39611.5 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1326769 0 0 0 3959778 1368 0 0 25 0 8 0 741687361 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30008/statm: 1384971 1310294 466 189 0 1379320 0
[pid=30008/tid=30010] ppid=30006 vsize=5539884 CPUtime=4935.18 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 146742 0 0 0 493364 154 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=5539884 CPUtime=4958.99 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 75828 0 0 0 495867 32 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=5539884 CPUtime=4959.75 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 96935 0 0 0 495875 100 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=5539884 CPUtime=4959.42 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 68136 0 0 0 495886 56 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=5539884 CPUtime=4960.04 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 81162 0 0 0 495933 71 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=5539884 CPUtime=4946.65 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 100094 0 0 0 494546 119 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=5539884 CPUtime=4950.68 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 371269 0 0 0 494647 421 0 0 25 0 8 0 741687558 5672841216 1310294 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39611.5
Current children cumulated vsize (KiB) 5539884



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011 s]
/proc/loadavg: 8.11 8.08 8.01 9/426 30170
/proc/meminfo: memFree=8286368/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=5539884 CPUtime=40000.4 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1327570 0 0 0 3998668 1369 0 0 25 0 8 0 741687361 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30008/statm: 1384971 1311095 466 189 0 1379320 0
[pid=30008/tid=30010] ppid=30006 vsize=5539884 CPUtime=4983.28 cores=0-7
/proc/30008/task/30010/stat : 30010 (claspmt) R 30006 30008 29906 0 -1 4202560 146743 0 0 0 498174 154 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30008/tid=30011] ppid=30006 vsize=5539884 CPUtime=5007.7 cores=0-7
/proc/30008/task/30011/stat : 30011 (claspmt) R 30006 30008 29906 0 -1 4202560 76531 0 0 0 500738 32 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30008/tid=30012] ppid=30006 vsize=5539884 CPUtime=5008.46 cores=0-7
/proc/30008/task/30012/stat : 30012 (claspmt) R 30006 30008 29906 0 -1 4202560 96937 0 0 0 500746 100 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30008/tid=30013] ppid=30006 vsize=5539884 CPUtime=5008.03 cores=0-7
/proc/30008/task/30013/stat : 30013 (claspmt) R 30006 30008 29906 0 -1 4202560 68150 0 0 0 500746 57 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30008/tid=30014] ppid=30006 vsize=5539884 CPUtime=5008.75 cores=0-7
/proc/30008/task/30014/stat : 30014 (claspmt) R 30006 30008 29906 0 -1 4202560 81197 0 0 0 500804 71 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30008/tid=30015] ppid=30006 vsize=5539884 CPUtime=4995.38 cores=0-7
/proc/30008/task/30015/stat : 30015 (claspmt) R 30006 30008 29906 0 -1 4202560 100123 0 0 0 499418 120 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30008/tid=30016] ppid=30006 vsize=5539884 CPUtime=4999.32 cores=0-7
/proc/30008/task/30016/stat : 30016 (claspmt) R 30006 30008 29906 0 -1 4202560 371283 0 0 0 499511 421 0 0 25 0 8 0 741687558 5672841216 1311095 33554432000 4194304 4964800 140737384317280 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 5539884

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011.4 s]
/proc/loadavg: 8.11 8.08 8.01 9/426 30170
/proc/meminfo: memFree=8286368/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=5527332 CPUtime=40000.8 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1327580 0 0 0 3998710 1370 0 0 25 0 1 0 741687361 5659987968 1307424 33554432000 4194304 4964800 140737384317280 18446744073709551615 216734835694 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/30008/statm: 1381833 1307424 476 189 0 1376182 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 5527332

[startup+5011.8 s]
/proc/loadavg: 8.11 8.08 8.01 2/420 30171
/proc/meminfo: memFree=8300428/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=4790028 CPUtime=40001.2 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1327580 0 0 0 3998747 1373 0 0 25 0 1 0 741687361 4904988672 1111995 33554432000 4194304 4964800 140737384317280 18446744073709551615 216735221895 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/30008/statm: 1197507 1111961 476 189 0 1191856 0
Current children cumulated CPU time (s) 40001.2
Current children cumulated vsize (KiB) 4790028

[startup+5011.9 s]
/proc/loadavg: 8.11 8.08 8.01 2/420 30171
/proc/meminfo: memFree=8300428/32951124 swapFree=65586204/67111528
[pid=30008] ppid=30006 vsize=2299660 CPUtime=40001.3 cores=0-7
/proc/30008/stat : 30008 (claspmt) R 30006 30008 29906 0 -1 4202496 1327580 0 0 0 3998747 1383 0 0 25 0 1 0 741687361 2354851840 479637 33554432000 4194304 4964800 140737384317280 18446744073709551615 216735221895 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/30008/statm: 574915 479141 476 189 0 569264 0
Current children cumulated CPU time (s) 40001.3
Current children cumulated vsize (KiB) 2299660

Child status: 0
Real time (s): 5011.98
CPU time (s): 40001.4
CPU user time (s): 39987.5
CPU system time (s): 13.9159
CPU usage (%): 798.115
Max. virtual memory (cumulated for all children) (KiB): 5545356

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39987.5
system time used= 13.9159
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1327581
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 101879
involuntary context switches= 186840

runsolver used 18.7781 second user time and 49.7854 second system time

The end

Launcher Data 

Begin job on node119 at 2011-05-16 04:56:07
IDJOB=3381385
IDBENCH=88221
IDSOLVER=1865
FILE ID=node119/3381385-1305514567
RUNJOBID= node119-1305513096-29920
PBS_JOBID= 13325538
Free space on /tmp= 73324 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m47-q841.5-n45-p15-s16393788.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381385-1305514567/watcher-3381385-1305514567 -o /tmp/evaluation-result-3381385-1305514567/solver-3381385-1305514567 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381385-1305514567.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 683c9f75b69575e6451b8563d12f176b
RANDOM SEED=27024573

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      13538932 kB
Buffers:        314312 kB
Cached:        5073664 kB
SwapCached:       4756 kB
Active:       15655264 kB
Inactive:      3412316 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      13538932 kB
SwapTotal:    67111528 kB
SwapFree:     65586204 kB
Dirty:            5044 kB
Writeback:          16 kB
AnonPages:    13678604 kB
Mapped:          33340 kB
Slab:           214892 kB
PageTables:      63588 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 120776868 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73324 MiB
End job on node119 at 2011-05-16 06:19:40