Trace number 3380911

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40002 5010.68

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k50-eijkbs6669-tseitin.cnf
MD5SUM7886cd26d6e0801ea382a1ffd6224023
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark2052.83
Satisfiable
(Un)Satisfiability was proved
Number of variables192503
Number of clauses564953
Sum of the clauses size1318221
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2376634
Number of clauses of size 3188317
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...494586/instance-3380911-1305494586.cnf
1.69/1.77	c Solving...
40000.24/5009.01	
40000.24/5009.01	*** INTERRUPTED! ***
40000.24/5009.01	*** Shutting down threads...
40001.91/5010.66	s UNKNOWN
40001.91/5010.66	
40001.91/5010.66	c INTERRUPTED   : 1
40001.91/5010.66	c Models        : 0+    
40001.91/5010.66	c Time          : 5009.177s (Solving: 5007.41s 1st Model: 0.00s Unsat: 0.00s)
40001.91/5010.66	c CPU Time      : 40000.490s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380911-1305494586/watcher-3380911-1305494586 -o /tmp/evaluation-result-3380911-1305494586/solver-3380911-1305494586 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380911-1305494586.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.60 6.07 4.16 1/331 6090
/proc/meminfo: memFree=20802900/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=45324 CPUtime=0 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 1837 0 0 0 0 0 0 0 21 0 1 0 641874288 46411776 1767 33554432000 4194304 4964800 140737326718640 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/6090/statm: 11331 1774 376 189 0 6205 0

[startup+0.0562731 s]
/proc/loadavg: 6.60 6.07 4.16 1/331 6090
/proc/meminfo: memFree=20802900/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=62644 CPUtime=0.04 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 12527 0 0 0 4 0 0 0 21 0 1 0 641874288 64147456 10662 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338044613 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/6090/statm: 15661 10664 386 189 0 10535 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 62644

[startup+0.100264 s]
/proc/loadavg: 6.60 6.07 4.16 1/331 6090
/proc/meminfo: memFree=20802900/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=70028 CPUtime=0.09 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 15141 0 0 0 8 1 0 0 21 0 1 0 641874288 71708672 12418 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338044613 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/6090/statm: 17507 12421 386 189 0 12381 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 70028

[startup+0.300223 s]
/proc/loadavg: 6.60 6.07 4.16 1/331 6090
/proc/meminfo: memFree=20802900/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=106468 CPUtime=0.29 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 27413 0 0 0 27 2 0 0 22 0 1 0 641874288 109023232 21243 33554432000 4194304 4964800 140737326718640 18446744073709551615 4615600 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/6090/statm: 26617 21245 399 189 0 21491 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 106468

[startup+0.700145 s]
/proc/loadavg: 6.60 6.07 4.16 1/331 6090
/proc/meminfo: memFree=20802900/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=124768 CPUtime=0.69 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 31910 0 0 0 66 3 0 0 23 0 1 0 641874288 127762432 25740 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338044613 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/6090/statm: 31192 25740 404 189 0 26066 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 124768

[startup+1.50099 s]
/proc/loadavg: 6.60 6.07 4.16 2/332 6091
/proc/meminfo: memFree=20695144/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=136516 CPUtime=1.49 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 34828 0 0 0 145 4 0 0 25 0 1 0 641874288 139792384 28658 33554432000 4194304 4964800 140737326718640 18446744073709551615 4615428 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/6090/statm: 34129 28658 404 189 0 29003 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 136516

[startup+3.10064 s]
/proc/loadavg: 6.60 6.07 4.16 9/339 6098
/proc/meminfo: memFree=20538624/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=560920 CPUtime=11.21 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 87026 0 0 0 1096 25 0 0 25 0 8 0 641874288 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670670 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/6090/statm: 140230 66849 464 189 0 134579 0
[pid=6090/tid=6092] ppid=6088 vsize=560920 CPUtime=1.16 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 6363 0 0 0 115 1 0 0 18 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=560920 CPUtime=1.17 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 7612 0 0 0 115 2 0 0 18 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4669184 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=560920 CPUtime=1.17 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 7811 0 0 0 114 3 0 0 18 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=560920 CPUtime=1.16 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 6070 0 0 0 113 3 0 0 18 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370462 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=560920 CPUtime=1.24 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 7205 0 0 0 122 2 0 0 19 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4674600 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=560920 CPUtime=1.16 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 7667 0 0 0 113 3 0 0 18 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=560920 CPUtime=1.2 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 7926 0 0 0 117 3 0 0 18 0 8 0 641874465 574382080 66849 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.21
Current children cumulated vsize (KiB) 560920

[startup+6.301 s]
/proc/loadavg: 6.80 6.12 4.19 9/339 6098
/proc/meminfo: memFree=20530480/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=562116 CPUtime=36.78 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 89429 0 0 0 3650 28 0 0 25 0 8 0 641874288 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/6090/statm: 140529 69252 464 189 0 134878 0
[pid=6090/tid=6092] ppid=6088 vsize=562116 CPUtime=4.36 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 6387 0 0 0 435 1 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=562116 CPUtime=4.33 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 7691 0 0 0 431 2 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=562116 CPUtime=4.36 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 8494 0 0 0 432 4 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=562116 CPUtime=4.36 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 6173 0 0 0 433 3 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=562116 CPUtime=4.44 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 7338 0 0 0 442 2 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=562116 CPUtime=4.37 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 8023 0 0 0 433 4 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=562116 CPUtime=4.4 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 8900 0 0 0 436 4 0 0 25 0 8 0 641874465 575606784 69252 33554432000 4194304 4964800 140737326718640 18446744073709551615 4321054 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 36.78
Current children cumulated vsize (KiB) 562116

[startup+12.7007 s]
/proc/loadavg: 6.89 6.15 4.21 9/339 6098
/proc/meminfo: memFree=20510160/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=566228 CPUtime=87.91 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 94759 0 0 0 8759 32 0 0 25 0 8 0 641874288 579817472 74582 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/6090/statm: 141557 74584 464 189 0 135906 0
[pid=6090/tid=6092] ppid=6088 vsize=566228 CPUtime=10.76 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 6531 0 0 0 1075 1 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 4333216 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=566228 CPUtime=10.66 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 7862 0 0 0 1064 2 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=566228 CPUtime=10.76 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 9762 0 0 0 1071 5 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668829 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=566228 CPUtime=10.76 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 6275 0 0 0 1073 3 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=566228 CPUtime=10.84 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 7863 0 0 0 1082 2 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=566228 CPUtime=10.77 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 8854 0 0 0 1072 5 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 4320897 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=566228 CPUtime=10.8 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 11081 0 0 0 1075 5 0 0 25 0 8 0 641874465 579817472 74584 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338044961 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 87.91
Current children cumulated vsize (KiB) 566228

[startup+25.5012 s]
/proc/loadavg: 7.14 6.24 4.27 9/339 6099
/proc/meminfo: memFree=20456040/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=573236 CPUtime=190.16 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 107612 0 0 0 18975 41 0 0 25 0 8 0 641874288 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370259 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/6090/statm: 143309 87435 464 189 0 137658 0
[pid=6090/tid=6092] ppid=6088 vsize=573236 CPUtime=23.57 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 6639 0 0 0 2355 2 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370381 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=573236 CPUtime=23.31 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 8392 0 0 0 2329 2 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4362944 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=573236 CPUtime=23.56 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 12284 0 0 0 2350 6 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=573236 CPUtime=23.55 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 6788 0 0 0 2352 3 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370496 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=573236 CPUtime=23.65 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 9344 0 0 0 2361 4 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370266 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=573236 CPUtime=23.56 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 11571 0 0 0 2349 7 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=573236 CPUtime=23.6 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 15576 0 0 0 2351 9 0 0 25 0 8 0 641874465 586993664 87435 33554432000 4194304 4964800 140737326718640 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 190.16
Current children cumulated vsize (KiB) 573236

[startup+51.1011 s]
/proc/loadavg: 7.49 6.39 4.37 9/339 6099
/proc/meminfo: memFree=20384872/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=582820 CPUtime=394.66 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 125455 0 0 0 39411 55 0 0 25 0 8 0 641874288 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370269 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/6090/statm: 145705 105278 466 189 0 140054 0
[pid=6090/tid=6092] ppid=6088 vsize=582820 CPUtime=49.16 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 7279 0 0 0 4914 2 0 0 25 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=582820 CPUtime=48.63 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 11148 0 0 0 4860 3 0 0 25 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670496 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=582820 CPUtime=49.15 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 15166 0 0 0 4906 9 0 0 21 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=582820 CPUtime=49.16 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 7641 0 0 0 4912 4 0 0 25 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=582820 CPUtime=49.24 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 11156 0 0 0 4919 5 0 0 25 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4321082 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=582820 CPUtime=49.16 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 14371 0 0 0 4907 9 0 0 25 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670366 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=582820 CPUtime=49.19 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 20643 0 0 0 4906 13 0 0 25 0 8 0 641874465 596807680 105278 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670440 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 394.66
Current children cumulated vsize (KiB) 582820

[startup+102.313 s]
/proc/loadavg: 7.78 6.64 4.56 9/339 6101
/proc/meminfo: memFree=20257016/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=803244 CPUtime=803.7 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 157241 0 0 0 80290 80 0 0 25 0 8 0 641874288 822521856 137064 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/6090/statm: 200811 137064 466 189 0 195160 0
[pid=6090/tid=6092] ppid=6088 vsize=803244 CPUtime=100.37 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 8404 0 0 0 10034 3 0 0 25 0 8 0 641874465 822521856 137064 33554432000 4194304 4964800 140737326718640 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=803244 CPUtime=99.27 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 14911 0 0 0 9921 6 0 0 25 0 8 0 641874465 822521856 137064 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=803244 CPUtime=100.34 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 20503 0 0 0 10022 12 0 0 25 0 8 0 641874465 822521856 137064 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0

################
# More data... #
################

[pid=6090] ppid=6088 vsize=3313872 CPUtime=37709.8 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 769302 0 0 0 3770359 622 0 0 25 0 8 0 641874288 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370224 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/6090/statm: 828468 744128 466 189 0 822817 0
[pid=6090/tid=6092] ppid=6088 vsize=3313872 CPUtime=4703.88 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 42116 0 0 0 470358 30 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=3313872 CPUtime=4715.49 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 106595 0 0 0 471479 70 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=3313872 CPUtime=4718.35 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 95856 0 0 0 471771 64 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=3313872 CPUtime=4720.18 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 64683 0 0 0 471975 43 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=3313872 CPUtime=4720.12 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 62847 0 0 0 471966 46 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=3313872 CPUtime=4719.77 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 100280 0 0 0 471909 68 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=3313872 CPUtime=4715.37 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 203372 0 0 0 471316 221 0 0 25 0 8 0 641874465 3393404928 744128 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37709.8
Current children cumulated vsize (KiB) 3313872

[startup+4782.3 s]
/proc/loadavg: 8.06 8.06 8.00 9/339 6240
/proc/meminfo: memFree=17806916/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3316644 CPUtime=38189.1 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 772904 0 0 0 3818279 628 0 0 25 0 8 0 641874288 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/6090/statm: 829161 747730 466 189 0 823510 0
[pid=6090/tid=6092] ppid=6088 vsize=3316644 CPUtime=4763.87 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 42130 0 0 0 476357 30 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=3316644 CPUtime=4775.49 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 106701 0 0 0 477478 71 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=3316644 CPUtime=4778.33 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 96447 0 0 0 477768 65 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4644128 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=3316644 CPUtime=4780.17 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 65018 0 0 0 477974 43 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=3316644 CPUtime=4780.11 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 62868 0 0 0 477965 46 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=3316644 CPUtime=4779.76 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 100464 0 0 0 477907 69 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=3316644 CPUtime=4775.36 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 205009 0 0 0 477312 224 0 0 25 0 8 0 641874465 3396243456 747730 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38189.1
Current children cumulated vsize (KiB) 3316644

[startup+4842.3 s]
/proc/loadavg: 8.06 8.06 8.00 9/339 6242
/proc/meminfo: memFree=17784204/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3318360 CPUtime=38668.4 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 778501 0 0 0 3866206 636 0 0 25 0 8 0 641874288 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/6090/statm: 829590 753251 466 189 0 823939 0
[pid=6090/tid=6092] ppid=6088 vsize=3318360 CPUtime=4823.88 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 42471 0 0 0 482357 31 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=3318360 CPUtime=4835.5 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 107211 0 0 0 483479 71 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338081422 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=3318360 CPUtime=4838.32 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 96764 0 0 0 483767 65 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=3318360 CPUtime=4840.17 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 65098 0 0 0 483974 43 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=3318360 CPUtime=4840.1 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 63810 0 0 0 483964 46 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=3318360 CPUtime=4839.75 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 101888 0 0 0 483906 69 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=3318360 CPUtime=4835.35 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 205054 0 0 0 483311 224 0 0 25 0 8 0 641874465 3398000640 753251 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38668.4
Current children cumulated vsize (KiB) 3318360

[startup+4902.3 s]
/proc/loadavg: 8.06 8.06 8.00 9/339 6244
/proc/meminfo: memFree=17783564/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3319216 CPUtime=39147.7 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 778705 0 0 0 3914136 637 0 0 25 0 8 0 641874288 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370288 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/6090/statm: 829804 753455 466 189 0 824153 0
[pid=6090/tid=6092] ppid=6088 vsize=3319216 CPUtime=4883.88 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 42488 0 0 0 488357 31 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=3319216 CPUtime=4895.49 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 107211 0 0 0 489478 71 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=3319216 CPUtime=4898.31 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 96885 0 0 0 489765 66 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=3319216 CPUtime=4900.15 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 65110 0 0 0 489971 44 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370229 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=3319216 CPUtime=4900.1 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 63813 0 0 0 489964 46 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=3319216 CPUtime=4899.76 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 101891 0 0 0 489907 69 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4675677 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=3319216 CPUtime=4895.36 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 205102 0 0 0 489311 225 0 0 25 0 8 0 641874465 3398877184 753455 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39147.7
Current children cumulated vsize (KiB) 3319216

[startup+4962.3 s]
/proc/loadavg: 8.05 8.06 8.00 9/339 6245
/proc/meminfo: memFree=17782284/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3319500 CPUtime=39627.1 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 778957 0 0 0 3962069 638 0 0 25 0 8 0 641874288 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/6090/statm: 829875 753707 466 189 0 824224 0
[pid=6090/tid=6092] ppid=6088 vsize=3319500 CPUtime=4943.89 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 42488 0 0 0 494358 31 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=3319500 CPUtime=4955.49 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 107211 0 0 0 495478 71 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=3319500 CPUtime=4958.3 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 96889 0 0 0 495764 66 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4321099 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=3319500 CPUtime=4960.15 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 65110 0 0 0 495971 44 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=3319500 CPUtime=4960.11 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 63990 0 0 0 495964 47 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=3319500 CPUtime=4959.76 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 101960 0 0 0 495907 69 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338081374 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=3319500 CPUtime=4955.36 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 205104 0 0 0 495311 225 0 0 25 0 8 0 641874465 3399168000 753707 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39627.1
Current children cumulated vsize (KiB) 3319500



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009 s]
/proc/loadavg: 8.02 8.05 8.00 9/339 6246
/proc/meminfo: memFree=17780036/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3320140 CPUtime=40000.2 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 779503 0 0 0 3999386 638 0 0 25 0 8 0 641874288 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4644876 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/6090/statm: 830035 754253 466 189 0 824384 0
[pid=6090/tid=6092] ppid=6088 vsize=3320140 CPUtime=4990.6 cores=0-7
/proc/6090/task/6092/stat : 6092 (claspmt) R 6088 6090 5686 0 -1 4202560 42506 0 0 0 499029 31 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6090/tid=6093] ppid=6088 vsize=3320140 CPUtime=5002.2 cores=0-7
/proc/6090/task/6093/stat : 6093 (claspmt) R 6088 6090 5686 0 -1 4202560 107211 0 0 0 500149 71 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6090/tid=6094] ppid=6088 vsize=3320140 CPUtime=5005 cores=0-7
/proc/6090/task/6094/stat : 6094 (claspmt) R 6088 6090 5686 0 -1 4202560 97246 0 0 0 500434 66 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6090/tid=6095] ppid=6088 vsize=3320140 CPUtime=5006.87 cores=0-7
/proc/6090/task/6095/stat : 6095 (claspmt) R 6088 6090 5686 0 -1 4202560 65112 0 0 0 500643 44 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6090/tid=6096] ppid=6088 vsize=3320140 CPUtime=5006.83 cores=0-7
/proc/6090/task/6096/stat : 6096 (claspmt) R 6088 6090 5686 0 -1 4202560 64081 0 0 0 500636 47 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=6090/tid=6097] ppid=6088 vsize=3320140 CPUtime=5006.48 cores=0-7
/proc/6090/task/6097/stat : 6097 (claspmt) R 6088 6090 5686 0 -1 4202560 102035 0 0 0 500578 70 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4669386 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6090/tid=6098] ppid=6088 vsize=3320140 CPUtime=5002.07 cores=0-7
/proc/6090/task/6098/stat : 6098 (claspmt) R 6088 6090 5686 0 -1 4202560 205107 0 0 0 499982 225 0 0 25 0 8 0 641874465 3399823360 754253 33554432000 4194304 4964800 140737326718640 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 3320140

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.41 s]
/proc/loadavg: 8.02 8.05 8.00 2/333 6248
/proc/meminfo: memFree=17803644/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3295884 CPUtime=40000.7 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 779515 0 0 0 3999434 639 0 0 25 0 1 0 641874288 3374985216 748201 33554432000 4194304 4964800 140737326718640 18446744073709551615 4646607 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/6090/statm: 823971 748201 476 189 0 818320 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 3295884

[startup+5010.2 s]
/proc/loadavg: 8.02 8.05 8.00 2/333 6248
/proc/meminfo: memFree=17803644/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=3242560 CPUtime=40001.5 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 779515 0 0 0 3999512 639 0 0 25 0 1 0 641874288 3320381440 733614 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338040369 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/6090/statm: 810640 733614 476 189 0 804989 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 3242560

[startup+5010.6 s]
/proc/loadavg: 8.02 8.05 8.00 2/333 6248
/proc/meminfo: memFree=17884616/32951124 swapFree=57171160/67111528
[pid=6090] ppid=6088 vsize=1806048 CPUtime=40001.9 cores=0-7
/proc/6090/stat : 6090 (claspmt) R 6088 6090 5686 0 -1 4202496 779515 0 0 0 3999546 645 0 0 25 0 1 0 641874288 1849393152 381472 33554432000 4194304 4964800 140737326718640 18446744073709551615 270338427015 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/6090/statm: 451512 380960 476 189 0 445861 0
Current children cumulated CPU time (s) 40001.9
Current children cumulated vsize (KiB) 1806048

Child status: 0
Real time (s): 5010.68
CPU time (s): 40002
CPU user time (s): 39995.5
CPU system time (s): 6.53101
CPU usage (%): 798.335
Max. virtual memory (cumulated for all children) (KiB): 3320140

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.5
system time used= 6.53101
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 779516
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 60667
involuntary context switches= 184708

runsolver used 14.9577 second user time and 39.722 second system time

The end

Launcher Data 

Begin job on node143 at 2011-05-15 23:23:07
IDJOB=3380911
IDBENCH=82493
IDSOLVER=1865
FILE ID=node143/3380911-1305494586
RUNJOBID= node143-1305494300-5700
PBS_JOBID= 13325431
Free space on /tmp= 73536 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k50-eijkbs6669-tseitin.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380911-1305494586/watcher-3380911-1305494586 -o /tmp/evaluation-result-3380911-1305494586/solver-3380911-1305494586 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380911-1305494586.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 7886cd26d6e0801ea382a1ffd6224023
RANDOM SEED=459751925

node143.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      20802824 kB
Buffers:        175940 kB
Cached:        1280752 kB
SwapCached:      39776 kB
Active:        4300980 kB
Inactive:      7581028 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      20802824 kB
SwapTotal:    67111528 kB
SwapFree:     57171160 kB
Dirty:           10660 kB
Writeback:           0 kB
AnonPages:    10385488 kB
Mapped:          23240 kB
Slab:           135772 kB
PageTables:      62680 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 74730964 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73536 MiB
End job on node143 at 2011-05-16 00:46:38