Trace number 3380600

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001 5011.68

General information on the benchmark

NameAPPLICATIONS/
bitverif/maxxor/maxxor064.cnf
MD5SUM902b28046d424b16042ca612ddad48e0
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark809.913
Satisfiable
(Un)Satisfiability was proved
Number of variables51064
Number of clauses152039
Sum of the clauses size354755
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2101358
Number of clauses of size 350679
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...483529/instance-3380600-1305483529.cnf
0.29/0.38	c Solving...
40000.18/5010.92	
40000.18/5010.92	*** INTERRUPTED! ***
40000.18/5010.92	*** Shutting down threads...
40000.95/5011.64	s UNKNOWN
40000.95/5011.64	
40000.95/5011.64	c INTERRUPTED   : 1
40000.95/5011.64	c Models        : 0+    
40000.95/5011.64	c Time          : 5010.922s (Solving: 5010.54s 1st Model: 0.00s Unsat: 0.00s)
40000.95/5011.64	c CPU Time      : 40000.250s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380600-1305483529/watcher-3380600-1305483529 -o /tmp/evaluation-result-3380600-1305483529/solver-3380600-1305483529 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380600-1305483529.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 8.10 7.41 1/411 25165
/proc/meminfo: memFree=17103844/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=31020 CPUtime=0 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 3393 0 0 0 0 0 0 0 24 0 1 0 738583387 31764480 2924 33554432000 4194304 4964800 140734013902720 18446744073709551615 252458065664 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/25165/statm: 7755 2924 386 189 0 2629 0

[startup+0.069095 s]
/proc/loadavg: 8.11 8.10 7.41 1/411 25165
/proc/meminfo: memFree=17103844/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=42696 CPUtime=0.05 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 7389 0 0 0 5 0 0 0 24 0 1 0 738583387 43720704 5679 33554432000 4194304 4964800 140734013902720 18446744073709551615 252457763024 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/25165/statm: 10674 5695 399 189 0 5548 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 42696

[startup+0.101088 s]
/proc/loadavg: 8.11 8.10 7.41 1/411 25165
/proc/meminfo: memFree=17103844/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=46360 CPUtime=0.09 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 8250 0 0 0 8 1 0 0 24 0 1 0 738583387 47472640 6539 33554432000 4194304 4964800 140734013902720 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/25165/statm: 11590 6539 403 189 0 6464 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 46360

[startup+0.301053 s]
/proc/loadavg: 8.11 8.10 7.41 1/411 25165
/proc/meminfo: memFree=17103844/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=52696 CPUtime=0.29 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 9839 0 0 0 28 1 0 0 25 0 1 0 738583387 53960704 8128 33554432000 4194304 4964800 140734013902720 18446744073709551615 4622592 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/25165/statm: 13174 8128 404 189 0 8048 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 52696

[startup+0.700981 s]
/proc/loadavg: 8.11 8.10 7.41 1/411 25165
/proc/meminfo: memFree=17103844/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=362960 CPUtime=0.74 cores=0-7
/proc/25165/stat : 25165 (claspmt) S 25163 25165 24967 0 -1 4202496 22442 0 0 0 70 4 0 0 19 0 8 0 738583387 371671040 17138 33554432000 4194304 4964800 140734013902720 18446744073709551615 252469881753 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 90740 17138 438 189 0 85089 0
Current children cumulated CPU time (s) 0.74
Current children cumulated vsize (KiB) 362960

[startup+1.50079 s]
/proc/loadavg: 7.54 7.99 7.38 2/419 25173
/proc/meminfo: memFree=17037936/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=363108 CPUtime=3.8 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 23540 0 0 0 374 6 0 0 16 0 8 0 738583387 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 90777 18236 464 189 0 85126 0
[pid=25165/tid=25167] ppid=25163 vsize=363108 CPUtime=0.33 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 1367 0 0 0 33 0 0 0 16 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=363108 CPUtime=0.32 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 2164 0 0 0 32 0 0 0 16 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4342213 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=363108 CPUtime=0.32 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 2199 0 0 0 32 0 0 0 16 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=363108 CPUtime=0.32 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 1358 0 0 0 32 0 0 0 16 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=363108 CPUtime=1.08 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 1918 0 0 0 108 0 0 0 19 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668588 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=363108 CPUtime=0.31 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 2153 0 0 0 31 0 0 0 16 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=363108 CPUtime=0.33 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 2148 0 0 0 33 0 0 0 16 0 8 0 738583426 371822592 18236 33554432000 4194304 4964800 140734013902720 18446744073709551615 4360553 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3.8
Current children cumulated vsize (KiB) 363108

[startup+3.1005 s]
/proc/loadavg: 7.54 7.99 7.38 11/419 25173
/proc/meminfo: memFree=17029132/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=364184 CPUtime=16.57 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 26893 0 0 0 1649 8 0 0 18 0 8 0 738583387 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4370269 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 91046 21589 464 189 0 85395 0
[pid=25165/tid=25167] ppid=25163 vsize=364184 CPUtime=1.93 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 1579 0 0 0 193 0 0 0 18 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=364184 CPUtime=1.92 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 2712 0 0 0 192 0 0 0 18 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=364184 CPUtime=1.93 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 2725 0 0 0 192 1 0 0 18 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=364184 CPUtime=1.9 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 1627 0 0 0 190 0 0 0 18 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670496 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=364184 CPUtime=2.68 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 1924 0 0 0 268 0 0 0 22 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4342180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=364184 CPUtime=1.9 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 2570 0 0 0 190 0 0 0 18 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=364184 CPUtime=1.93 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 3329 0 0 0 192 1 0 0 18 0 8 0 738583426 372924416 21589 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 16.57
Current children cumulated vsize (KiB) 364184

[startup+6.3009 s]
/proc/loadavg: 7.54 7.99 7.38 9/419 25173
/proc/meminfo: memFree=17001620/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=371264 CPUtime=42.14 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 33561 0 0 0 4199 15 0 0 25 0 8 0 738583387 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 92816 28257 464 189 0 87165 0
[pid=25165/tid=25167] ppid=25163 vsize=371264 CPUtime=5.12 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 1953 0 0 0 512 0 0 0 19 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=371264 CPUtime=5.12 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 3948 0 0 0 510 2 0 0 25 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=371264 CPUtime=5.12 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 3797 0 0 0 510 2 0 0 25 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=371264 CPUtime=5.07 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 2052 0 0 0 506 1 0 0 25 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=371264 CPUtime=5.88 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 1939 0 0 0 588 0 0 0 25 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=371264 CPUtime=5.11 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 3269 0 0 0 510 1 0 0 25 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=371264 CPUtime=5.13 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 5625 0 0 0 511 2 0 0 25 0 8 0 738583426 380174336 28257 33554432000 4194304 4964800 140734013902720 18446744073709551615 252457733051 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 42.14
Current children cumulated vsize (KiB) 371264

[startup+12.7007 s]
/proc/loadavg: 7.61 7.99 7.39 9/419 25173
/proc/meminfo: memFree=16965600/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=376004 CPUtime=93.24 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 41612 0 0 0 9304 20 0 0 25 0 8 0 738583387 385028096 36308 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 94001 36308 464 189 0 88350 0
[pid=25165/tid=25167] ppid=25163 vsize=376004 CPUtime=11.52 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 2347 0 0 0 1152 0 0 0 25 0 8 0 738583426 385028096 36308 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=376004 CPUtime=11.52 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 5579 0 0 0 1149 3 0 0 25 0 8 0 738583426 385028096 36308 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=376004 CPUtime=11.53 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 4678 0 0 0 1150 3 0 0 25 0 8 0 738583426 385028096 36309 33554432000 4194304 4964800 140734013902720 18446744073709551615 252457726661 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=376004 CPUtime=11.37 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 2749 0 0 0 1136 1 0 0 25 0 8 0 738583426 385028096 36309 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=376004 CPUtime=12.28 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 1952 0 0 0 1228 0 0 0 25 0 8 0 738583426 385028096 36309 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=376004 CPUtime=11.5 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 4531 0 0 0 1148 2 0 0 25 0 8 0 738583426 385028096 36309 33554432000 4194304 4964800 140734013902720 18446744073709551615 4362960 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=376004 CPUtime=11.53 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 8564 0 0 0 1149 4 0 0 25 0 8 0 738583426 385028096 36309 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 93.24
Current children cumulated vsize (KiB) 376004

[startup+25.5003 s]
/proc/loadavg: 7.67 7.99 7.39 9/419 25174
/proc/meminfo: memFree=16911204/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=385552 CPUtime=195.45 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 55590 0 0 0 19517 28 0 0 25 0 8 0 738583387 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 96388 50286 466 189 0 90737 0
[pid=25165/tid=25167] ppid=25163 vsize=385552 CPUtime=24.33 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 2867 0 0 0 2432 1 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=385552 CPUtime=24.32 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 7831 0 0 0 2427 5 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669283 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=385552 CPUtime=24.32 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 5398 0 0 0 2429 3 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=385552 CPUtime=24 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 3157 0 0 0 2398 2 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=385552 CPUtime=25.08 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 1955 0 0 0 2508 0 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=385552 CPUtime=24.3 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 5488 0 0 0 2427 3 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=385552 CPUtime=24.33 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 17150 0 0 0 2424 9 0 0 25 0 8 0 738583426 394805248 50286 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 195.45
Current children cumulated vsize (KiB) 385552

[startup+51.1006 s]
/proc/loadavg: 7.80 7.99 7.41 9/419 25174
/proc/meminfo: memFree=16815932/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=408468 CPUtime=399.88 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 79579 0 0 0 39942 46 0 0 25 0 8 0 738583387 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/25165/statm: 102117 74275 466 189 0 96466 0
[pid=25165/tid=25167] ppid=25163 vsize=408468 CPUtime=49.92 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 3756 0 0 0 4990 2 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=408468 CPUtime=49.93 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 11647 0 0 0 4984 9 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=408468 CPUtime=49.92 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 7841 0 0 0 4987 5 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670678 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=408468 CPUtime=49.25 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 5662 0 0 0 4921 4 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=408468 CPUtime=50.69 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 4017 0 0 0 5067 2 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669574 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=408468 CPUtime=49.9 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 7562 0 0 0 4986 4 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=408468 CPUtime=49.93 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 26379 0 0 0 4978 15 0 0 25 0 8 0 738583426 418271232 74275 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 399.88

################
# More data... #
################

[pid=25165] ppid=25163 vsize=3479432 CPUtime=37696.7 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 811694 0 0 0 3769005 667 0 0 25 0 8 0 738583387 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669413 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/25165/statm: 869858 802077 466 189 0 864207 0
[pid=25165/tid=25167] ppid=25163 vsize=3479432 CPUtime=4700.08 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 21621 0 0 0 469971 37 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=3479432 CPUtime=4711.86 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 90916 0 0 0 471100 86 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=3479432 CPUtime=4718.98 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 33974 0 0 0 471859 39 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4332409 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=3479432 CPUtime=4699.65 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 106141 0 0 0 469858 107 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=3479432 CPUtime=4719.83 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 132034 0 0 0 471879 104 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670363 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=3479432 CPUtime=4720.31 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 30049 0 0 0 471991 40 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4684084 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=3479432 CPUtime=4720.48 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 368380 0 0 0 471831 217 0 0 25 0 8 0 738583426 3562938368 802077 33554432000 4194304 4964800 140734013902720 18446744073709551615 4627371 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37696.7
Current children cumulated vsize (KiB) 3479432

[startup+4782.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/419 25313
/proc/meminfo: memFree=13891852/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3479432 CPUtime=38175.7 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812635 0 0 0 3816903 668 0 0 25 0 8 0 738583387 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/25165/statm: 869858 803018 466 189 0 864207 0
[pid=25165/tid=25167] ppid=25163 vsize=3479432 CPUtime=4760.06 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 21621 0 0 0 475969 37 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670670 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=3479432 CPUtime=4771.85 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 90916 0 0 0 477099 86 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4361278 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=3479432 CPUtime=4778.96 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 33976 0 0 0 477856 40 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=3479432 CPUtime=4758.77 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 106141 0 0 0 475769 108 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4362439 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=3479432 CPUtime=4779.81 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 132972 0 0 0 477877 104 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=3479432 CPUtime=4780.29 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 30049 0 0 0 477989 40 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=3479432 CPUtime=4780.46 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 368380 0 0 0 477829 217 0 0 25 0 8 0 738583426 3562938368 803018 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38175.7
Current children cumulated vsize (KiB) 3479432

[startup+4842.3 s]
/proc/loadavg: 8.30 8.11 8.02 9/419 25317
/proc/meminfo: memFree=13892700/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3479432 CPUtime=38653.8 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812635 0 0 0 3864708 671 0 0 25 0 8 0 738583387 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669376 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/25165/statm: 869858 802597 466 189 0 864207 0
[pid=25165/tid=25167] ppid=25163 vsize=3479432 CPUtime=4819.46 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 21621 0 0 0 481908 38 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=3479432 CPUtime=4831.79 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 90916 0 0 0 483092 87 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670142 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=3479432 CPUtime=4838.79 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 33976 0 0 0 483839 40 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=3479432 CPUtime=4818.49 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 106141 0 0 0 481741 108 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=3479432 CPUtime=4839.46 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 132972 0 0 0 483842 104 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=3479432 CPUtime=4839.94 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 30049 0 0 0 483954 40 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=3479432 CPUtime=4840.35 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 368380 0 0 0 483818 217 0 0 25 0 8 0 738583426 3562938368 802597 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669512 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38653.8
Current children cumulated vsize (KiB) 3479432

[startup+4902.3 s]
/proc/loadavg: 8.16 8.11 8.02 9/419 25319
/proc/meminfo: memFree=13892516/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3479432 CPUtime=39132.9 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812640 0 0 0 3912617 672 0 0 25 0 8 0 738583387 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668987 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/25165/statm: 869858 802602 466 189 0 864207 0
[pid=25165/tid=25167] ppid=25163 vsize=3479432 CPUtime=4878.66 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 21621 0 0 0 487828 38 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=3479432 CPUtime=4891.79 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 90916 0 0 0 489092 87 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=3479432 CPUtime=4898.74 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 33976 0 0 0 489834 40 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=3479432 CPUtime=4878.49 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 106145 0 0 0 487741 108 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=3479432 CPUtime=4899.47 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 132972 0 0 0 489842 105 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4660114 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=3479432 CPUtime=4899.92 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 30049 0 0 0 489952 40 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4653348 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=3479432 CPUtime=4900.34 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 368381 0 0 0 489817 217 0 0 25 0 8 0 738583426 3562938368 802602 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39132.9
Current children cumulated vsize (KiB) 3479432

[startup+4962.3 s]
/proc/loadavg: 8.06 8.08 8.02 9/419 25320
/proc/meminfo: memFree=13892368/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3479432 CPUtime=39612.1 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812641 0 0 0 3960533 672 0 0 25 0 8 0 738583387 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668987 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/25165/statm: 869858 802603 466 189 0 864207 0
[pid=25165/tid=25167] ppid=25163 vsize=3479432 CPUtime=4937.83 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 21621 0 0 0 493745 38 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=3479432 CPUtime=4951.8 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 90916 0 0 0 495093 87 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=3479432 CPUtime=4958.73 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 33976 0 0 0 495833 40 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=3479432 CPUtime=4938.49 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 106146 0 0 0 493741 108 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=3479432 CPUtime=4959.47 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 132972 0 0 0 495842 105 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=3479432 CPUtime=4959.91 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 30049 0 0 0 495951 40 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=3479432 CPUtime=4960.33 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 368381 0 0 0 495816 217 0 0 25 0 8 0 738583426 3562938368 802603 33554432000 4194304 4964800 140734013902720 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39612
Current children cumulated vsize (KiB) 3479432



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.9 s]
/proc/loadavg: 8.08 8.08 8.02 9/419 25322
/proc/meminfo: memFree=13892480/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3479432 CPUtime=40000.2 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812645 0 0 0 3999344 674 0 0 25 0 8 0 738583387 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668734 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/25165/statm: 869858 802607 466 189 0 864207 0
[pid=25165/tid=25167] ppid=25163 vsize=3479432 CPUtime=4986.37 cores=0-7
/proc/25165/task/25167/stat : 25167 (claspmt) R 25163 25165 24967 0 -1 4202560 21621 0 0 0 498599 38 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=25165/tid=25168] ppid=25163 vsize=3479432 CPUtime=5000.41 cores=0-7
/proc/25165/task/25168/stat : 25168 (claspmt) R 25163 25165 24967 0 -1 4202560 90916 0 0 0 499954 87 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=25165/tid=25169] ppid=25163 vsize=3479432 CPUtime=5007.31 cores=0-7
/proc/25165/task/25169/stat : 25169 (claspmt) R 25163 25165 24967 0 -1 4202560 33976 0 0 0 500691 40 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=25165/tid=25170] ppid=25163 vsize=3479432 CPUtime=4986.44 cores=0-7
/proc/25165/task/25170/stat : 25170 (claspmt) R 25163 25165 24967 0 -1 4202560 106146 0 0 0 498536 108 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=25165/tid=25171] ppid=25163 vsize=3479432 CPUtime=5008.09 cores=0-7
/proc/25165/task/25171/stat : 25171 (claspmt) R 25163 25165 24967 0 -1 4202560 132976 0 0 0 500704 105 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4362375 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=25165/tid=25172] ppid=25163 vsize=3479432 CPUtime=5008.52 cores=0-7
/proc/25165/task/25172/stat : 25172 (claspmt) R 25163 25165 24967 0 -1 4202560 30049 0 0 0 500812 40 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=25165/tid=25173] ppid=25163 vsize=3479432 CPUtime=5008.94 cores=0-7
/proc/25165/task/25173/stat : 25173 (claspmt) R 25163 25165 24967 0 -1 4202560 368381 0 0 0 500676 218 0 0 25 0 8 0 738583426 3562938368 802607 33554432000 4194304 4964800 140734013902720 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 3479432

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011.02 s]
/proc/loadavg: 8.08 8.08 8.02 9/419 25322
/proc/meminfo: memFree=13892480/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3470652 CPUtime=40000.4 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812655 0 0 0 3999361 675 0 0 25 0 1 0 738583387 3553947648 800422 33554432000 4194304 4964800 140734013902720 18446744073709551615 252457722868 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/25165/statm: 867663 800422 476 189 0 862012 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 3470652

[startup+5011.4 s]
/proc/loadavg: 8.08 8.08 8.02 9/419 25322
/proc/meminfo: memFree=13892480/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=3453808 CPUtime=40000.7 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812655 0 0 0 3999398 675 0 0 25 0 1 0 738583387 3536699392 795477 33554432000 4194304 4964800 140734013902720 18446744073709551615 252457722427 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/25165/statm: 863452 795477 476 189 0 857801 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 3453808

[startup+5011.62 s]
/proc/loadavg: 8.08 8.08 8.02 2/413 25323
/proc/meminfo: memFree=13919808/32951124 swapFree=59305804/67111528
[pid=25165] ppid=25163 vsize=1616404 CPUtime=40000.9 cores=0-7
/proc/25165/stat : 25165 (claspmt) R 25163 25165 24967 0 -1 4202496 812655 0 0 0 3999413 682 0 0 25 0 1 0 738583387 1655197696 346692 33554432000 4194304 4964800 140734013902720 18446744073709551615 252458109063 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/25165/statm: 387717 346646 476 189 0 382066 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 1616404

Child status: 0
Real time (s): 5011.68
CPU time (s): 40001
CPU user time (s): 39994.1
CPU system time (s): 6.88295
CPU usage (%): 798.157
Max. virtual memory (cumulated for all children) (KiB): 3479432

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39994.1
system time used= 6.88295
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 812656
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 179639
involuntary context switches= 142437

runsolver used 18.2282 second user time and 50.1354 second system time

The end

Launcher Data 

Begin job on node111 at 2011-05-15 20:18:49
IDJOB=3380600
IDBENCH=70817
IDSOLVER=1865
FILE ID=node111/3380600-1305483529
RUNJOBID= node111-1305481206-24981
PBS_JOBID= 13325369
Free space on /tmp= 73632 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/APPLICATIONS/bitverif/maxxor/maxxor064.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380600-1305483529/watcher-3380600-1305483529 -o /tmp/evaluation-result-3380600-1305483529/solver-3380600-1305483529 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380600-1305483529.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 902b28046d424b16042ca612ddad48e0
RANDOM SEED=67690608

node111.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5428.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17104248 kB
Buffers:        300672 kB
Cached:        6822608 kB
SwapCached:      13712 kB
Active:        6826380 kB
Inactive:      8759980 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17104248 kB
SwapTotal:    67111528 kB
SwapFree:     59305804 kB
Dirty:            3504 kB
Writeback:           4 kB
AnonPages:     8458776 kB
Mapped:          26868 kB
Slab:           131532 kB
PageTables:      55812 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 123456568 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73628 MiB
End job on node111 at 2011-05-15 21:42:22