Trace number 3381161

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.8 5009.4

General information on the benchmark

NameSAT03/handmade/gomes/bqwh/
bqwh.60.1152.shuffled-as.sat03-1650.cnf
MD5SUM98d6317b259165bb4fb7a39e783668e0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark373.939
Satisfiable
(Un)Satisfiability was proved
Number of variables7534
Number of clauses72812
Sum of the clauses size161368
Maximum clause length14
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 269406
Number of clauses of size 3128
Number of clauses of size 4291
Number of clauses of size 5538
Number of clauses of size over 52449

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...506017/instance-3381161-1305506017.cnf
0.09/0.10	c Solving...
40000.17/5008.82	
40000.17/5008.82	*** INTERRUPTED! ***
40000.17/5008.82	*** Shutting down threads...
40000.72/5009.39	s UNKNOWN
40000.72/5009.39	
40000.72/5009.39	c INTERRUPTED   : 1
40000.72/5009.39	c Models        : 0+    
40000.72/5009.39	c Time          : 5008.823s (Solving: 5008.72s 1st Model: 0.00s Unsat: 0.00s)
40000.72/5009.39	c CPU Time      : 40000.240s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381161-1305506017/watcher-3381161-1305506017 -o /tmp/evaluation-result-3381161-1305506017/solver-3381161-1305506017 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381161-1305506017.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.34 6.57 7.34 2/366 29522
/proc/meminfo: memFree=13289424/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=23860 CPUtime=0 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 1378 0 0 0 0 0 0 0 20 0 1 0 662201351 24432640 1159 33554432000 4194304 4964800 140735556473856 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29522/statm: 5965 1160 386 189 0 839 0

[startup+0.088006 s]
/proc/loadavg: 5.34 6.57 7.34 2/366 29522
/proc/meminfo: memFree=13289424/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=27828 CPUtime=0.08 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 2610 0 0 0 8 0 0 0 20 0 1 0 662201351 28495872 2116 33554432000 4194304 4964800 140735556473856 18446744073709551615 4621728 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29522/statm: 6957 2116 400 189 0 1831 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 27828

[startup+0.101004 s]
/proc/loadavg: 5.34 6.57 7.34 2/366 29522
/proc/meminfo: memFree=13289424/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=27984 CPUtime=0.09 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 2658 0 0 0 9 0 0 0 20 0 1 0 662201351 28655616 2164 33554432000 4194304 4964800 140735556473856 18446744073709551615 4685934 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29522/statm: 6996 2164 403 189 0 1870 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27984

[startup+0.346294 s]
/proc/loadavg: 5.34 6.57 7.34 2/366 29522
/proc/meminfo: memFree=13289424/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=322980 CPUtime=1.75 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 6357 0 0 0 174 1 0 0 18 0 8 0 662201351 330731520 5365 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668568 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 80745 5365 464 189 0 75094 0
Current children cumulated CPU time (s) 1.75
Current children cumulated vsize (KiB) 322980

[startup+0.700893 s]
/proc/loadavg: 5.34 6.57 7.34 2/366 29522
/proc/meminfo: memFree=13289424/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=325488 CPUtime=4.4 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 9542 0 0 0 437 3 0 0 19 0 8 0 662201351 333299712 8550 33554432000 4194304 4964800 140735556473856 18446744073709551615 4342204 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 81372 8550 464 189 0 75721 0
Current children cumulated CPU time (s) 4.4
Current children cumulated vsize (KiB) 325488

[startup+1.50074 s]
/proc/loadavg: 5.34 6.57 7.34 9/374 29530
/proc/meminfo: memFree=13245528/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=331224 CPUtime=10.78 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 16354 0 0 0 1071 7 0 0 20 0 8 0 662201351 339173376 15362 33554432000 4194304 4964800 140735556473856 18446744073709551615 4660139 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 82806 15363 464 189 0 77155 0
[pid=29522/tid=29524] ppid=29520 vsize=331224 CPUtime=1.35 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 1663 0 0 0 134 1 0 0 19 0 8 0 662201361 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=331224 CPUtime=1.3 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 1923 0 0 0 129 1 0 0 25 0 8 0 662201361 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 4361293 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=331224 CPUtime=1.31 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 2046 0 0 0 131 0 0 0 19 0 8 0 662201362 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=331224 CPUtime=1.35 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 1849 0 0 0 134 1 0 0 19 0 8 0 662201362 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=331224 CPUtime=1.36 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 702 0 0 0 136 0 0 0 19 0 8 0 662201362 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 208079893189 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=331224 CPUtime=1.35 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 1586 0 0 0 135 0 0 0 19 0 8 0 662201362 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=331224 CPUtime=1.28 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 2383 0 0 0 128 0 0 0 19 0 8 0 662201362 339173376 15363 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 10.78
Current children cumulated vsize (KiB) 331224

[startup+3.10044 s]
/proc/loadavg: 5.34 6.57 7.34 9/374 29530
/proc/meminfo: memFree=13208844/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=342676 CPUtime=23.57 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 29084 0 0 0 2342 15 0 0 25 0 8 0 662201351 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321061 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 85669 28092 464 189 0 80018 0
[pid=29522/tid=29524] ppid=29520 vsize=342676 CPUtime=2.94 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 2576 0 0 0 293 1 0 0 23 0 8 0 662201361 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4684089 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=342676 CPUtime=2.9 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 3320 0 0 0 288 2 0 0 25 0 8 0 662201361 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=342676 CPUtime=2.91 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 3551 0 0 0 290 1 0 0 24 0 8 0 662201362 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4627571 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=342676 CPUtime=2.95 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 3177 0 0 0 294 1 0 0 24 0 8 0 662201362 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 208079893537 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=342676 CPUtime=2.96 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 2916 0 0 0 295 1 0 0 25 0 8 0 662201362 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=342676 CPUtime=2.95 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 2812 0 0 0 294 1 0 0 24 0 8 0 662201362 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=342676 CPUtime=2.88 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 4608 0 0 0 286 2 0 0 23 0 8 0 662201362 350900224 28092 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.57
Current children cumulated vsize (KiB) 342676

[startup+6.30083 s]
/proc/loadavg: 5.56 6.59 7.34 9/374 29530
/proc/meminfo: memFree=13152272/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=349148 CPUtime=49.12 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 37422 0 0 0 4893 19 0 0 25 0 8 0 662201351 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670499 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 87287 36430 466 189 0 81636 0
[pid=29522/tid=29524] ppid=29520 vsize=349148 CPUtime=6.15 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 3392 0 0 0 613 2 0 0 25 0 8 0 662201361 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653539 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=349148 CPUtime=6.11 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 4066 0 0 0 608 3 0 0 25 0 8 0 662201361 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=349148 CPUtime=6.11 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 4775 0 0 0 609 2 0 0 25 0 8 0 662201362 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4343849 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=349148 CPUtime=6.15 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 4006 0 0 0 613 2 0 0 25 0 8 0 662201362 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=349148 CPUtime=6.16 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 4349 0 0 0 614 2 0 0 25 0 8 0 662201362 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=349148 CPUtime=6.15 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 3673 0 0 0 614 1 0 0 21 0 8 0 662201362 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=349148 CPUtime=6.03 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 6199 0 0 0 601 2 0 0 25 0 8 0 662201362 357527552 36430 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 49.12
Current children cumulated vsize (KiB) 349148

[startup+12.7006 s]
/proc/loadavg: 5.75 6.62 7.35 9/374 29530
/proc/meminfo: memFree=13069592/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=366472 CPUtime=100.22 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 56401 0 0 0 9994 28 0 0 25 0 8 0 662201351 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320928 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 91618 55409 466 189 0 85967 0
[pid=29522/tid=29524] ppid=29520 vsize=366472 CPUtime=12.54 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 4725 0 0 0 1252 2 0 0 25 0 8 0 662201361 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=366472 CPUtime=12.51 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 5459 0 0 0 1247 4 0 0 25 0 8 0 662201361 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=366472 CPUtime=12.51 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 7093 0 0 0 1248 3 0 0 25 0 8 0 662201362 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=366472 CPUtime=12.54 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 6950 0 0 0 1251 3 0 0 25 0 8 0 662201362 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668924 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=366472 CPUtime=12.56 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 6714 0 0 0 1253 3 0 0 25 0 8 0 662201362 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=366472 CPUtime=12.55 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 5870 0 0 0 1253 2 0 0 25 0 8 0 662201362 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=366472 CPUtime=12.36 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 9588 0 0 0 1232 4 0 0 25 0 8 0 662201362 375267328 55409 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.22
Current children cumulated vsize (KiB) 366472

[startup+25.5012 s]
/proc/loadavg: 6.25 6.68 7.36 9/374 29530
/proc/meminfo: memFree=12987288/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=385984 CPUtime=202.44 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 78784 0 0 0 20204 40 0 0 25 0 8 0 662201351 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668984 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 96496 77792 466 189 0 90845 0
[pid=29522/tid=29524] ppid=29520 vsize=385984 CPUtime=25.33 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 6203 0 0 0 2530 3 0 0 25 0 8 0 662201361 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=385984 CPUtime=25.3 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 6316 0 0 0 2526 4 0 0 25 0 8 0 662201361 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 208079899610 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=385984 CPUtime=25.3 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 10781 0 0 0 2525 5 0 0 25 0 8 0 662201362 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=385984 CPUtime=25.34 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 9281 0 0 0 2530 4 0 0 25 0 8 0 662201362 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=385984 CPUtime=25.35 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 11086 0 0 0 2530 5 0 0 25 0 8 0 662201362 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4660139 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=385984 CPUtime=25.34 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 7798 0 0 0 2531 3 0 0 25 0 8 0 662201362 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653316 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=385984 CPUtime=25 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 13759 0 0 0 2494 6 0 0 25 0 8 0 662201362 395247616 77792 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 202.44
Current children cumulated vsize (KiB) 385984

[startup+51.1013 s]
/proc/loadavg: 6.85 6.79 7.37 9/374 29531
/proc/meminfo: memFree=12873796/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=667748 CPUtime=406.87 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 105763 0 0 0 40629 58 0 0 25 0 8 0 662201351 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 166937 104771 466 189 0 161286 0
[pid=29522/tid=29524] ppid=29520 vsize=667748 CPUtime=50.91 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 8658 0 0 0 5086 5 0 0 25 0 8 0 662201361 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=667748 CPUtime=50.89 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 7562 0 0 0 5085 4 0 0 25 0 8 0 662201361 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=667748 CPUtime=50.91 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 14121 0 0 0 5083 8 0 0 25 0 8 0 662201362 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=667748 CPUtime=50.92 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 11774 0 0 0 5086 6 0 0 25 0 8 0 662201362 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=667748 CPUtime=50.94 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 16721 0 0 0 5086 8 0 0 25 0 8 0 662201362 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=667748 CPUtime=50.93 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 10704 0 0 0 5087 6 0 0 25 0 8 0 662201362 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=667748 CPUtime=50.3 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 18536 0 0 0 5021 9 0 0 25 0 8 0 662201362 683773952 104771 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.87

################
# More data... #
################

[pid=29522] ppid=29520 vsize=3287264 CPUtime=37711.9 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 773280 0 0 0 3770534 660 0 0 25 0 8 0 662201351 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 821816 765736 466 189 0 816165 0
[pid=29522/tid=29524] ppid=29520 vsize=3287264 CPUtime=4715.52 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 45777 0 0 0 471496 56 0 0 25 0 8 0 662201361 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=3287264 CPUtime=4721.19 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 52548 0 0 0 472076 43 0 0 25 0 8 0 662201361 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=3287264 CPUtime=4690.81 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 99214 0 0 0 468986 95 0 0 25 0 8 0 662201362 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4363327 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=3287264 CPUtime=4719.16 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 83100 0 0 0 471844 72 0 0 25 0 8 0 662201362 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=3287264 CPUtime=4721.35 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 72551 0 0 0 472072 63 0 0 25 0 8 0 662201362 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=3287264 CPUtime=4721.02 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 95898 0 0 0 472012 90 0 0 25 0 8 0 662201362 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=3287264 CPUtime=4706.2 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 185953 0 0 0 470488 132 0 0 25 0 8 0 662201362 3366158336 765736 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37711.9
Current children cumulated vsize (KiB) 3287264

[startup+4782.3 s]
/proc/loadavg: 8.08 8.07 8.01 9/374 29669
/proc/meminfo: memFree=10216072/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3287264 CPUtime=38191.1 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 774168 0 0 0 3818449 661 0 0 25 0 8 0 662201351 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 821816 766615 466 189 0 816165 0
[pid=29522/tid=29524] ppid=29520 vsize=3287264 CPUtime=4775.51 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 45929 0 0 0 477495 56 0 0 25 0 8 0 662201361 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653558 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=3287264 CPUtime=4780.48 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 52710 0 0 0 478005 43 0 0 25 0 8 0 662201361 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=3287264 CPUtime=4750.78 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 99214 0 0 0 474983 95 0 0 25 0 8 0 662201362 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=3287264 CPUtime=4779.14 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 83674 0 0 0 477842 72 0 0 25 0 8 0 662201362 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=3287264 CPUtime=4781.34 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 72551 0 0 0 478071 63 0 0 25 0 8 0 662201362 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=3287264 CPUtime=4781.01 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 95898 0 0 0 478011 90 0 0 25 0 8 0 662201362 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=3287264 CPUtime=4766.17 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 185953 0 0 0 476485 132 0 0 25 0 8 0 662201362 3366158336 766615 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38191.1
Current children cumulated vsize (KiB) 3287264

[startup+4842.3 s]
/proc/loadavg: 8.03 8.05 8.00 9/374 29671
/proc/meminfo: memFree=10209828/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3287264 CPUtime=38670.2 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 775783 0 0 0 3866355 665 0 0 25 0 8 0 662201351 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 821816 768230 466 189 0 816165 0
[pid=29522/tid=29524] ppid=29520 vsize=3287264 CPUtime=4835.43 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 45929 0 0 0 483487 56 0 0 25 0 8 0 662201361 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=3287264 CPUtime=4840.03 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 52713 0 0 0 483960 43 0 0 25 0 8 0 662201361 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=3287264 CPUtime=4810.75 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 99295 0 0 0 480980 95 0 0 25 0 8 0 662201362 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4653299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=3287264 CPUtime=4838.83 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 83674 0 0 0 483811 72 0 0 25 0 8 0 662201362 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=3287264 CPUtime=4841.34 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 72551 0 0 0 484071 63 0 0 25 0 8 0 662201362 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=3287264 CPUtime=4840.99 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 95898 0 0 0 484009 90 0 0 25 0 8 0 662201362 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=3287264 CPUtime=4826.15 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 185983 0 0 0 482483 132 0 0 25 0 8 0 662201362 3366158336 768230 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38670.2
Current children cumulated vsize (KiB) 3287264

[startup+4902.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/374 29672
/proc/meminfo: memFree=10209668/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3287264 CPUtime=39149.4 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 775802 0 0 0 3914273 665 0 0 25 0 8 0 662201351 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670349 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 821816 768249 466 189 0 816165 0
[pid=29522/tid=29524] ppid=29520 vsize=3287264 CPUtime=4895.43 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 45929 0 0 0 489487 56 0 0 25 0 8 0 662201361 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=3287264 CPUtime=4900.03 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 52713 0 0 0 489960 43 0 0 25 0 8 0 662201361 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=3287264 CPUtime=4870.76 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 99295 0 0 0 486980 96 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=3287264 CPUtime=4898.35 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 83674 0 0 0 489763 72 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=3287264 CPUtime=4901.33 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 72564 0 0 0 490069 64 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=3287264 CPUtime=4900.97 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 95904 0 0 0 490007 90 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=3287264 CPUtime=4885.85 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 185983 0 0 0 488453 132 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39149.4
Current children cumulated vsize (KiB) 3287264

[startup+4962.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/374 29674
/proc/meminfo: memFree=10209496/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3287264 CPUtime=39628.6 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 775802 0 0 0 3962196 666 0 0 25 0 8 0 662201351 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 821816 768249 466 189 0 816165 0
[pid=29522/tid=29524] ppid=29520 vsize=3287264 CPUtime=4955.44 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 45929 0 0 0 495487 57 0 0 25 0 8 0 662201361 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=3287264 CPUtime=4960.03 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 52713 0 0 0 495960 43 0 0 25 0 8 0 662201361 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=3287264 CPUtime=4930.74 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 99295 0 0 0 492978 96 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=3287264 CPUtime=4958.36 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 83674 0 0 0 495763 73 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=3287264 CPUtime=4961.33 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 72564 0 0 0 496069 64 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=3287264 CPUtime=4960.97 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 95904 0 0 0 496006 91 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=3287264 CPUtime=4945.14 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 185983 0 0 0 494382 132 0 0 25 0 8 0 662201362 3366158336 768249 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39628.6
Current children cumulated vsize (KiB) 3287264



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.8 s]
/proc/loadavg: 8.04 8.04 8.00 9/374 29675
/proc/meminfo: memFree=10142876/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3352800 CPUtime=40000.2 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 792505 0 0 0 3999333 684 0 0 25 0 8 0 662201351 3433267200 784952 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670313 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29522/statm: 838200 784952 466 189 0 832549 0
[pid=29522/tid=29524] ppid=29520 vsize=3352800 CPUtime=5001.95 cores=0-7
/proc/29522/task/29524/stat : 29524 (claspmt) R 29520 29522 29294 0 -1 4202560 47177 0 0 0 500137 58 0 0 25 0 8 0 662201361 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29522/tid=29525] ppid=29520 vsize=3352800 CPUtime=5006.54 cores=0-7
/proc/29522/task/29525/stat : 29525 (claspmt) R 29520 29522 29294 0 -1 4202560 54493 0 0 0 500609 45 0 0 25 0 8 0 662201361 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29522/tid=29526] ppid=29520 vsize=3352800 CPUtime=4977.25 cores=0-7
/proc/29522/task/29526/stat : 29526 (claspmt) R 29520 29522 29294 0 -1 4202560 99350 0 0 0 497629 96 0 0 25 0 8 0 662201362 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 4321013 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29522/tid=29527] ppid=29520 vsize=3352800 CPUtime=5004.86 cores=0-7
/proc/29522/task/29527/stat : 29527 (claspmt) R 29520 29522 29294 0 -1 4202560 83714 0 0 0 500413 73 0 0 25 0 8 0 662201362 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29522/tid=29528] ppid=29520 vsize=3352800 CPUtime=5007.84 cores=0-7
/proc/29522/task/29528/stat : 29528 (claspmt) R 29520 29522 29294 0 -1 4202560 73946 0 0 0 500719 65 0 0 25 0 8 0 662201362 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29522/tid=29529] ppid=29520 vsize=3352800 CPUtime=5007.48 cores=0-7
/proc/29522/task/29529/stat : 29529 (claspmt) R 29520 29522 29294 0 -1 4202560 99714 0 0 0 500654 94 0 0 25 0 8 0 662201362 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 208079893537 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29522/tid=29530] ppid=29520 vsize=3352800 CPUtime=4991.09 cores=0-7
/proc/29522/task/29530/stat : 29530 (claspmt) R 29520 29522 29294 0 -1 4202560 191960 0 0 0 498971 138 0 0 25 0 8 0 662201362 3433267200 784953 33554432000 4194304 4964800 140735556473856 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 3352800

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009 s]
/proc/loadavg: 8.04 8.04 8.00 9/374 29675
/proc/meminfo: memFree=10142876/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3349424 CPUtime=40000.4 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 792544 0 0 0 3999357 684 0 0 25 0 1 0 662201351 3429810176 780380 33554432000 4194304 4964800 140735556473856 18446744073709551615 208079888945 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29522/statm: 837356 780380 476 189 0 831705 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 3349424

[startup+5009.2 s]
/proc/loadavg: 8.04 8.04 8.00 9/374 29675
/proc/meminfo: memFree=10142876/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=3346992 CPUtime=40000.6 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 792544 0 0 0 3999377 684 0 0 25 0 1 0 662201351 3427319808 778250 33554432000 4194304 4964800 140735556473856 18446744073709551615 208079887428 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29522/statm: 836748 778250 476 189 0 831097 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 3346992

[startup+5009.3 s]
/proc/loadavg: 8.04 8.04 8.00 2/368 29676
/proc/meminfo: memFree=11517684/32950928 swapFree=57376972/67111528
[pid=29522] ppid=29520 vsize=1730948 CPUtime=40000.7 cores=0-7
/proc/29522/stat : 29522 (claspmt) R 29520 29522 29294 0 -1 4202496 792544 0 0 0 3999380 692 0 0 25 0 1 0 662201351 1772490752 382817 33554432000 4194304 4964800 140735556473856 18446744073709551615 208080275591 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29522/statm: 432737 382817 476 189 0 427086 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 1730948

Child status: 0
Real time (s): 5009.4
CPU time (s): 40000.8
CPU user time (s): 39993.8
CPU system time (s): 7.01393
CPU usage (%): 798.515
Max. virtual memory (cumulated for all children) (KiB): 3352800

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.8
system time used= 7.01393
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 792545
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 165397
involuntary context switches= 191783

runsolver used 16.9314 second user time and 42.3826 second system time

The end

Launcher Data 

Begin job on node127 at 2011-05-16 02:33:37
IDJOB=3381161
IDBENCH=83797
IDSOLVER=1865
FILE ID=node127/3381161-1305506017
RUNJOBID= node127-1305505658-29308
PBS_JOBID= 13325499
Free space on /tmp= 73484 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT03/handmade/gomes/bqwh/bqwh.60.1152.shuffled-as.sat03-1650.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381161-1305506017/watcher-3381161-1305506017 -o /tmp/evaluation-result-3381161-1305506017/solver-3381161-1305506017 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381161-1305506017.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 98d6317b259165bb4fb7a39e783668e0
RANDOM SEED=1326888399

node127.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.60
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13289704 kB
Buffers:         14676 kB
Cached:         144772 kB
SwapCached:     117804 kB
Active:        5223264 kB
Inactive:     14235084 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13289704 kB
SwapTotal:    67111528 kB
SwapFree:     57376972 kB
Dirty:            1728 kB
Writeback:           0 kB
AnonPages:    19187300 kB
Mapped:          25152 kB
Slab:            56476 kB
PageTables:      77328 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 72332328 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73484 MiB
End job on node127 at 2011-05-16 03:57:08