Trace number 3322556

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40001.1 5009.42

General information on the benchmark

NameSAT03/handmade/gomes/bqwh/
bqwh.60.1152.shuffled-as.sat03-1650.cnf
MD5SUM98d6317b259165bb4fb7a39e783668e0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark373.939
Satisfiable
(Un)Satisfiability was proved
Number of variables7534
Number of clauses72812
Sum of the clauses size161368
Maximum clause length14
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 269406
Number of clauses of size 3128
Number of clauses of size 4291
Number of clauses of size 5538
Number of clauses of size over 52449

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...388276/instance-3322556-1304388276.cnf
0.09/0.10	c Solving...
40000.60/5008.91	
40000.60/5008.91	*** INTERRUPTED! ***
40000.60/5008.91	*** Shutting down threads...
40001.13/5009.41	s UNKNOWN
40001.13/5009.41	
40001.13/5009.41	c INTERRUPTED   : 1
40001.13/5009.41	c Models        : 0+    
40001.13/5009.41	c Time          : 5008.918s (Solving: 5008.82s 1st Model: 0.00s Unsat: 0.00s)
40001.13/5009.41	c CPU Time      : 40000.640s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322556-1304388276/watcher-3322556-1304388276 -o /tmp/evaluation-result-3322556-1304388276/solver-3322556-1304388276 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322556-1304388276.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.82 8.24 8.36 1/356 29240
/proc/meminfo: memFree=4353204/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=23860 CPUtime=0.01 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 1411 0 0 0 1 0 0 0 23 0 1 0 629058270 24432640 1191 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367207621 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29240/statm: 5965 1192 386 189 0 839 0

[startup+0.034802 s]
/proc/loadavg: 7.82 8.24 8.36 1/356 29240
/proc/meminfo: memFree=4353204/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=27292 CPUtime=0.02 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 2487 0 0 0 2 0 0 0 23 0 1 0 629058270 27947008 1992 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367207621 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29240/statm: 6823 1993 397 189 0 1697 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 27292

[startup+0.100803 s]
/proc/loadavg: 7.82 8.24 8.36 1/356 29240
/proc/meminfo: memFree=4353204/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=27984 CPUtime=0.09 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 2657 0 0 0 9 0 0 0 23 0 1 0 629058270 28655616 2162 33554432000 4194304 4964704 140736449135712 18446744073709551615 4324281 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29240/statm: 6996 2162 403 189 0 1870 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27984

[startup+0.400617 s]
/proc/loadavg: 7.82 8.24 8.36 1/356 29240
/proc/meminfo: memFree=4353204/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=390660 CPUtime=3.31 cores=0-7
/proc/29240/stat : 29240 (claspmt) S 29238 29240 29199 0 -1 4202496 8529 0 0 0 328 3 0 0 25 0 8 0 629058270 400035840 7534 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367651822 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 98094 10819 464 189 0 92443 0
Current children cumulated CPU time (s) 3.31
Current children cumulated vsize (KiB) 390660

[startup+0.988489 s]
/proc/loadavg: 7.82 8.24 8.36 1/356 29240
/proc/meminfo: memFree=4353204/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=394488 CPUtime=7.96 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 14505 0 0 0 791 5 0 0 25 0 8 0 629058270 403955712 13510 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367207969 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 98622 13510 464 189 0 92971 0
Current children cumulated CPU time (s) 7.96
Current children cumulated vsize (KiB) 394488

[startup+1.81996 s]
/proc/loadavg: 7.82 8.24 8.36 1/356 29240
/proc/meminfo: memFree=4353204/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=398852 CPUtime=13.82 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 19852 0 0 0 1373 9 0 0 25 0 8 0 629058270 408424448 18857 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670140 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 99713 18857 464 189 0 94062 0
Current children cumulated CPU time (s) 13.82
Current children cumulated vsize (KiB) 398852

[startup+3.10117 s]
/proc/loadavg: 7.82 8.24 8.36 10/365 29263
/proc/meminfo: memFree=4261600/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=405628 CPUtime=23.72 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 29919 0 0 0 2358 14 0 0 18 0 8 0 629058270 415363072 28924 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367207451 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 101407 28924 464 189 0 95756 0
[pid=29240/tid=29252] ppid=29238 vsize=405628 CPUtime=2.95 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 2874 0 0 0 294 1 0 0 21 0 8 0 629058281 415363072 28926 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367207969 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=405628 CPUtime=2.94 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 3576 0 0 0 293 1 0 0 24 0 8 0 629058281 415363072 28926 33554432000 4194304 4964704 140736449135712 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=405628 CPUtime=2.94 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 3537 0 0 0 293 1 0 0 21 0 8 0 629058281 415363072 28926 33554432000 4194304 4964704 140736449135712 18446744073709551615 4342896 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=405628 CPUtime=2.94 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 3524 0 0 0 293 1 0 0 24 0 8 0 629058281 415363072 28926 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=405628 CPUtime=2.97 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 3458 0 0 0 295 2 0 0 25 0 8 0 629058281 415363072 28926 33554432000 4194304 4964704 140736449135712 18446744073709551615 4362339 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=405628 CPUtime=2.92 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 3775 0 0 0 291 1 0 0 19 0 8 0 629058281 415363072 28926 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670611 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=405628 CPUtime=2.95 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 3802 0 0 0 293 2 0 0 24 0 8 0 629058281 415363072 28928 33554432000 4194304 4964704 140736449135712 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.72
Current children cumulated vsize (KiB) 405628

[startup+6.30058 s]
/proc/loadavg: 7.92 8.25 8.37 10/364 29277
/proc/meminfo: memFree=4197668/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=414772 CPUtime=49.24 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 40879 0 0 0 4905 19 0 0 25 0 8 0 629058270 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4669200 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 103693 39884 466 189 0 98042 0
[pid=29240/tid=29252] ppid=29238 vsize=414772 CPUtime=6.14 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 3873 0 0 0 612 2 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4653279 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=414772 CPUtime=6.15 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 4722 0 0 0 613 2 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670360 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=414772 CPUtime=6.14 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 4756 0 0 0 612 2 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=414772 CPUtime=6.13 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 4948 0 0 0 611 2 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4342199 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=414772 CPUtime=6.16 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 5769 0 0 0 614 2 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=414772 CPUtime=6.08 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 4808 0 0 0 606 2 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=414772 CPUtime=6.15 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 5589 0 0 0 612 3 0 0 25 0 8 0 629058281 424726528 39884 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670283 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.24
Current children cumulated vsize (KiB) 414772

[startup+12.7004 s]
/proc/loadavg: 8.00 8.26 8.37 10/364 29306
/proc/meminfo: memFree=4141560/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=423148 CPUtime=100.35 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 56013 0 0 0 10006 29 0 0 25 0 8 0 629058270 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4660362 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 105787 55018 466 189 0 100136 0
[pid=29240/tid=29252] ppid=29238 vsize=423148 CPUtime=12.54 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 4689 0 0 0 1252 2 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=423148 CPUtime=12.54 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 5646 0 0 0 1252 2 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=423148 CPUtime=12.55 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 6777 0 0 0 1251 4 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4342199 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=423148 CPUtime=12.52 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 7262 0 0 0 1249 3 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=423148 CPUtime=12.56 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 9589 0 0 0 1252 4 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=423148 CPUtime=12.4 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 6121 0 0 0 1237 3 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4343791 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=423148 CPUtime=12.54 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 8074 0 0 0 1250 4 0 0 25 0 8 0 629058281 433303552 55018 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.35
Current children cumulated vsize (KiB) 423148

[startup+25.501 s]
/proc/loadavg: 8.22 8.30 8.38 10/364 29354
/proc/meminfo: memFree=4046232/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=433924 CPUtime=202.55 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 77307 0 0 0 20214 41 0 0 25 0 8 0 629058270 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4668928 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 108481 76312 466 189 0 102830 0
[pid=29240/tid=29252] ppid=29238 vsize=433924 CPUtime=25.33 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 5786 0 0 0 2530 3 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4631252 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=433924 CPUtime=25.34 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 7300 0 0 0 2531 3 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=433924 CPUtime=25.33 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 9994 0 0 0 2528 5 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=433924 CPUtime=25.31 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 9849 0 0 0 2527 4 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=433924 CPUtime=25.36 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 12768 0 0 0 2530 6 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=433924 CPUtime=25.04 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 7929 0 0 0 2499 5 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=433924 CPUtime=25.33 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 12286 0 0 0 2527 6 0 0 25 0 8 0 629058281 444338176 76312 33554432000 4194304 4964704 140736449135712 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 202.55
Current children cumulated vsize (KiB) 433924

[startup+51.1013 s]
/proc/loadavg: 8.49 8.35 8.39 10/364 29444
/proc/meminfo: memFree=3918984/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=786980 CPUtime=407 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 108598 0 0 0 40642 58 0 0 25 0 8 0 629058270 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4668907 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 196745 107592 466 189 0 191094 0
[pid=29240/tid=29252] ppid=29238 vsize=786980 CPUtime=50.92 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 8644 0 0 0 5087 5 0 0 20 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=786980 CPUtime=50.94 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 8503 0 0 0 5090 4 0 0 25 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=786980 CPUtime=50.92 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 14545 0 0 0 5085 7 0 0 25 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670288 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=786980 CPUtime=50.9 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 13624 0 0 0 5085 5 0 0 25 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=786980 CPUtime=50.95 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 18530 0 0 0 5086 9 0 0 25 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=786980 CPUtime=50.33 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 10445 0 0 0 5026 7 0 0 25 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4321082 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=786980 CPUtime=50.93 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 18231 0 0 0 5084 9 0 0 25 0 8 0 629058281 805867520 107592 33554432000 4194304 4964704 140736449135712 18446744073709551615 4654508 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 407
Current children cumulated vsize (KiB) 786980

[startup+102.301 s]
/proc/loadavg: 8.86 8.47 8.43 10/364 29645
/proc/meminfo: memFree=3713872/32950928 swapFree=62390272/67111528
[pid=29240] ppid=29238 vsize=822508 CPUtime=815.98 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 157715 0 0 0 81509 89 0 0 25 0 8 0 629058270 842248192 156431 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320944 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29240/statm: 205627 156431 466 189 0 199976 0
[pid=29240/tid=29252] ppid=29238 vsize=822508 CPUtime=102.12 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 11735 0 0 0 10204 8 0 0 25 0 8 0 629058281 842248192 156431 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=822508 CPUtime=102.14 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 10619 0 0 0 10209 5 0 0 25 0 8 0 629058281 842248192 156431 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=822508 CPUtime=102.12 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 19998 0 0 0 10201 11 0 0 25 0 8 0 629058281 842248192 156431 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 98485 0 0 0 463824 99 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3351312 CPUtime=4661.5 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 188063 0 0 0 466034 116 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4365732 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37232.4
Current children cumulated vsize (KiB) 3351312

[startup+4722.3 s]
/proc/loadavg: 8.02 8.04 8.03 9/359 30633
/proc/meminfo: memFree=1118964/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3351312 CPUtime=37711.8 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 790560 0 0 0 3770519 662 0 0 25 0 8 0 629058270 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4342918 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/29240/statm: 837828 784709 466 189 0 832177 0
[pid=29240/tid=29252] ppid=29238 vsize=3351312 CPUtime=4709.27 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 53659 0 0 0 470866 61 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=3351312 CPUtime=4713.87 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 43089 0 0 0 471347 40 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=3351312 CPUtime=4719.94 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 99239 0 0 0 471903 91 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=3351312 CPUtime=4711.73 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 78860 0 0 0 471102 71 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=3351312 CPUtime=4720.2 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 73337 0 0 0 471964 56 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=3351312 CPUtime=4699.23 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 98485 0 0 0 469824 99 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4342162 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3351312 CPUtime=4721.51 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 188063 0 0 0 472035 116 0 0 25 0 8 0 629058281 3431743488 784709 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37711.8
Current children cumulated vsize (KiB) 3351312

[startup+4782.3 s]
/proc/loadavg: 8.05 8.05 8.03 9/359 30635
/proc/meminfo: memFree=1099976/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3416848 CPUtime=38190.9 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 795618 0 0 0 3818417 668 0 0 25 0 8 0 629058270 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/29240/statm: 854212 789764 466 189 0 848561 0
[pid=29240/tid=29252] ppid=29238 vsize=3416848 CPUtime=4768.83 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 53659 0 0 0 476822 61 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=3416848 CPUtime=4773.86 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 43111 0 0 0 477346 40 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=3416848 CPUtime=4779.91 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 99273 0 0 0 477900 91 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4343154 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=3416848 CPUtime=4771.67 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 79464 0 0 0 477095 72 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=3416848 CPUtime=4780.14 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 73500 0 0 0 477957 57 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=3416848 CPUtime=4758.92 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 98671 0 0 0 475792 100 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3416848 CPUtime=4781.48 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 189415 0 0 0 478031 117 0 0 25 0 8 0 629058281 3498852352 789764 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38190.9
Current children cumulated vsize (KiB) 3416848

[startup+4842.31 s]
/proc/loadavg: 8.05 8.05 8.03 9/359 30636
/proc/meminfo: memFree=1099592/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3416848 CPUtime=38670 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 795736 0 0 0 3866329 670 0 0 25 0 8 0 629058270 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/29240/statm: 854212 789882 466 189 0 848561 0
[pid=29240/tid=29252] ppid=29238 vsize=3416848 CPUtime=4828.71 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 53659 0 0 0 482810 61 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=3416848 CPUtime=4833.84 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 43111 0 0 0 483344 40 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=3416848 CPUtime=4839.86 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 99363 0 0 0 483894 92 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4653154 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=3416848 CPUtime=4831.69 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 79464 0 0 0 483097 72 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=3416848 CPUtime=4840.16 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 73523 0 0 0 483959 57 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=3416848 CPUtime=4818.21 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 98671 0 0 0 481721 100 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4632289 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3416848 CPUtime=4841.49 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 189415 0 0 0 484032 117 0 0 25 0 8 0 629058281 3498852352 789882 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38670
Current children cumulated vsize (KiB) 3416848

[startup+4902.3 s]
/proc/loadavg: 8.06 8.06 8.03 9/359 30638
/proc/meminfo: memFree=1044936/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3423848 CPUtime=39149.2 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 810212 0 0 0 3914234 684 0 0 25 0 8 0 629058270 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670291 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/29240/statm: 855962 804358 466 189 0 850311 0
[pid=29240/tid=29252] ppid=29238 vsize=3423848 CPUtime=4888.71 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 55560 0 0 0 488807 64 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=3423848 CPUtime=4893.83 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 45166 0 0 0 489341 42 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=3423848 CPUtime=4899.84 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 101495 0 0 0 489890 94 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=3423848 CPUtime=4891.67 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 79548 0 0 0 489095 72 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=3423848 CPUtime=4900.15 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 73533 0 0 0 489958 57 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=3423848 CPUtime=4877.5 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 98748 0 0 0 487650 100 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3423848 CPUtime=4901.48 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 191750 0 0 0 490030 118 0 0 25 0 8 0 629058281 3506020352 804358 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39149.2
Current children cumulated vsize (KiB) 3423848

[startup+4962.3 s]
/proc/loadavg: 8.02 8.04 8.02 9/359 30640
/proc/meminfo: memFree=989616/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3445336 CPUtime=39628.3 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 823117 0 0 0 3962140 694 0 0 25 0 8 0 629058270 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4333305 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/29240/statm: 861334 817263 466 189 0 855683 0
[pid=29240/tid=29252] ppid=29238 vsize=3445336 CPUtime=4948.71 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 56553 0 0 0 494806 65 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=3445336 CPUtime=4953.83 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 46246 0 0 0 495340 43 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=3445336 CPUtime=4959.8 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 103184 0 0 0 495884 96 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320916 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=3445336 CPUtime=4951.68 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 80905 0 0 0 495094 74 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=3445336 CPUtime=4960.1 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 73659 0 0 0 495953 57 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=3445336 CPUtime=4936.75 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 100015 0 0 0 493574 101 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3445336 CPUtime=4961.46 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 194449 0 0 0 496027 119 0 0 25 0 8 0 629058281 3528024064 817263 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39628.3
Current children cumulated vsize (KiB) 3445336



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.9 s]
/proc/loadavg: 8.01 8.03 8.02 9/359 30641
/proc/meminfo: memFree=989512/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3445336 CPUtime=40000.6 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 823118 0 0 0 3999364 696 0 0 25 0 8 0 629058270 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/29240/statm: 861334 817264 466 189 0 855683 0
[pid=29240/tid=29252] ppid=29238 vsize=3445336 CPUtime=4995.32 cores=0-7
/proc/29240/task/29252/stat : 29252 (claspmt) R 29238 29240 29199 0 -1 4202560 56553 0 0 0 499467 65 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29253] ppid=29238 vsize=3445336 CPUtime=5000.44 cores=0-7
/proc/29240/task/29253/stat : 29253 (claspmt) R 29238 29240 29199 0 -1 4202560 46246 0 0 0 500001 43 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29254] ppid=29238 vsize=3445336 CPUtime=5006.38 cores=0-7
/proc/29240/task/29254/stat : 29254 (claspmt) R 29238 29240 29199 0 -1 4202560 103184 0 0 0 500541 97 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29255] ppid=29238 vsize=3445336 CPUtime=4997.8 cores=0-7
/proc/29240/task/29255/stat : 29255 (claspmt) R 29238 29240 29199 0 -1 4202560 80905 0 0 0 499706 74 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29256] ppid=29238 vsize=3445336 CPUtime=5006.72 cores=0-7
/proc/29240/task/29256/stat : 29256 (claspmt) R 29238 29240 29199 0 -1 4202560 73659 0 0 0 500615 57 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29257] ppid=29238 vsize=3445336 CPUtime=4983.26 cores=0-7
/proc/29240/task/29257/stat : 29257 (claspmt) R 29238 29240 29199 0 -1 4202560 100016 0 0 0 498224 102 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4653233 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29258] ppid=29238 vsize=3445336 CPUtime=5008.06 cores=0-7
/proc/29240/task/29258/stat : 29258 (claspmt) R 29238 29240 29199 0 -1 4202560 194449 0 0 0 500687 119 0 0 25 0 8 0 629058281 3528024064 817264 33554432000 4194304 4964704 140736449135712 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 3445336

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.02 s]
/proc/loadavg: 8.01 8.03 8.02 9/359 30641
/proc/meminfo: memFree=989512/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=3444392 CPUtime=40000.7 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 823128 0 0 0 3999378 696 0 0 25 0 1 0 629058270 3527057408 816910 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367203377 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/29240/statm: 861098 816910 476 189 0 855447 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 3444392

[startup+5009.4 s]
/proc/loadavg: 8.01 8.03 8.02 2/353 30642
/proc/meminfo: memFree=3861524/32950928 swapFree=62390276/67111528
[pid=29240] ppid=29238 vsize=516716 CPUtime=40001.1 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 29199 0 -1 4202496 823128 0 0 0 3999403 710 0 0 25 0 1 0 629058270 529117184 35877 33554432000 4194304 4964704 140736449135712 18446744073709551615 251367590023 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/29240/statm: 129179 35877 476 189 0 123528 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 516716

Child status: 0
Real time (s): 5009.42
CPU time (s): 40001.1
CPU user time (s): 39994
CPU system time (s): 7.10992
CPU usage (%): 798.518
Max. virtual memory (cumulated for all children) (KiB): 3445336

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39994
system time used= 7.10992
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 823129
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 149521
involuntary context switches= 198564

runsolver used 16.7065 second user time and 42.5075 second system time

The end

Launcher Data 

Begin job on node113 at 2011-05-03 04:04:36
IDJOB=3322556
IDBENCH=83797
IDSOLVER=1711
FILE ID=node113/3322556-1304388276
RUNJOBID= node113-1304388276-29213
PBS_JOBID= 13181821
Free space on /tmp= 73496 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT03/handmade/gomes/bqwh/bqwh.60.1152.shuffled-as.sat03-1650.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322556-1304388276/watcher-3322556-1304388276 -o /tmp/evaluation-result-3322556-1304388276/solver-3322556-1304388276 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322556-1304388276.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 98d6317b259165bb4fb7a39e783668e0
RANDOM SEED=200810673

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:       4353236 kB
Buffers:         52552 kB
Cached:         275548 kB
SwapCached:      53820 kB
Active:       17903720 kB
Inactive:     10452172 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:       4353236 kB
SwapTotal:    67111528 kB
SwapFree:     62390272 kB
Dirty:            2780 kB
Writeback:           0 kB
AnonPages:    27984344 kB
Mapped:          20376 kB
Slab:            82380 kB
PageTables:      92176 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 87040732 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73516 MiB
End job on node113 at 2011-05-03 05:28:07