Trace number 3381010

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.5 5010.09

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-7x39.cnf
MD5SUMf717554c4e7f802b3078c48464dbcd9d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1092
Number of clauses64155
Sum of the clauses size253344
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21638
Number of clauses of size 30
Number of clauses of size 462517
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...500269/instance-3381010-1305500269.cnf
1.10/1.15	c Solving...
40000.15/5009.82	
40000.15/5009.82	*** INTERRUPTED! ***
40000.15/5009.82	*** Shutting down threads...
40000.38/5010.08	s UNKNOWN
40000.38/5010.08	
40000.38/5010.08	c INTERRUPTED   : 1
40000.38/5010.08	c Models        : 0+    
40000.38/5010.08	c Time          : 5009.818s (Solving: 5008.67s 1st Model: 0.00s Unsat: 0.00s)
40000.38/5010.08	c CPU Time      : 40000.200s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381010-1305500269/watcher-3381010-1305500269 -o /tmp/evaluation-result-3381010-1305500269/solver-3381010-1305500269 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381010-1305500269.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.45 7.92 7.96 1/392 26502
/proc/meminfo: memFree=12258524/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=23044 CPUtime=0 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 1151 0 0 0 0 0 0 0 22 0 1 0 661629302 23597056 983 33554432000 4194304 4964800 140735058838832 18446744073709551615 212435677893 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/26502/statm: 5761 984 386 189 0 635 0

[startup+0.0318889 s]
/proc/loadavg: 7.45 7.92 7.96 1/392 26502
/proc/meminfo: memFree=12258524/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=26312 CPUtime=0.02 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 2117 0 0 0 2 0 0 0 22 0 1 0 661629302 26943488 1738 33554432000 4194304 4964800 140735058838832 18446744073709551615 212435677893 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/26502/statm: 6578 1747 397 189 0 1452 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 26312

[startup+0.100874 s]
/proc/loadavg: 7.45 7.92 7.96 1/392 26502
/proc/meminfo: memFree=12258524/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=27192 CPUtime=0.09 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 2318 0 0 0 9 0 0 0 22 0 1 0 661629302 27844608 1938 33554432000 4194304 4964800 140735058838832 18446744073709551615 4607616 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/26502/statm: 6798 1938 399 189 0 1672 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27192

[startup+0.300837 s]
/proc/loadavg: 7.45 7.92 7.96 1/392 26502
/proc/meminfo: memFree=12258524/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=28116 CPUtime=0.29 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 2578 0 0 0 29 0 0 0 23 0 1 0 661629302 28790784 2198 33554432000 4194304 4964800 140735058838832 18446744073709551615 4600389 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/26502/statm: 7029 2198 400 189 0 1903 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 28116

[startup+0.700759 s]
/proc/loadavg: 7.45 7.92 7.96 1/392 26502
/proc/meminfo: memFree=12258524/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=30756 CPUtime=0.69 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 3223 0 0 0 69 0 0 0 25 0 1 0 661629302 31494144 2843 33554432000 4194304 4964800 140735058838832 18446744073709551615 4612252 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/26502/statm: 7689 2843 400 189 0 2563 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 30756

[startup+1.50057 s]
/proc/loadavg: 7.45 7.92 7.96 2/393 26503
/proc/meminfo: memFree=12247860/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=528240 CPUtime=3.54 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 16962 0 0 0 345 9 0 0 18 0 8 0 661629302 540917760 16416 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320955 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 132060 16416 464 189 0 126409 0
Current children cumulated CPU time (s) 3.54
Current children cumulated vsize (KiB) 528240

[startup+3.10127 s]
/proc/loadavg: 7.49 7.92 7.96 9/400 26510
/proc/meminfo: memFree=12167952/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=533960 CPUtime=16.32 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 27029 0 0 0 1616 16 0 0 20 0 8 0 661629302 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4325336 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 133490 26483 466 189 0 127839 0
[pid=26502/tid=26504] ppid=26500 vsize=533960 CPUtime=1.89 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 2669 0 0 0 187 2 0 0 20 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4334301 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=533960 CPUtime=1.89 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 3162 0 0 0 188 1 0 0 19 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=533960 CPUtime=1.88 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 2948 0 0 0 186 2 0 0 20 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=533960 CPUtime=1.88 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 3375 0 0 0 186 2 0 0 20 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=533960 CPUtime=1.91 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 2261 0 0 0 190 1 0 0 20 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670321 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=533960 CPUtime=1.88 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 2486 0 0 0 187 1 0 0 20 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=533960 CPUtime=1.89 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 3644 0 0 0 187 2 0 0 20 0 8 0 661629417 546775040 26483 33554432000 4194304 4964800 140735058838832 18446744073709551615 4342157 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 16.32
Current children cumulated vsize (KiB) 533960

[startup+6.30066 s]
/proc/loadavg: 7.49 7.92 7.96 9/400 26511
/proc/meminfo: memFree=12118272/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=539764 CPUtime=41.85 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 38413 0 0 0 4162 23 0 0 25 0 8 0 661629302 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 134941 37867 466 189 0 129290 0
[pid=26502/tid=26504] ppid=26500 vsize=539764 CPUtime=5.08 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 3470 0 0 0 506 2 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=539764 CPUtime=5.08 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 5520 0 0 0 506 2 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4321013 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=539764 CPUtime=5.04 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 3890 0 0 0 501 3 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=539764 CPUtime=5.08 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 4737 0 0 0 505 3 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4660144 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=539764 CPUtime=5.11 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 3627 0 0 0 509 2 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=539764 CPUtime=5.08 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 3972 0 0 0 506 2 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=539764 CPUtime=5.09 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 5489 0 0 0 505 4 0 0 25 0 8 0 661629417 552718336 37867 33554432000 4194304 4964800 140735058838832 18446744073709551615 4325305 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 41.85
Current children cumulated vsize (KiB) 539764

[startup+12.7004 s]
/proc/loadavg: 7.53 7.92 7.96 9/400 26511
/proc/meminfo: memFree=12062384/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=548128 CPUtime=92.96 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 53254 0 0 0 9265 31 0 0 25 0 8 0 661629302 561283072 52708 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 137032 52709 466 189 0 131381 0
[pid=26502/tid=26504] ppid=26500 vsize=548128 CPUtime=11.48 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 4706 0 0 0 1145 3 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 4334298 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=548128 CPUtime=11.49 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 7906 0 0 0 1146 3 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=548128 CPUtime=11.35 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 4736 0 0 0 1132 3 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=548128 CPUtime=11.48 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 6580 0 0 0 1144 4 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=548128 CPUtime=11.5 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 5575 0 0 0 1147 3 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=548128 CPUtime=11.48 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 5428 0 0 0 1145 3 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=548128 CPUtime=11.49 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 9636 0 0 0 1143 6 0 0 25 0 8 0 661629417 561283072 52709 33554432000 4194304 4964800 140735058838832 18446744073709551615 212435678241 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 92.96
Current children cumulated vsize (KiB) 548128

[startup+25.501 s]
/proc/loadavg: 7.63 7.92 7.96 9/400 26511
/proc/meminfo: memFree=11957460/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=553636 CPUtime=195.18 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 76730 0 0 0 19471 47 0 0 25 0 8 0 661629302 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4660139 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 138409 75882 466 189 0 132758 0
[pid=26502/tid=26504] ppid=26500 vsize=553636 CPUtime=24.27 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 7139 0 0 0 2422 5 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=553636 CPUtime=24.29 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 10290 0 0 0 2424 5 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670321 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=553636 CPUtime=23.99 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 7368 0 0 0 2394 5 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=553636 CPUtime=24.28 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 8588 0 0 0 2423 5 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=553636 CPUtime=24.3 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 10184 0 0 0 2425 5 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=553636 CPUtime=24.28 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 7296 0 0 0 2424 4 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=553636 CPUtime=24.28 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 15163 0 0 0 2418 10 0 0 25 0 8 0 661629417 566923264 75882 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 195.18
Current children cumulated vsize (KiB) 553636

[startup+51.1012 s]
/proc/loadavg: 7.76 7.93 7.96 9/400 26512
/proc/meminfo: memFree=11882328/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=565580 CPUtime=399.6 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 96111 0 0 0 39895 65 0 0 25 0 8 0 661629302 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670321 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 141395 95231 466 189 0 135744 0
[pid=26502/tid=26504] ppid=26500 vsize=565580 CPUtime=49.86 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 7815 0 0 0 4980 6 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4342167 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=565580 CPUtime=49.88 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 13361 0 0 0 4980 8 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=565580 CPUtime=49.26 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 9909 0 0 0 4918 8 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=565580 CPUtime=49.87 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 10792 0 0 0 4980 7 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=565580 CPUtime=49.9 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 12069 0 0 0 4983 7 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=565580 CPUtime=49.87 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 7873 0 0 0 4982 5 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=565580 CPUtime=49.88 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 21409 0 0 0 4973 15 0 0 25 0 8 0 661629417 579153920 95231 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320848 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 399.6
Current children cumulated vsize (KiB) 565580

[startup+102.315 s]
/proc/loadavg: 7.90 7.94 7.96 9/400 26513
/proc/meminfo: memFree=11777492/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=714356 CPUtime=808.57 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 121824 0 0 0 80767 90 0 0 22 0 8 0 661629302 731500544 120799 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670346 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 178589 120799 466 189 0 172938 0
[pid=26502/tid=26504] ppid=26500 vsize=714356 CPUtime=101.04 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 10828 0 0 0 10096 8 0 0 25 0 8 0 661629417 731500544 120799 33554432000 4194304 4964800 140735058838832 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=714356 CPUtime=101.09 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 15383 0 0 0 10098 11 0 0 25 0 8 0 661629417 731500544 120799 33554432000 4194304 4964800 140735058838832 18446744073709551615 4653161 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=714356 CPUtime=100.13 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 12523 0 0 0 10001 12 0 0 25 0 8 0 661629417 731500544 120799 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 3 0 0 0

################
# More data... #
################

[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=4659.79 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 66490 0 0 0 465893 86 0 0 25 0 8 0 661629417 2175397888 425582 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=4659.8 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56923 0 0 0 465881 99 0 0 25 0 8 0 661629417 2175397888 425582 33554432000 4194304 4964800 140735058838832 18446744073709551615 4644896 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=4659.66 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38015 0 0 0 465925 41 0 0 25 0 8 0 661629417 2175397888 425582 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=4660.22 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42093 0 0 0 465975 47 0 0 25 0 8 0 661629417 2175397888 425582 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=4660.35 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 51498 0 0 0 465959 76 0 0 25 0 8 0 661629417 2175397888 425582 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4635.6 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 71554 0 0 0 463454 106 0 0 25 0 8 0 661629417 2175397888 425582 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37224.7
Current children cumulated vsize (KiB) 2124412

[startup+4722.32 s]
/proc/loadavg: 8.12 8.08 8.04 9/400 26660
/proc/meminfo: memFree=10518824/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2124412 CPUtime=37704 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 446908 0 0 0 3769754 645 0 0 25 0 8 0 661629302 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4689696 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 531103 433838 466 189 0 525452 0
[pid=26502/tid=26504] ppid=26500 vsize=2124412 CPUtime=4687.13 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 43053 0 0 0 468628 85 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=4719.8 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 69015 0 0 0 471890 90 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=4719.82 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56948 0 0 0 471883 99 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=4719.65 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38252 0 0 0 471924 41 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=4720.23 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42097 0 0 0 471976 47 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=4720.37 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 52106 0 0 0 471960 77 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4695.61 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 75349 0 0 0 469448 113 0 0 25 0 8 0 661629417 2175397888 433838 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37704
Current children cumulated vsize (KiB) 2124412

[startup+4782.3 s]
/proc/loadavg: 8.12 8.08 8.04 9/400 26662
/proc/meminfo: memFree=10511968/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2124412 CPUtime=38183.1 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 448782 0 0 0 3817658 650 0 0 25 0 8 0 661629302 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 531103 435604 466 189 0 525452 0
[pid=26502/tid=26504] ppid=26500 vsize=2124412 CPUtime=4746.35 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 43294 0 0 0 474550 85 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=4779.79 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 70337 0 0 0 477886 93 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=4779.79 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56948 0 0 0 477880 99 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=4779.62 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38267 0 0 0 477921 41 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=4780.22 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42314 0 0 0 477974 48 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4379895 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=4780.34 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 52106 0 0 0 477956 78 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4755.61 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 75425 0 0 0 475447 114 0 0 25 0 8 0 661629417 2175397888 435604 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38183.1
Current children cumulated vsize (KiB) 2124412

[startup+4842.3 s]
/proc/loadavg: 8.11 8.08 8.04 9/400 26664
/proc/meminfo: memFree=10502376/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2124412 CPUtime=38662.3 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 451255 0 0 0 3865573 655 0 0 25 0 8 0 661629302 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4333305 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 531103 437840 466 189 0 525452 0
[pid=26502/tid=26504] ppid=26500 vsize=2124412 CPUtime=4805.58 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 43463 0 0 0 480472 86 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=4839.79 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 70444 0 0 0 483885 94 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=4839.79 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56992 0 0 0 483880 99 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=4839.61 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38501 0 0 0 483920 41 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=4840.21 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42510 0 0 0 483973 48 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=4840.34 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 52106 0 0 0 483956 78 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4815.61 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 75503 0 0 0 481447 114 0 0 25 0 8 0 661629417 2175397888 437840 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38662.3
Current children cumulated vsize (KiB) 2124412

[startup+4902.3 s]
/proc/loadavg: 8.10 8.08 8.04 9/400 26665
/proc/meminfo: memFree=10501100/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2124412 CPUtime=39141.5 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 451543 0 0 0 3913493 657 0 0 25 0 8 0 661629302 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 531103 438128 466 189 0 525452 0
[pid=26502/tid=26504] ppid=26500 vsize=2124412 CPUtime=4864.82 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 43463 0 0 0 486396 86 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=4899.78 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 70448 0 0 0 489884 94 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=4899.79 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56992 0 0 0 489879 100 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=4899.59 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38714 0 0 0 489918 41 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=4900.21 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42519 0 0 0 489973 48 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=4900.33 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 52106 0 0 0 489955 78 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4660449 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4875.6 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 75565 0 0 0 487446 114 0 0 25 0 8 0 661629417 2175397888 438128 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39141.5
Current children cumulated vsize (KiB) 2124412

[startup+4962.3 s]
/proc/loadavg: 8.09 8.08 8.04 9/400 26667
/proc/meminfo: memFree=10499308/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2124412 CPUtime=39620.7 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 452088 0 0 0 3961405 661 0 0 25 0 8 0 661629302 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 531103 438455 466 189 0 525452 0
[pid=26502/tid=26504] ppid=26500 vsize=2124412 CPUtime=4924.03 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 43745 0 0 0 492316 87 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=4959.78 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 70609 0 0 0 495883 95 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=4959.78 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56992 0 0 0 495878 100 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=4959.58 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38730 0 0 0 495916 42 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=4960.2 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42524 0 0 0 495972 48 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=4960.33 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 52106 0 0 0 495954 79 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4935.61 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 75645 0 0 0 493446 115 0 0 25 0 8 0 661629417 2175397888 438455 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39620.7
Current children cumulated vsize (KiB) 2124412



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.8 s]
/proc/loadavg: 8.18 8.10 8.04 9/400 26668
/proc/meminfo: memFree=10497916/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2124412 CPUtime=40000.1 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 452482 0 0 0 3999352 663 0 0 25 0 8 0 661629302 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4627536 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/26502/statm: 531103 438849 466 189 0 525452 0
[pid=26502/tid=26504] ppid=26500 vsize=2124412 CPUtime=4970.92 cores=0-7
/proc/26502/task/26504/stat : 26504 (claspmt) R 26500 26502 26110 0 -1 4202560 43745 0 0 0 497005 87 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26502/tid=26505] ppid=26500 vsize=2124412 CPUtime=5007.29 cores=0-7
/proc/26502/task/26505/stat : 26505 (claspmt) R 26500 26502 26110 0 -1 4202560 70629 0 0 0 500634 95 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26502/tid=26506] ppid=26500 vsize=2124412 CPUtime=5007.3 cores=0-7
/proc/26502/task/26506/stat : 26506 (claspmt) R 26500 26502 26110 0 -1 4202560 56999 0 0 0 500630 100 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26502/tid=26507] ppid=26500 vsize=2124412 CPUtime=5007.08 cores=0-7
/proc/26502/task/26507/stat : 26507 (claspmt) R 26500 26502 26110 0 -1 4202560 38730 0 0 0 500666 42 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26502/tid=26508] ppid=26500 vsize=2124412 CPUtime=5007.72 cores=0-7
/proc/26502/task/26508/stat : 26508 (claspmt) R 26500 26502 26110 0 -1 4202560 42524 0 0 0 500724 48 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26502/tid=26509] ppid=26500 vsize=2124412 CPUtime=5007.85 cores=0-7
/proc/26502/task/26509/stat : 26509 (claspmt) R 26500 26502 26110 0 -1 4202560 52106 0 0 0 500706 79 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4362919 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26502/tid=26510] ppid=26500 vsize=2124412 CPUtime=4983.12 cores=0-7
/proc/26502/task/26510/stat : 26510 (claspmt) R 26500 26502 26110 0 -1 4202560 75942 0 0 0 498196 116 0 0 25 0 8 0 661629417 2175397888 438849 33554432000 4194304 4964800 140735058838832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 2124412

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010 s]
/proc/loadavg: 8.18 8.10 8.04 9/400 26668
/proc/meminfo: memFree=10497916/32951124 swapFree=56316152/67111528
[pid=26502] ppid=26500 vsize=2117776 CPUtime=40000.4 cores=0-7
/proc/26502/stat : 26502 (claspmt) R 26500 26502 26110 0 -1 4202496 452493 0 0 0 3999375 663 0 0 25 0 1 0 661629302 2168602624 434610 33554432000 4194304 4964800 140735058838832 18446744073709551615 212435672417 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/26502/statm: 529444 434610 476 189 0 523793 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 2117776

Child status: 0
Real time (s): 5010.09
CPU time (s): 40000.5
CPU user time (s): 39993.8
CPU system time (s): 6.71998
CPU usage (%): 798.398
Max. virtual memory (cumulated for all children) (KiB): 2124412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.8
system time used= 6.71998
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 452494
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 281252
involuntary context switches= 189019

runsolver used 17.5803 second user time and 46.198 second system time

The end

Launcher Data 

Begin job on node102 at 2011-05-16 00:57:50
IDJOB=3381010
IDBENCH=83170
IDSOLVER=1865
FILE ID=node102/3381010-1305500269
RUNJOBID= node102-1305494700-26124
PBS_JOBID= 13325433
Free space on /tmp= 73604 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-7x39.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381010-1305500269/watcher-3381010-1305500269 -o /tmp/evaluation-result-3381010-1305500269/solver-3381010-1305500269 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381010-1305500269.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f717554c4e7f802b3078c48464dbcd9d
RANDOM SEED=687008465

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      12258928 kB
Buffers:         74516 kB
Cached:         812244 kB
SwapCached:      33064 kB
Active:       12035692 kB
Inactive:      8438136 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      12258928 kB
SwapTotal:    67111528 kB
SwapFree:     56316152 kB
Dirty:            2340 kB
Writeback:           0 kB
AnonPages:    19565660 kB
Mapped:          20116 kB
Slab:            64576 kB
PageTables:      86204 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 102325412 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73604 MiB
End job on node102 at 2011-05-16 02:21:21