Trace number 3381206

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.5 5007.51

General information on the benchmark

NameSAT03/industrial/kukula/addm_bench/
am_7_7.shuffled-as.sat03-363.cnf
MD5SUMc7f51e1ee7556440ab29e1c4bfe29538
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables4264
Number of clauses14751
Sum of the clauses size40035
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 24214
Number of clauses of size 310535
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...509183/instance-3381206-1305509183.cnf
0.01/0.02	c Solving...
40000.34/5007.41	
40000.34/5007.41	*** INTERRUPTED! ***
40000.34/5007.41	*** Shutting down threads...
40000.34/5007.50	s UNKNOWN
40000.34/5007.50	
40000.34/5007.50	c INTERRUPTED   : 1
40000.34/5007.50	c Models        : 0+    
40000.34/5007.50	c Time          : 5007.428s (Solving: 5007.40s 1st Model: 0.00s Unsat: 0.00s)
40000.34/5007.50	c CPU Time      : 40000.400s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381206-1305509183/watcher-3381206-1305509183 -o /tmp/evaluation-result-3381206-1305509183/solver-3381206-1305509183 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381206-1305509183.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.05 8.04 8.00 1/295 15409
/proc/meminfo: memFree=26126548/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=23024 CPUtime=0 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 1124 0 0 0 0 0 0 0 25 0 1 0 741148928 23576576 980 33554432000 4194304 4964800 140737313234832 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/15409/statm: 5756 981 399 189 0 630 0

[startup+0.016256 s]
/proc/loadavg: 8.05 8.04 8.00 1/295 15409
/proc/meminfo: memFree=26126548/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=23328 CPUtime=0.01 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 1193 0 0 0 1 0 0 0 25 0 1 0 741148928 23887872 1049 33554432000 4194304 4964800 140737313234832 18446744073709551615 4607198 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/15409/statm: 5832 1049 403 189 0 706 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 23328

[startup+0.118284 s]
/proc/loadavg: 8.05 8.04 8.00 1/295 15409
/proc/meminfo: memFree=26126548/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=253376 CPUtime=0.59 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 2895 0 0 0 58 1 0 0 25 0 8 0 741148928 259457024 2446 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669503 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 63344 2446 464 189 0 57693 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 253376

[startup+0.300157 s]
/proc/loadavg: 8.05 8.04 8.00 1/295 15409
/proc/meminfo: memFree=26126548/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=320232 CPUtime=2.04 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 4556 0 0 0 203 1 0 0 25 0 8 0 741148928 327917568 4107 33554432000 4194304 4964800 140737313234832 18446744073709551615 4362204 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 80058 4107 464 189 0 74407 0
Current children cumulated CPU time (s) 2.04
Current children cumulated vsize (KiB) 320232

[startup+0.701084 s]
/proc/loadavg: 8.05 8.04 8.00 1/295 15409
/proc/meminfo: memFree=26126548/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=320760 CPUtime=5.25 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 5578 0 0 0 523 2 0 0 25 0 8 0 741148928 328458240 5129 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669011 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 80190 5129 464 189 0 74539 0
Current children cumulated CPU time (s) 5.25
Current children cumulated vsize (KiB) 320760

[startup+1.50094 s]
/proc/loadavg: 8.05 8.04 8.00 10/303 15417
/proc/meminfo: memFree=26106404/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=322480 CPUtime=11.63 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 7762 0 0 0 1159 4 0 0 25 0 8 0 741148928 330219520 7313 33554432000 4194304 4964800 140737313234832 18446744073709551615 4653304 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 80620 7315 464 189 0 74969 0
[pid=15409/tid=15411] ppid=15407 vsize=322480 CPUtime=1.45 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 555 0 0 0 145 0 0 0 19 0 8 0 741148930 330219520 7316 33554432000 4194304 4964800 140737313234832 18446744073709551615 4627269 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=322480 CPUtime=1.43 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 951 0 0 0 143 0 0 0 19 0 8 0 741148930 330219520 7316 33554432000 4194304 4964800 140737313234832 18446744073709551615 260020056773 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=322480 CPUtime=1.44 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 652 0 0 0 144 0 0 0 19 0 8 0 741148930 330219520 7316 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=322480 CPUtime=1.44 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 822 0 0 0 144 0 0 0 19 0 8 0 741148930 330219520 7316 33554432000 4194304 4964800 140737313234832 18446744073709551615 4683957 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=322480 CPUtime=1.46 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 816 0 0 0 146 0 0 0 20 0 8 0 741148930 330219520 7316 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=322480 CPUtime=1.42 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 679 0 0 0 142 0 0 0 18 0 8 0 741148930 330219520 7316 33554432000 4194304 4964800 140737313234832 18446744073709551615 4342232 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=322480 CPUtime=1.45 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 1498 0 0 0 144 1 0 0 25 0 8 0 741148930 330219520 7317 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.63
Current children cumulated vsize (KiB) 322480

[startup+3.10064 s]
/proc/loadavg: 8.05 8.04 8.00 9/303 15417
/proc/meminfo: memFree=26094144/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=325720 CPUtime=24.42 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 10792 0 0 0 2436 6 0 0 25 0 8 0 741148928 333537280 10289 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669272 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 81430 10289 464 189 0 75779 0
[pid=15409/tid=15411] ppid=15407 vsize=325720 CPUtime=3.03 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 750 0 0 0 303 0 0 0 24 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=325720 CPUtime=3.03 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 1393 0 0 0 303 0 0 0 24 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=325720 CPUtime=3.04 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 929 0 0 0 304 0 0 0 24 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320911 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=325720 CPUtime=3.04 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 1115 0 0 0 304 0 0 0 24 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 4332103 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=325720 CPUtime=3.05 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 1265 0 0 0 305 0 0 0 25 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 260020057121 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=325720 CPUtime=3.02 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 835 0 0 0 302 0 0 0 21 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 4684067 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=325720 CPUtime=3.04 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 2495 0 0 0 303 1 0 0 25 0 8 0 741148930 333537280 10290 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320993 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.42
Current children cumulated vsize (KiB) 325720

[startup+6.30106 s]
/proc/loadavg: 8.04 8.04 8.00 9/303 15417
/proc/meminfo: memFree=26080252/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=327164 CPUtime=49.98 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 13847 0 0 0 4990 8 0 0 25 0 8 0 741148928 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4362287 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 81791 13263 466 189 0 76140 0
[pid=15409/tid=15411] ppid=15407 vsize=327164 CPUtime=6.2 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 988 0 0 0 620 0 0 0 25 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=327164 CPUtime=6.23 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 1886 0 0 0 623 0 0 0 25 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=327164 CPUtime=6.24 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 1128 0 0 0 624 0 0 0 25 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=327164 CPUtime=6.25 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 1539 0 0 0 624 1 0 0 25 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669280 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=327164 CPUtime=6.25 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 1577 0 0 0 625 0 0 0 25 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669280 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=327164 CPUtime=6.23 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 1078 0 0 0 622 1 0 0 18 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669312 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=327164 CPUtime=6.25 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 3398 0 0 0 623 2 0 0 25 0 8 0 741148930 335015936 13263 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.98
Current children cumulated vsize (KiB) 327164

[startup+12.7009 s]
/proc/loadavg: 8.04 8.04 8.00 9/303 15417
/proc/meminfo: memFree=26063552/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=331060 CPUtime=101.1 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 17410 0 0 0 10100 10 0 0 25 0 8 0 741148928 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670328 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 82765 16826 466 189 0 77114 0
[pid=15409/tid=15411] ppid=15407 vsize=331060 CPUtime=12.53 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 1267 0 0 0 1253 0 0 0 25 0 8 0 741148930 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=331060 CPUtime=12.63 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 2285 0 0 0 1263 0 0 0 25 0 8 0 741148930 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=331060 CPUtime=12.64 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 1333 0 0 0 1264 0 0 0 25 0 8 0 741148930 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=331060 CPUtime=12.64 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 2211 0 0 0 1263 1 0 0 25 0 8 0 741148930 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=331060 CPUtime=12.64 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 2078 0 0 0 1264 0 0 0 25 0 8 0 741148930 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669376 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=331060 CPUtime=12.62 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 1301 0 0 0 1261 1 0 0 25 0 8 0 741148930 339005440 16826 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320897 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=331060 CPUtime=12.64 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 4508 0 0 0 1262 2 0 0 25 0 8 0 741148930 339005440 16829 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.1
Current children cumulated vsize (KiB) 331060

[startup+25.5005 s]
/proc/loadavg: 8.03 8.03 8.00 9/303 15418
/proc/meminfo: memFree=26040272/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=337128 CPUtime=203.34 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 23079 0 0 0 20319 15 0 0 25 0 8 0 741148928 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4654449 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 84282 22495 466 189 0 78631 0
[pid=15409/tid=15411] ppid=15407 vsize=337128 CPUtime=25.21 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 1817 0 0 0 2520 1 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=337128 CPUtime=25.42 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 2849 0 0 0 2541 1 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=337128 CPUtime=25.43 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 1639 0 0 0 2543 0 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=337128 CPUtime=25.44 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 3071 0 0 0 2542 2 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=337128 CPUtime=25.45 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 2640 0 0 0 2544 1 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=337128 CPUtime=25.41 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 1768 0 0 0 2540 1 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=337128 CPUtime=25.43 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 6414 0 0 0 2540 3 0 0 25 0 8 0 741148930 345219072 22495 33554432000 4194304 4964800 140737313234832 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.34
Current children cumulated vsize (KiB) 337128

[startup+51.1008 s]
/proc/loadavg: 8.08 8.05 8.00 9/303 15418
/proc/meminfo: memFree=26006364/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=342728 CPUtime=407.84 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 31601 0 0 0 40763 21 0 0 18 0 8 0 741148928 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670335 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 85682 30992 466 189 0 80031 0
[pid=15409/tid=15411] ppid=15407 vsize=342728 CPUtime=50.74 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 2520 0 0 0 5072 2 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=342728 CPUtime=51.02 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 3756 0 0 0 5100 2 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=342728 CPUtime=50.84 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 2314 0 0 0 5083 1 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=342728 CPUtime=51.03 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 4185 0 0 0 5100 3 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=342728 CPUtime=51.04 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 3592 0 0 0 5102 2 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=342728 CPUtime=51 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 2442 0 0 0 5098 2 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=342728 CPUtime=51.03 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 9259 0 0 0 5098 5 0 0 25 0 8 0 741148930 350953472 30992 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321076 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 407.84

################
# More data... #
################


[startup+4662.3 s]
/proc/loadavg: 8.02 8.08 8.08 9/303 16028
/proc/meminfo: memFree=25632788/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=688236 CPUtime=37243.2 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 132227 0 0 0 3723781 543 0 0 25 0 8 0 741148928 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172059 129624 466 189 0 166408 0
[pid=15409/tid=15411] ppid=15407 vsize=688236 CPUtime=4639.6 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 24883 0 0 0 463883 77 0 0 25 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=688236 CPUtime=4655.7 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17045 0 0 0 465506 64 0 0 25 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=688236 CPUtime=4651.62 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10205 0 0 0 465101 61 0 0 25 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4362938 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=688236 CPUtime=4660.84 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15779 0 0 0 466008 76 0 0 25 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=688236 CPUtime=4661.52 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 13940 0 0 0 466093 59 0 0 25 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=688236 CPUtime=4651.96 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 9930 0 0 0 465130 66 0 0 20 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=688236 CPUtime=4660.52 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30115 0 0 0 465970 82 0 0 25 0 8 0 741148930 704753664 129624 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37243.2
Current children cumulated vsize (KiB) 688236

[startup+4722.31 s]
/proc/loadavg: 8.05 8.08 8.08 9/303 16030
/proc/meminfo: memFree=25631760/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=688792 CPUtime=37722.7 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 132654 0 0 0 3771720 547 0 0 25 0 8 0 741148928 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172198 130036 466 189 0 166547 0
[pid=15409/tid=15411] ppid=15407 vsize=688792 CPUtime=4699.03 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 25097 0 0 0 469825 78 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=688792 CPUtime=4715.7 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17103 0 0 0 471505 65 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=688792 CPUtime=4711.62 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10244 0 0 0 471101 61 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=688792 CPUtime=4720.83 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15854 0 0 0 472007 76 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=688792 CPUtime=4721.53 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 13964 0 0 0 472093 60 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670155 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=688792 CPUtime=4711.96 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 9930 0 0 0 471130 66 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=688792 CPUtime=4720.51 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30127 0 0 0 471969 82 0 0 25 0 8 0 741148930 705323008 130036 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37722.7
Current children cumulated vsize (KiB) 688792

[startup+4782.3 s]
/proc/loadavg: 8.02 8.06 8.08 9/303 16032
/proc/meminfo: memFree=25630832/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=689248 CPUtime=38201.9 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 133054 0 0 0 3819641 553 0 0 25 0 8 0 741148928 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670346 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172312 130421 466 189 0 166661 0
[pid=15409/tid=15411] ppid=15407 vsize=689248 CPUtime=4758.44 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 25257 0 0 0 475766 78 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669376 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=689248 CPUtime=4775.68 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17170 0 0 0 477501 67 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320953 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=689248 CPUtime=4771.6 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10250 0 0 0 477098 62 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=689248 CPUtime=4780.81 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15854 0 0 0 478005 76 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=689248 CPUtime=4781.5 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 14005 0 0 0 478090 60 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=689248 CPUtime=4771.95 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 9934 0 0 0 477128 67 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=689248 CPUtime=4780.5 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30245 0 0 0 477967 83 0 0 25 0 8 0 741148930 705789952 130421 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38201.9
Current children cumulated vsize (KiB) 689248

[startup+4842.3 s]
/proc/loadavg: 8.00 8.05 8.07 9/303 16033
/proc/meminfo: memFree=25629672/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=689648 CPUtime=38681.3 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 133380 0 0 0 3867570 558 0 0 25 0 8 0 741148928 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669280 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172412 130738 466 189 0 166761 0
[pid=15409/tid=15411] ppid=15407 vsize=689648 CPUtime=4817.85 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 25368 0 0 0 481706 79 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=689648 CPUtime=4835.66 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17208 0 0 0 483499 67 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=689648 CPUtime=4831.59 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10296 0 0 0 483097 62 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=689648 CPUtime=4840.8 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15855 0 0 0 484003 77 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=689648 CPUtime=4841.49 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 14089 0 0 0 484088 61 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=689648 CPUtime=4831.93 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 9938 0 0 0 483126 67 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4332288 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=689648 CPUtime=4840.49 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30249 0 0 0 483965 84 0 0 25 0 8 0 741148930 706199552 130738 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38681.3
Current children cumulated vsize (KiB) 689648

[startup+4902.3 s]
/proc/loadavg: 8.09 8.07 8.07 9/303 16035
/proc/meminfo: memFree=25628492/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=689900 CPUtime=39160.6 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 133648 0 0 0 3915497 565 0 0 25 0 8 0 741148928 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172475 131003 466 189 0 166824 0
[pid=15409/tid=15411] ppid=15407 vsize=689900 CPUtime=4877.28 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 25506 0 0 0 487648 80 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=689900 CPUtime=4895.65 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17235 0 0 0 489497 68 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=689900 CPUtime=4891.57 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10310 0 0 0 489094 63 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=689900 CPUtime=4900.78 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15875 0 0 0 490000 78 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=689900 CPUtime=4901.48 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 14103 0 0 0 490087 61 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=689900 CPUtime=4891.91 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 9938 0 0 0 489123 68 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=689900 CPUtime=4900.48 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30249 0 0 0 489963 85 0 0 25 0 8 0 741148930 706457600 131003 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39160.6
Current children cumulated vsize (KiB) 689900

[startup+4962.3 s]
/proc/loadavg: 8.13 8.08 8.08 9/303 16036
/proc/meminfo: memFree=25627092/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=690280 CPUtime=39640 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 134043 0 0 0 3963426 570 0 0 25 0 8 0 741148928 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670634 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172570 131379 466 189 0 166919 0
[pid=15409/tid=15411] ppid=15407 vsize=690280 CPUtime=4937.18 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 25673 0 0 0 493637 81 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=690280 CPUtime=4955.26 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17254 0 0 0 495457 69 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=690280 CPUtime=4951.48 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10326 0 0 0 495085 63 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4644138 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=690280 CPUtime=4960.76 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15916 0 0 0 495997 79 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=690280 CPUtime=4961.46 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 14150 0 0 0 496084 62 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=690280 CPUtime=4951.91 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 9974 0 0 0 495122 69 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668805 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=690280 CPUtime=4960.46 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30278 0 0 0 495960 86 0 0 25 0 8 0 741148930 706846720 131379 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39640
Current children cumulated vsize (KiB) 690280



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.4 s]
/proc/loadavg: 8.11 8.08 8.08 9/303 16038
/proc/meminfo: memFree=25625864/32951124 swapFree=67083432/67111528
[pid=15409] ppid=15407 vsize=690456 CPUtime=40000.3 cores=0-7
/proc/15409/stat : 15409 (claspmt) R 15407 15409 15045 0 -1 4202496 134393 0 0 0 3999459 575 0 0 25 0 8 0 741148928 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4669557 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/15409/statm: 172614 131612 466 189 0 166963 0
[pid=15409/tid=15411] ppid=15407 vsize=690456 CPUtime=4982.27 cores=0-7
/proc/15409/task/15411/stat : 15411 (claspmt) R 15407 15409 15045 0 -1 4202560 25812 0 0 0 498145 82 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 260020500974 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15412] ppid=15407 vsize=690456 CPUtime=4999.94 cores=0-7
/proc/15409/task/15412/stat : 15412 (claspmt) R 15407 15409 15045 0 -1 4202560 17265 0 0 0 499924 70 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15409/tid=15413] ppid=15407 vsize=690456 CPUtime=4996.58 cores=0-7
/proc/15409/task/15413/stat : 15413 (claspmt) R 15407 15409 15045 0 -1 4202560 10328 0 0 0 499595 63 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15409/tid=15414] ppid=15407 vsize=690456 CPUtime=5005.86 cores=0-7
/proc/15409/task/15414/stat : 15414 (claspmt) R 15407 15409 15045 0 -1 4202560 15955 0 0 0 500506 80 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15409/tid=15415] ppid=15407 vsize=690456 CPUtime=5006.57 cores=0-7
/proc/15409/task/15415/stat : 15415 (claspmt) R 15407 15409 15045 0 -1 4202560 14187 0 0 0 500595 62 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15409/tid=15416] ppid=15407 vsize=690456 CPUtime=4997.02 cores=0-7
/proc/15409/task/15416/stat : 15416 (claspmt) R 15407 15409 15045 0 -1 4202560 10014 0 0 0 499632 70 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4670446 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15409/tid=15417] ppid=15407 vsize=690456 CPUtime=5005.56 cores=0-7
/proc/15409/task/15417/stat : 15417 (claspmt) R 15407 15409 15045 0 -1 4202560 30354 0 0 0 500470 86 0 0 25 0 8 0 741148930 707026944 131612 33554432000 4194304 4964800 140737313234832 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 690456

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 5007.51
CPU time (s): 40000.5
CPU user time (s): 39994.7
CPU system time (s): 5.78012
CPU usage (%): 798.81
Max. virtual memory (cumulated for all children) (KiB): 690456

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39994.7
system time used= 5.78012
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 134404
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 666224
involuntary context switches= 194535

runsolver used 13.473 second user time and 32.697 second system time

The end

Launcher Data 

Begin job on node110 at 2011-05-16 03:26:23
IDJOB=3381206
IDBENCH=84237
IDSOLVER=1865
FILE ID=node110/3381206-1305509183
RUNJOBID= node110-1305503705-15059
PBS_JOBID= 13325472
Free space on /tmp= 73612 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT03/industrial/kukula/addm_bench/am_7_7.shuffled-as.sat03-363.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381206-1305509183/watcher-3381206-1305509183 -o /tmp/evaluation-result-3381206-1305509183/solver-3381206-1305509183 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381206-1305509183.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c7f51e1ee7556440ab29e1c4bfe29538
RANDOM SEED=1818051926

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      26126952 kB
Buffers:        268360 kB
Cached:        2826596 kB
SwapCached:       1416 kB
Active:        3498232 kB
Inactive:      3000892 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      26126952 kB
SwapTotal:    67111528 kB
SwapFree:     67083432 kB
Dirty:            1204 kB
Writeback:           0 kB
AnonPages:     3403792 kB
Mapped:          26516 kB
Slab:           236008 kB
PageTables:      23428 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 63403948 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73688 MiB
End job on node110 at 2011-05-16 04:49:53