Trace number 3322419

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 35563.4 5101.49

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-21x39.cnf
MD5SUMe7679ca49bb432c9b90305d907365c4e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables3276
Number of clauses628173
Sum of the clauses size2502864
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 24914
Number of clauses of size 30
Number of clauses of size 4623259
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...384952/instance-3322419-1304384952.cnf
58.69/58.72	c Solving...
35561.85/5100.01	
35561.85/5100.01	*** INTERRUPTED! ***
35561.85/5100.01	*** Shutting down threads...
35563.28/5101.48	s UNKNOWN
35563.28/5101.48	
35563.28/5101.48	c INTERRUPTED   : 1
35563.28/5101.48	c Models        : 0+    
35563.28/5101.48	c Time          : 5100.017s (Solving: 5041.29s 1st Model: 0.00s Unsat: 0.00s)
35563.28/5101.48	c CPU Time      : 35561.890s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322419-1304384952/watcher-3322419-1304384952 -o /tmp/evaluation-result-3322419-1304384952/solver-3322419-1304384952 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322419-1304384952.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.00 1.00 1.00 2/298 32253
/proc/meminfo: memFree=6165220/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=22792 CPUtime=0 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1056 0 0 0 0 0 0 0 22 0 1 0 282686535 23339008 924 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660320435 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 5698 925 386 189 0 572 0

[startup+0.0437869 s]
/proc/loadavg: 1.00 1.00 1.00 2/298 32253
/proc/meminfo: memFree=6165220/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=28676 CPUtime=0.03 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 2955 0 0 0 3 0 0 0 22 0 1 0 282686535 29364224 2387 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660323013 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 7169 2388 386 189 0 2043 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 28676

[startup+0.100781 s]
/proc/loadavg: 1.00 1.00 1.00 2/298 32253
/proc/meminfo: memFree=6165220/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=42120 CPUtime=0.09 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 6471 0 0 0 8 1 0 0 22 0 1 0 282686535 43130880 5225 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660359637 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 10530 5247 386 189 0 5404 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 42120

[startup+0.300745 s]
/proc/loadavg: 1.00 1.00 1.00 2/298 32253
/proc/meminfo: memFree=6165220/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=75724 CPUtime=0.29 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 16727 0 0 0 26 3 0 0 23 0 1 0 282686535 77541376 13930 33554432000 4194304 4964704 140733862522928 18446744073709551615 4615501 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 18931 13931 397 189 0 13805 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 75724

[startup+0.700671 s]
/proc/loadavg: 1.00 1.00 1.00 2/298 32253
/proc/meminfo: memFree=6165220/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=76520 CPUtime=0.69 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 16959 0 0 0 66 3 0 0 25 0 1 0 282686535 78356480 14162 33554432000 4194304 4964704 140733862522928 18446744073709551615 4607513 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 19130 14162 399 189 0 14004 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 76520

[startup+1.50053 s]
/proc/loadavg: 1.00 1.00 1.00 3/299 32254
/proc/meminfo: memFree=6109792/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=76520 CPUtime=1.49 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 16959 0 0 0 146 3 0 0 25 0 1 0 282686535 78356480 14162 33554432000 4194304 4964704 140733862522928 18446744073709551615 4607585 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 19130 14162 399 189 0 14004 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 76520

[startup+3.10024 s]
/proc/loadavg: 1.08 1.02 1.01 3/299 32254
/proc/meminfo: memFree=6108700/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=76520 CPUtime=3.09 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 16959 0 0 0 306 3 0 0 25 0 1 0 282686535 78356480 14162 33554432000 4194304 4964704 140733862522928 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 19130 14162 399 189 0 14004 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 76520

[startup+6.3007 s]
/proc/loadavg: 1.08 1.02 1.01 3/299 32254
/proc/meminfo: memFree=6107588/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=77980 CPUtime=6.29 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 17331 0 0 0 626 3 0 0 25 0 1 0 282686535 79851520 14534 33554432000 4194304 4964704 140733862522928 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 19495 14534 400 189 0 14369 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 77980

[startup+12.7006 s]
/proc/loadavg: 1.22 1.05 1.02 3/299 32255
/proc/meminfo: memFree=6096480/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=89108 CPUtime=12.69 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 20112 0 0 0 1265 4 0 0 25 0 1 0 282686535 91246592 17315 33554432000 4194304 4964704 140733862522928 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 22277 17315 400 189 0 17151 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 89108

[startup+25.5003 s]
/proc/loadavg: 1.34 1.08 1.02 3/299 32255
/proc/meminfo: memFree=6077908/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=106352 CPUtime=25.49 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 24418 0 0 0 2543 6 0 0 25 0 1 0 282686535 108904448 21621 33554432000 4194304 4964704 140733862522928 18446744073709551615 4600585 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 26588 21621 400 189 0 21462 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 106352

[startup+51.1008 s]
/proc/loadavg: 1.56 1.15 1.05 3/299 32256
/proc/meminfo: memFree=6060524/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=119844 CPUtime=51.09 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 27801 0 0 0 5101 8 0 0 25 0 1 0 282686535 122720256 25004 33554432000 4194304 4964704 140733862522928 18446744073709551615 4607575 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/32253/statm: 29961 25004 400 189 0 24835 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 119844

[startup+102.312 s]
/proc/loadavg: 5.56 2.29 1.44 10/306 32264
/proc/meminfo: memFree=5197164/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=1169416 CPUtime=360.18 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 243521 0 0 0 35889 129 0 0 25 0 8 0 282686535 1197481984 239345 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670260 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/32253/statm: 292354 239345 466 189 0 286703 0
[pid=32253/tid=32257] ppid=32251 vsize=1169416 CPUtime=33.44 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 27727 0 0 0 3329 15 0 0 25 0 8 0 282692408 1197481984 239345 33554432000 4194304 4964704 140733862522928 18446744073709551615 4627540 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=1169416 CPUtime=38.72 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 28513 0 0 0 3856 16 0 0 25 0 8 0 282692408 1197481984 239345 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=1169416 CPUtime=40.47 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 20001 0 0 0 4037 10 0 0 25 0 8 0 282692408 1197481984 239345 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=1169416 CPUtime=37.21 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 23947 0 0 0 3706 15 0 0 21 0 8 0 282692408 1197481984 239345 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=1169416 CPUtime=36.33 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 24080 0 0 0 3621 12 0 0 25 0 8 0 282692408 1197481984 239345 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=1169416 CPUtime=40.72 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 22636 0 0 0 4060 12 0 0 21 0 8 0 282692408 1197481984 239346 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=1169416 CPUtime=36.84 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 31187 0 0 0 3666 18 0 0 25 0 8 0 282692408 1197481984 239346 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 360.18
Current children cumulated vsize (KiB) 1169416

[startup+162.301 s]
/proc/loadavg: 7.77 3.52 1.91 10/306 32266
/proc/meminfo: memFree=4971576/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=1473536 CPUtime=778.95 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 298572 0 0 0 77725 170 0 0 25 0 8 0 282686535 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/32253/statm: 368384 294319 466 189 0 362733 0
[pid=32253/tid=32257] ppid=32251 vsize=1473536 CPUtime=78.25 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 37804 0 0 0 7802 23 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660319210 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=1473536 CPUtime=86.49 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 38321 0 0 0 8627 22 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=1473536 CPUtime=98.32 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 21875 0 0 0 9820 12 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4653526 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=1473536 CPUtime=96.42 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 29282 0 0 0 9622 20 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=1473536 CPUtime=90.13 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 32045 0 0 0 8996 17 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=1473536 CPUtime=97.36 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 24467 0 0 0 9722 14 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=1473536 CPUtime=89.95 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 39457 0 0 0 8974 21 0 0 25 0 8 0 282692408 1508900864 294319 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 778.95
Current children cumulated vsize (KiB) 1473536

[startup+222.301 s]
/proc/loadavg: 8.58 4.53 2.36 10/306 32267
/proc/meminfo: memFree=4858820/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=1686016 CPUtime=1198.18 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 324052 0 0 0 119623 195 0 0 25 0 8 0 282686535 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/32253/statm: 421504 319538 466 189 0 415853 0
[pid=32253/tid=32257] ppid=32251 vsize=1686016 CPUtime=117.7 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 40057 0 0 0 11745 25 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=1686016 CPUtime=139.76 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 39728 0 0 0 13952 24 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=1686016 CPUtime=157.83 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 22623 0 0 0 15771 12 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=1686016 CPUtime=151.43 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 34948 0 0 0 15115 28 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4684027 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=1686016 CPUtime=138.68 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 34650 0 0 0 13849 19 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=1686016 CPUtime=151.37 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 25203 0 0 0 15121 16 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=1686016 CPUtime=144.95 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 47122 0 0 0 14468 27 0 0 25 0 8 0 282692408 1726480384 319538 33554432000 4194304 4964704 140733862522928 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1198.18
Current children cumulated vsize (KiB) 1686016

[startup+282.301 s]
/proc/loadavg: 8.87 5.29 2.74 10/306 32269
/proc/meminfo: memFree=4801740/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=1686500 CPUtime=1617.56 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 337263 0 0 0 161545 211 0 0 25 0 8 0 282686535 1726976000 332684 33554432000 4194304 4964704 140733862522928 18446744073709551615 4660755 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/32253/statm: 421625 332684 466 189 0 415974 0
[pid=32253/tid=32257] ppid=32251 vsize=1686500 CPUtime=166.54 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 40554 0 0 0 16628 26 0 0 25 0 8 0 282692408 1726976000 332684 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=1686500 CPUtime=183.98 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 42026 0 0 0 18371 27 0 0 25 0 8 0 282692408 1726976000 332684 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=1686500 CPUtime=217.82 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 23020 0 0 0 21769 13 0 0 25 0 8 0 282692408 1726976000 332684 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=1686500 CPUtime=211.43 cores=0-7

################
# More data... #
################

[pid=32253] ppid=32251 vsize=4277192 CPUtime=33724.9 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1027998 0 0 0 3371397 1092 0 0 21 0 8 0 282686535 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670291 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/32253/statm: 1069298 1013846 466 189 0 1063647 0
[pid=32253/tid=32257] ppid=32251 vsize=4277192 CPUtime=4077.64 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 114675 0 0 0 407644 120 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=4277192 CPUtime=4168.9 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 223658 0 0 0 416644 246 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670252 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=4277192 CPUtime=4502.83 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 65239 0 0 0 450217 66 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660767214 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=4277192 CPUtime=4108.84 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 101382 0 0 0 410768 116 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=4277192 CPUtime=4498.57 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 86803 0 0 0 449776 81 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=4277192 CPUtime=4561.26 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 50008 0 0 0 456079 47 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=4277192 CPUtime=3700.45 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 205502 0 0 0 369802 243 0 0 25 0 8 0 282692408 4379844608 1013846 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 33724.9
Current children cumulated vsize (KiB) 4277192

[startup+4902.3 s]
/proc/loadavg: 9.05 9.06 8.98 10/306 443
/proc/meminfo: memFree=1482948/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4277192 CPUtime=34144.3 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1028398 0 0 0 3413336 1093 0 0 25 0 8 0 282686535 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4654493 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/32253/statm: 1069298 1014163 466 189 0 1063647 0
[pid=32253/tid=32257] ppid=32251 vsize=4277192 CPUtime=4111.6 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 114737 0 0 0 411040 120 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=4277192 CPUtime=4228.9 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 223669 0 0 0 422644 246 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=4277192 CPUtime=4528.87 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 65339 0 0 0 452821 66 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=4277192 CPUtime=4168.84 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 101411 0 0 0 416768 116 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=4277192 CPUtime=4557.98 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 86859 0 0 0 455716 82 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=4277192 CPUtime=4621.26 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 50008 0 0 0 462079 47 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=4277192 CPUtime=3760.45 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 205632 0 0 0 375802 243 0 0 25 0 8 0 282692408 4379844608 1014163 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 34144.3
Current children cumulated vsize (KiB) 4277192

[startup+4962.3 s]
/proc/loadavg: 9.02 9.05 8.98 10/306 445
/proc/meminfo: memFree=1465472/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4342728 CPUtime=34563.7 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1030857 0 0 0 3455273 1096 0 0 25 0 8 0 282686535 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4362248 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/32253/statm: 1085682 1016607 466 189 0 1080031 0
[pid=32253/tid=32257] ppid=32251 vsize=4342728 CPUtime=4141.59 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 114740 0 0 0 414039 120 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=4342728 CPUtime=4288.9 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 223669 0 0 0 428644 246 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=4342728 CPUtime=4558.88 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 65339 0 0 0 455822 66 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=4342728 CPUtime=4228.85 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 103260 0 0 0 422767 118 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=4342728 CPUtime=4617.38 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 87305 0 0 0 461656 82 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=4342728 CPUtime=4681.27 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 50008 0 0 0 468080 47 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=4342728 CPUtime=3820.45 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 205780 0 0 0 381802 243 0 0 25 0 8 0 282692408 4446953472 1016607 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 34563.7
Current children cumulated vsize (KiB) 4342728

[startup+5022.3 s]
/proc/loadavg: 9.05 9.05 8.99 10/306 446
/proc/meminfo: memFree=1432252/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4408264 CPUtime=35011.1 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1038173 0 0 0 3500003 1108 0 0 25 0 8 0 282686535 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660359840 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/32253/statm: 1102066 1023923 466 189 0 1096415 0
[pid=32253/tid=32257] ppid=32251 vsize=4408264 CPUtime=4171.6 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 114745 0 0 0 417040 120 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=4408264 CPUtime=4348.9 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 223669 0 0 0 434644 246 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=4408264 CPUtime=4616.88 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 65339 0 0 0 461622 66 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=4408264 CPUtime=4288.85 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 105995 0 0 0 428764 121 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=4408264 CPUtime=4676.78 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 91648 0 0 0 467588 90 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=4408264 CPUtime=4741.27 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 50008 0 0 0 474080 47 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=4408264 CPUtime=3880.45 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 205996 0 0 0 387802 243 0 0 25 0 8 0 282692408 4514062336 1023923 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35011.1
Current children cumulated vsize (KiB) 4408264

[startup+5082.3 s]
/proc/loadavg: 9.02 9.04 8.99 10/306 448
/proc/meminfo: memFree=1381584/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4415324 CPUtime=35438.1 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1049019 0 0 0 3542685 1122 0 0 25 0 8 0 282686535 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/32253/statm: 1103831 1034738 466 189 0 1098180 0
[pid=32253/tid=32257] ppid=32251 vsize=4415324 CPUtime=4201.79 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 114758 0 0 0 420059 120 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=4415324 CPUtime=4408.9 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 225347 0 0 0 440644 246 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=4415324 CPUtime=4676.88 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 65777 0 0 0 467621 67 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=4415324 CPUtime=4348.85 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 107435 0 0 0 434762 123 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=4415324 CPUtime=4736.18 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 94806 0 0 0 473522 96 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=4415324 CPUtime=4801.26 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 50008 0 0 0 480078 48 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4320916 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=4415324 CPUtime=3917.84 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 206023 0 0 0 391541 243 0 0 25 0 8 0 282692408 4521291776 1034738 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35438.1
Current children cumulated vsize (KiB) 4415324



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.01 9.04 8.99 10/306 448
/proc/meminfo: memFree=1378292/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4480860 CPUtime=35561.9 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1049148 0 0 0 3555063 1122 0 0 25 0 8 0 282686535 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/32253/statm: 1120215 1034867 466 189 0 1114564 0
[pid=32253/tid=32257] ppid=32251 vsize=4480860 CPUtime=4210.68 cores=0-7
/proc/32253/task/32257/stat : 32257 (claspmt) R 32251 32253 30950 0 -1 4202560 114763 0 0 0 420948 120 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=32253/tid=32258] ppid=32251 vsize=4480860 CPUtime=4426.61 cores=0-7
/proc/32253/task/32258/stat : 32258 (claspmt) R 32251 32253 30950 0 -1 4202560 225440 0 0 0 442415 246 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=32253/tid=32259] ppid=32251 vsize=4480860 CPUtime=4694.59 cores=0-7
/proc/32253/task/32259/stat : 32259 (claspmt) R 32251 32253 30950 0 -1 4202560 65777 0 0 0 469392 67 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=32253/tid=32260] ppid=32251 vsize=4480860 CPUtime=4366.56 cores=0-7
/proc/32253/task/32260/stat : 32260 (claspmt) R 32251 32253 30950 0 -1 4202560 107456 0 0 0 436533 123 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=32253/tid=32261] ppid=32251 vsize=4480860 CPUtime=4753.71 cores=0-7
/proc/32253/task/32261/stat : 32261 (claspmt) R 32251 32253 30950 0 -1 4202560 94811 0 0 0 475275 96 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=32253/tid=32262] ppid=32251 vsize=4480860 CPUtime=4818.97 cores=0-7
/proc/32253/task/32262/stat : 32262 (claspmt) R 32251 32253 30950 0 -1 4202560 50008 0 0 0 481849 48 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=32253/tid=32263] ppid=32251 vsize=4480860 CPUtime=3926.67 cores=0-7
/proc/32253/task/32263/stat : 32263 (claspmt) R 32251 32253 30950 0 -1 4202560 206023 0 0 0 392424 243 0 0 25 0 8 0 282692408 4588400640 1034867 33554432000 4194304 4964704 140733862522928 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35561.8
Current children cumulated vsize (KiB) 4480860

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 9.01 9.04 8.99 3/300 449
/proc/meminfo: memFree=1423208/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4396456 CPUtime=35562.5 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1049158 0 0 0 3555126 1122 0 0 25 0 1 0 282686535 4501970944 1023533 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660317537 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/32253/statm: 1099114 1023533 476 189 0 1093463 0
Current children cumulated CPU time (s) 35562.5
Current children cumulated vsize (KiB) 4396456

[startup+5101 s]
/proc/loadavg: 9.01 9.04 8.99 3/300 449
/proc/meminfo: memFree=1423208/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=4396456 CPUtime=35562.9 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1049158 0 0 0 3555165 1122 0 0 25 0 1 0 282686535 4501970944 1022048 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660317387 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/32253/statm: 1099114 1022048 476 189 0 1093463 0
Current children cumulated CPU time (s) 35562.9
Current children cumulated vsize (KiB) 4396456

[startup+5101.4 s]
/proc/loadavg: 9.01 9.04 8.99 3/300 449
/proc/meminfo: memFree=1423208/32951124 swapFree=67111524/67111528
[pid=32253] ppid=32251 vsize=2699528 CPUtime=35563.3 cores=0-7
/proc/32253/stat : 32253 (claspmt) R 32251 32253 30950 0 -1 4202496 1049158 0 0 0 3555199 1129 0 0 25 0 1 0 282686535 2764316672 580523 33554432000 4194304 4964704 140733862522928 18446744073709551615 268660705415 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/32253/statm: 674882 580019 476 189 0 669231 0
Current children cumulated CPU time (s) 35563.3
Current children cumulated vsize (KiB) 2699528

Child status: 0
Real time (s): 5101.49
CPU time (s): 35563.4
CPU user time (s): 35552
CPU system time (s): 11.3853
CPU usage (%): 697.117
Max. virtual memory (cumulated for all children) (KiB): 4480860

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35552
system time used= 11.3853
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1049159
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 128067
involuntary context switches= 247403

runsolver used 14.7678 second user time and 36.1055 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-03 03:09:13
IDJOB=3322419
IDBENCH=83184
IDSOLVER=1711
FILE ID=node107/3322419-1304384952
RUNJOBID= node107-1304371484-30964
PBS_JOBID= 13181776
Free space on /tmp= 72900 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-21x39.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322419-1304384952/watcher-3322419-1304384952 -o /tmp/evaluation-result-3322419-1304384952/solver-3322419-1304384952 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322419-1304384952.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e7679ca49bb432c9b90305d907365c4e
RANDOM SEED=1250149254

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       6165500 kB
Buffers:        315796 kB
Cached:        3167692 kB
SwapCached:          4 kB
Active:       23114388 kB
Inactive:      3274984 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       6165500 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           15608 kB
Writeback:           0 kB
AnonPages:    22905748 kB
Mapped:          27464 kB
Slab:           266976 kB
PageTables:      62880 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 62813636 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72980 MiB
End job on node107 at 2011-05-03 04:34:15