Trace number 3380703

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.4 5012.75

General information on the benchmark

NameCRAFTED/rbsat/random/forced/
rbsat-v2640c305320gyes10.cnf
MD5SUMe4e4b653016e98f210aa8da2214e783a
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2640
Number of clauses305320
Sum of the clauses size613120
Maximum clause length33
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2305240
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 580

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...487557/instance-3380703-1305487557.cnf
1.59/1.69	c Solving...
40000.38/5011.82	
40000.38/5011.82	*** INTERRUPTED! ***
40000.38/5011.82	*** Shutting down threads...
40001.32/5012.74	s UNKNOWN
40001.32/5012.74	
40001.32/5012.74	c INTERRUPTED   : 1
40001.32/5012.74	c Models        : 0+    
40001.32/5012.74	c Time          : 5011.824s (Solving: 5010.13s 1st Model: 0.00s Unsat: 0.00s)
40001.32/5012.74	c CPU Time      : 40000.440s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380703-1305487557/watcher-3380703-1305487557 -o /tmp/evaluation-result-3380703-1305487557/solver-3380703-1305487557 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380703-1305487557.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.42 7.89 7.79 1/463 12932
/proc/meminfo: memFree=29760668/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=24048 CPUtime=0 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 1572 0 0 0 0 0 0 0 25 0 1 0 660354410 24625152 1190 33554432000 4194304 4964800 140733766398368 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/12932/statm: 6012 1191 386 189 0 886 0

[startup+0.0536131 s]
/proc/loadavg: 7.42 7.89 7.79 1/463 12932
/proc/meminfo: memFree=29760668/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=31300 CPUtime=0.04 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 4158 0 0 0 4 0 0 0 25 0 1 0 660354410 32051200 2918 33554432000 4194304 4964800 140733766398368 18446744073709551615 251813900997 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/12932/statm: 7825 2921 386 189 0 2699 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 31300

[startup+0.100643 s]
/proc/loadavg: 7.42 7.89 7.79 1/463 12932
/proc/meminfo: memFree=29760668/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=39612 CPUtime=0.09 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 6386 0 0 0 8 1 0 0 25 0 1 0 660354410 40562688 5122 33554432000 4194304 4964800 140733766398368 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/12932/statm: 9903 5126 397 189 0 4777 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 39612

[startup+0.300605 s]
/proc/loadavg: 7.42 7.89 7.79 1/463 12932
/proc/meminfo: memFree=29760668/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=39876 CPUtime=0.29 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 6442 0 0 0 28 1 0 0 25 0 1 0 660354410 40833024 5178 33554432000 4194304 4964800 140733766398368 18446744073709551615 4607641 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/12932/statm: 9969 5178 399 189 0 4843 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 39876

[startup+0.700547 s]
/proc/loadavg: 7.42 7.89 7.79 1/463 12932
/proc/meminfo: memFree=29760668/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=39876 CPUtime=0.69 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 6442 0 0 0 68 1 0 0 25 0 1 0 660354410 40833024 5178 33554432000 4194304 4964800 140733766398368 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/12932/statm: 9969 5178 399 189 0 4843 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 39876

[startup+1.50033 s]
/proc/loadavg: 7.42 7.89 7.79 2/464 12933
/proc/meminfo: memFree=29739464/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=43176 CPUtime=1.49 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 7296 0 0 0 148 1 0 0 25 0 1 0 660354410 44212224 6032 33554432000 4194304 4964800 140733766398368 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/12932/statm: 10794 6032 400 189 0 5668 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 43176

[startup+3.10107 s]
/proc/loadavg: 7.42 7.89 7.79 9/471 12940
/proc/meminfo: memFree=29719816/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=532584 CPUtime=11.21 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 20859 0 0 0 1114 7 0 0 17 0 8 0 660354410 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 133146 18976 464 189 0 127495 0
[pid=12932/tid=12934] ppid=12930 vsize=532584 CPUtime=1.15 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 1877 0 0 0 115 0 0 0 25 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670650 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=532584 CPUtime=1.15 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 1618 0 0 0 115 0 0 0 25 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=532584 CPUtime=1.15 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 1389 0 0 0 115 0 0 0 17 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=532584 CPUtime=1.14 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 1768 0 0 0 114 0 0 0 17 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=532584 CPUtime=1.38 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 1036 0 0 0 138 0 0 0 19 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=532584 CPUtime=1.15 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 1813 0 0 0 115 0 0 0 25 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4465565 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=532584 CPUtime=1.15 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 2464 0 0 0 115 0 0 0 17 0 8 0 660354580 545366016 18976 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627878 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.21
Current children cumulated vsize (KiB) 532584

[startup+6.30053 s]
/proc/loadavg: 7.47 7.89 7.79 9/471 12941
/proc/meminfo: memFree=29640544/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=535884 CPUtime=36.74 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 37247 0 0 0 3661 13 0 0 25 0 8 0 660354410 548745216 35364 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320981 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 133971 35364 464 189 0 128320 0
[pid=12932/tid=12934] ppid=12930 vsize=535884 CPUtime=4.35 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 4198 0 0 0 434 1 0 0 25 0 8 0 660354580 548745216 35364 33554432000 4194304 4964800 140733766398368 18446744073709551615 4653299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=535884 CPUtime=4.35 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 3425 0 0 0 435 0 0 0 25 0 8 0 660354580 548745216 35364 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=535884 CPUtime=4.35 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 3419 0 0 0 434 1 0 0 23 0 8 0 660354580 548745216 35364 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=535884 CPUtime=4.28 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 3812 0 0 0 427 1 0 0 23 0 8 0 660354580 548745216 35365 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=535884 CPUtime=4.57 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 1527 0 0 0 457 0 0 0 25 0 8 0 660354580 548745216 35365 33554432000 4194304 4964800 140733766398368 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=535884 CPUtime=4.35 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 3800 0 0 0 434 1 0 0 25 0 8 0 660354580 548745216 35365 33554432000 4194304 4964800 140733766398368 18446744073709551615 251813901345 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=535884 CPUtime=4.35 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 7344 0 0 0 433 2 0 0 23 0 8 0 660354580 548745216 35365 33554432000 4194304 4964800 140733766398368 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 36.74
Current children cumulated vsize (KiB) 535884

[startup+12.7005 s]
/proc/loadavg: 7.51 7.89 7.79 9/471 12941
/proc/meminfo: memFree=29518540/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=551208 CPUtime=87.82 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 64402 0 0 0 8755 27 0 0 25 0 8 0 660354410 564436992 62519 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670264 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 137802 62520 466 189 0 132151 0
[pid=12932/tid=12934] ppid=12930 vsize=551208 CPUtime=10.75 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 6841 0 0 0 1073 2 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670423 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=551208 CPUtime=10.76 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 5107 0 0 0 1075 1 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=551208 CPUtime=10.75 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 5824 0 0 0 1073 2 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=551208 CPUtime=10.59 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 7134 0 0 0 1055 4 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670351 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=551208 CPUtime=10.96 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 4732 0 0 0 1094 2 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627873 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=551208 CPUtime=10.75 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 5500 0 0 0 1072 3 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=551208 CPUtime=10.75 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 16630 0 0 0 1069 6 0 0 25 0 8 0 660354580 564436992 62520 33554432000 4194304 4964800 140733766398368 18446744073709551615 251813900997 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 87.82
Current children cumulated vsize (KiB) 551208

[startup+25.5013 s]
/proc/loadavg: 7.62 7.90 7.79 9/471 12941
/proc/meminfo: memFree=29385136/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=571744 CPUtime=190.02 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 96743 0 0 0 18956 46 0 0 25 0 8 0 660354410 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 142936 94860 466 189 0 137285 0
[pid=12932/tid=12934] ppid=12930 vsize=571744 CPUtime=23.55 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 10173 0 0 0 2351 4 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670144 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=571744 CPUtime=23.55 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 9456 0 0 0 2351 4 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4660835 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=571744 CPUtime=23.55 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 8204 0 0 0 2351 4 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=571744 CPUtime=23.18 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 11082 0 0 0 2313 5 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4684739 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=571744 CPUtime=23.77 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 7093 0 0 0 2373 4 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=571744 CPUtime=23.56 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 7807 0 0 0 2351 5 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=571744 CPUtime=23.56 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 27669 0 0 0 2345 11 0 0 25 0 8 0 660354580 585465856 94860 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 190.02
Current children cumulated vsize (KiB) 571744

[startup+51.101 s]
/proc/loadavg: 7.75 7.91 7.80 9/471 12942
/proc/meminfo: memFree=29341960/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=572404 CPUtime=394.4 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 108380 0 0 0 39384 56 0 0 25 0 8 0 660354410 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 251813937770 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 143101 106492 466 189 0 137450 0
[pid=12932/tid=12934] ppid=12930 vsize=572404 CPUtime=49.14 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 10917 0 0 0 4910 4 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=572404 CPUtime=49.15 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 10828 0 0 0 4910 5 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=572404 CPUtime=49.15 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 8479 0 0 0 4911 4 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=572404 CPUtime=48.39 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 12628 0 0 0 4831 8 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=572404 CPUtime=49.36 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 7322 0 0 0 4932 4 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=572404 CPUtime=49.14 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 8267 0 0 0 4909 5 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=572404 CPUtime=49.15 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 34251 0 0 0 4899 16 0 0 25 0 8 0 660354580 586141696 106492 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 394.4
Current children cumulated vsize (KiB) 572404

[startup+102.318 s]
/proc/loadavg: 7.89 7.92 7.81 9/471 12943
/proc/meminfo: memFree=29213116/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=847224 CPUtime=803.29 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 139493 0 0 0 80245 84 0 0 25 0 8 0 660354410 867557376 137499 33554432000 4194304 4964800 140733766398368 18446744073709551615 4342157 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 211806 137499 466 189 0 206155 0
[pid=12932/tid=12934] ppid=12930 vsize=847224 CPUtime=100.35 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 13056 0 0 0 10028 7 0 0 25 0 8 0 660354580 867557376 137499 33554432000 4194304 4964800 140733766398368 18446744073709551615 4632401 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=847224 CPUtime=100.37 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 12970 0 0 0 10028 9 0 0 25 0 8 0 660354580 867557376 137499 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=847224 CPUtime=100.37 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 8632 0 0 0 10033 4 0 0 25 0 8 0 660354580 867557376 137499 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0

################
# More data... #
################

[pid=12932] ppid=12930 vsize=3266292 CPUtime=37689.1 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 751990 0 0 0 3768044 862 0 0 25 0 8 0 660354410 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 816573 741992 466 189 0 810922 0
[pid=12932/tid=12934] ppid=12930 vsize=3266292 CPUtime=4700.74 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 59883 0 0 0 469999 75 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=3266292 CPUtime=4719.41 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 36444 0 0 0 471907 34 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=3266292 CPUtime=4720.31 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 27712 0 0 0 471997 34 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=3266292 CPUtime=4686.12 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 31662 0 0 0 468580 32 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=3266292 CPUtime=4719.35 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 30761 0 0 0 471902 33 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=3266292 CPUtime=4719.6 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 32211 0 0 0 471919 41 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4332152 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=3266292 CPUtime=4701.57 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 485593 0 0 0 469603 554 0 0 25 0 8 0 660354580 3344683008 741992 33554432000 4194304 4964800 140733766398368 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37689.1
Current children cumulated vsize (KiB) 3266292

[startup+4782.3 s]
/proc/loadavg: 8.10 8.08 8.01 9/471 13080
/proc/meminfo: memFree=26757144/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3266292 CPUtime=38168 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 759651 0 0 0 3815923 875 0 0 25 0 8 0 660354410 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 816573 749653 466 189 0 810922 0
[pid=12932/tid=12934] ppid=12930 vsize=3266292 CPUtime=4760.73 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 60441 0 0 0 475996 77 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=3266292 CPUtime=4779.39 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 36445 0 0 0 477905 34 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=3266292 CPUtime=4780.29 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 27712 0 0 0 477995 34 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=3266292 CPUtime=4746.11 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 31715 0 0 0 474578 33 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=3266292 CPUtime=4779.31 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 30761 0 0 0 477898 33 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=3266292 CPUtime=4779.58 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 32211 0 0 0 477917 41 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670363 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=3266292 CPUtime=4760.64 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 492642 0 0 0 475499 565 0 0 25 0 8 0 660354580 3344683008 749653 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38168
Current children cumulated vsize (KiB) 3266292

[startup+4842.3 s]
/proc/loadavg: 8.03 8.06 8.00 9/471 13082
/proc/meminfo: memFree=26717320/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3278064 CPUtime=38647.1 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 769612 0 0 0 3863817 888 0 0 25 0 8 0 660354410 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627252 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 819516 759614 466 189 0 813865 0
[pid=12932/tid=12934] ppid=12930 vsize=3278064 CPUtime=4820.73 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 61029 0 0 0 481995 78 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=3278064 CPUtime=4839.39 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 36445 0 0 0 483905 34 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4378544 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=3278064 CPUtime=4840.29 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 27712 0 0 0 483995 34 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=3278064 CPUtime=4806.1 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 33553 0 0 0 480575 35 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=3278064 CPUtime=4839.3 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 30761 0 0 0 483897 33 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=3278064 CPUtime=4839.58 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 32212 0 0 0 483917 41 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=3278064 CPUtime=4819.7 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 500176 0 0 0 481397 573 0 0 25 0 8 0 660354580 3356737536 759614 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38647.1
Current children cumulated vsize (KiB) 3278064

[startup+4902.3 s]
/proc/loadavg: 8.01 8.05 8.00 9/471 13083
/proc/meminfo: memFree=26713956/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3278064 CPUtime=39126.1 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 770400 0 0 0 3911719 889 0 0 25 0 8 0 660354410 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 819516 760402 466 189 0 813865 0
[pid=12932/tid=12934] ppid=12930 vsize=3278064 CPUtime=4880.73 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 61568 0 0 0 487994 79 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=3278064 CPUtime=4899.39 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 36451 0 0 0 489905 34 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=3278064 CPUtime=4900.29 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 27797 0 0 0 489995 34 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=3278064 CPUtime=4866.07 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 33556 0 0 0 486572 35 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=3278064 CPUtime=4899.29 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 30761 0 0 0 489896 33 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=3278064 CPUtime=4899.58 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 32212 0 0 0 489917 41 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=3278064 CPUtime=4878.79 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 500306 0 0 0 487305 574 0 0 25 0 8 0 660354580 3356737536 760402 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39126.1
Current children cumulated vsize (KiB) 3278064

[startup+4962.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/471 13085
/proc/meminfo: memFree=26705992/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3278064 CPUtime=39605.1 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 772993 0 0 0 3959611 894 0 0 25 0 8 0 660354410 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320981 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 819516 762387 466 189 0 813865 0
[pid=12932/tid=12934] ppid=12930 vsize=3278064 CPUtime=4940.67 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 62245 0 0 0 493987 80 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=3278064 CPUtime=4959.39 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 36451 0 0 0 495905 34 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=3278064 CPUtime=4960.29 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 27797 0 0 0 495995 34 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320934 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=3278064 CPUtime=4926.07 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 33563 0 0 0 492572 35 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=3278064 CPUtime=4959.27 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 30761 0 0 0 495894 33 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=3278064 CPUtime=4959.58 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 32220 0 0 0 495917 41 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4321042 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=3278064 CPUtime=4937.84 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 500310 0 0 0 493210 574 0 0 25 0 8 0 660354580 3356737536 762387 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39605.1
Current children cumulated vsize (KiB) 3278064



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.8 s]
/proc/loadavg: 8.02 8.04 8.00 9/471 13086
/proc/meminfo: memFree=26705980/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3278064 CPUtime=40000.4 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 772994 0 0 0 3999143 895 0 0 25 0 8 0 660354410 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4320944 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/12932/statm: 819516 762379 466 189 0 813865 0
[pid=12932/tid=12934] ppid=12930 vsize=3278064 CPUtime=4990.19 cores=0-7
/proc/12932/task/12934/stat : 12934 (claspmt) R 12930 12932 12906 0 -1 4202560 62245 0 0 0 498939 80 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12932/tid=12935] ppid=12930 vsize=3278064 CPUtime=5008.91 cores=0-7
/proc/12932/task/12935/stat : 12935 (claspmt) R 12930 12932 12906 0 -1 4202560 36451 0 0 0 500857 34 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12932/tid=12936] ppid=12930 vsize=3278064 CPUtime=5009.79 cores=0-7
/proc/12932/task/12936/stat : 12936 (claspmt) R 12930 12932 12906 0 -1 4202560 27797 0 0 0 500944 35 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12932/tid=12937] ppid=12930 vsize=3278064 CPUtime=4975.58 cores=0-7
/proc/12932/task/12937/stat : 12937 (claspmt) R 12930 12932 12906 0 -1 4202560 33563 0 0 0 497523 35 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12932/tid=12938] ppid=12930 vsize=3278064 CPUtime=5008.78 cores=0-7
/proc/12932/task/12938/stat : 12938 (claspmt) R 12930 12932 12906 0 -1 4202560 30761 0 0 0 500845 33 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12932/tid=12939] ppid=12930 vsize=3278064 CPUtime=5009.09 cores=0-7
/proc/12932/task/12939/stat : 12939 (claspmt) R 12930 12932 12906 0 -1 4202560 32220 0 0 0 500868 41 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12932/tid=12940] ppid=12930 vsize=3278064 CPUtime=4986.58 cores=0-7
/proc/12932/task/12940/stat : 12940 (claspmt) R 12930 12932 12906 0 -1 4202560 500311 0 0 0 498084 574 0 0 25 0 8 0 660354580 3356737536 762379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 3278064

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5012.2 s]
/proc/loadavg: 8.02 8.04 8.00 9/471 13086
/proc/meminfo: memFree=26705980/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3248564 CPUtime=40000.8 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 773004 0 0 0 3999186 895 0 0 25 0 1 0 660354410 3326529536 753379 33554432000 4194304 4964800 140733766398368 18446744073709551615 4646607 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/12932/statm: 812141 753379 476 189 0 806490 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 3248564

[startup+5012.6 s]
/proc/loadavg: 8.02 8.04 8.00 2/465 13087
/proc/meminfo: memFree=26828044/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=3166620 CPUtime=40001.2 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 773004 0 0 0 3999227 896 0 0 25 0 1 0 660354410 3242618880 714571 33554432000 4194304 4964800 140733766398368 18446744073709551615 251814283399 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/12932/statm: 791655 714565 476 189 0 786004 0
Current children cumulated CPU time (s) 40001.2
Current children cumulated vsize (KiB) 3166620

[startup+5012.72 s]
/proc/loadavg: 8.02 8.04 8.00 2/465 13087
/proc/meminfo: memFree=26828044/32950928 swapFree=41934240/67111528
[pid=12932] ppid=12930 vsize=925468 CPUtime=40001.3 cores=0-7
/proc/12932/stat : 12932 (claspmt) R 12930 12932 12906 0 -1 4202496 773004 0 0 0 3999227 905 0 0 25 0 1 0 660354410 947679232 141610 33554432000 4194304 4964800 140733766398368 18446744073709551615 251814283399 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/12932/statm: 231367 141610 476 189 0 225716 0
Current children cumulated CPU time (s) 40001.3
Current children cumulated vsize (KiB) 925468

Child status: 0
Real time (s): 5012.75
CPU time (s): 40001.4
CPU user time (s): 39992.3
CPU system time (s): 9.08762
CPU usage (%): 797.993
Max. virtual memory (cumulated for all children) (KiB): 3278064

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.3
system time used= 9.08762
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 773005
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 46509
involuntary context switches= 190616

runsolver used 21.1158 second user time and 56.3774 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-15 21:25:57
IDJOB=3380703
IDBENCH=71425
IDSOLVER=1865
FILE ID=node126/3380703-1305487557
RUNJOBID= node126-1305487557-12920
PBS_JOBID= 13325392
Free space on /tmp= 73308 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2640c305320gyes10.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380703-1305487557/watcher-3380703-1305487557 -o /tmp/evaluation-result-3380703-1305487557/solver-3380703-1305487557 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380703-1305487557.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e4e4b653016e98f210aa8da2214e783a
RANDOM SEED=237233266

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      29761072 kB
Buffers:        192048 kB
Cached:        1844716 kB
SwapCached:      49688 kB
Active:         633896 kB
Inactive:      2285900 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      29761072 kB
SwapTotal:    67111528 kB
SwapFree:     41934240 kB
Dirty:            4684 kB
Writeback:           0 kB
AnonPages:      846728 kB
Mapped:          16440 kB
Slab:           111696 kB
PageTables:      86556 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 133632388 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73308 MiB
End job on node126 at 2011-05-15 22:49:31