Trace number 3380899

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.7 5011.5

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k50-pdtvisns3p00-tseitin.cnf
MD5SUMa8ad0173824ed9404a8c153da07c5f9c
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark891.683
Satisfiable
(Un)Satisfiability was proved
Number of variables183325
Number of clauses546914
Sum of the clauses size1276130
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2364608
Number of clauses of size 3182304
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...494414/instance-3380899-1305494414.cnf
1.49/1.56	c Solving...
40000.63/5010.62	
40000.63/5010.62	*** INTERRUPTED! ***
40000.63/5010.62	*** Shutting down threads...
40001.54/5011.49	s UNKNOWN
40001.54/5011.49	
40001.54/5011.49	c INTERRUPTED   : 1
40001.54/5011.49	c Models        : 0+    
40001.54/5011.49	c Time          : 5010.625s (Solving: 5009.07s 1st Model: 0.00s Unsat: 0.00s)
40001.54/5011.49	c CPU Time      : 40000.770s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380899-1305494414/watcher-3380899-1305494414 -o /tmp/evaluation-result-3380899-1305494414/solver-3380899-1305494414 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380899-1305494414.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.09 6.69 7.00 2/401 1823
/proc/meminfo: memFree=23060828/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=20872 CPUtime=0 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 486 0 0 0 0 0 0 0 23 0 1 0 661038516 21372928 416 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476353281 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/1823/statm: 5218 416 353 189 0 92 0

[startup+0.0565329 s]
/proc/loadavg: 8.09 6.69 7.00 2/401 1823
/proc/meminfo: memFree=23060828/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=61100 CPUtime=0.04 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 12118 0 0 0 3 1 0 0 23 0 1 0 661038516 62566400 10324 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476355781 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1823/statm: 15275 10325 386 189 0 10149 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 61100

[startup+0.100524 s]
/proc/loadavg: 8.09 6.69 7.00 2/401 1823
/proc/meminfo: memFree=23060828/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=68748 CPUtime=0.09 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 14781 0 0 0 8 1 0 0 23 0 1 0 661038516 70397952 12129 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476355781 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1823/statm: 17187 12131 386 189 0 12061 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 68748

[startup+0.300489 s]
/proc/loadavg: 8.09 6.69 7.00 2/401 1823
/proc/meminfo: memFree=23060828/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=109304 CPUtime=0.29 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 27703 0 0 0 26 3 0 0 25 0 1 0 661038516 111927296 21696 33554432000 4194304 4964800 140733767918352 18446744073709551615 4618436 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1823/statm: 27326 21696 399 189 0 22200 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 109304

[startup+0.700406 s]
/proc/loadavg: 8.09 6.69 7.00 2/401 1823
/proc/meminfo: memFree=23060828/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=120000 CPUtime=0.69 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 30558 0 0 0 65 4 0 0 25 0 1 0 661038516 122880000 24551 33554432000 4194304 4964800 140733767918352 18446744073709551615 4624165 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1823/statm: 30000 24551 404 189 0 24874 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 120000

[startup+1.50026 s]
/proc/loadavg: 7.52 6.60 6.97 2/402 1824
/proc/meminfo: memFree=22958544/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=129732 CPUtime=1.49 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 32960 0 0 0 145 4 0 0 25 0 1 0 661038516 132845568 26953 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476352777 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1823/statm: 32433 26953 406 189 0 27307 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 129732

[startup+3.10088 s]
/proc/loadavg: 7.52 6.60 6.97 9/409 1831
/proc/meminfo: memFree=22813656/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=617300 CPUtime=10.52 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 82435 0 0 0 1027 25 0 0 16 0 8 0 661038516 632115200 62942 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 154325 62942 464 189 0 148674 0
[pid=1823/tid=1825] ppid=1821 vsize=617300 CPUtime=1.05 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 4998 0 0 0 104 1 0 0 17 0 8 0 661038673 632115200 62942 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=617300 CPUtime=1.1 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 7468 0 0 0 107 3 0 0 17 0 8 0 661038673 632115200 62942 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=617300 CPUtime=1.08 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 7990 0 0 0 104 4 0 0 16 0 8 0 661038673 632115200 62942 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=617300 CPUtime=1.07 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 5741 0 0 0 104 3 0 0 16 0 8 0 661038673 632115200 62943 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=617300 CPUtime=1.37 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 6859 0 0 0 135 2 0 0 19 0 8 0 661038673 632115200 62943 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476356129 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=617300 CPUtime=1.09 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 7993 0 0 0 107 2 0 0 17 0 8 0 661038673 632115200 62943 33554432000 4194304 4964800 140733767918352 18446744073709551615 4362996 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=617300 CPUtime=1.11 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 7411 0 0 0 108 3 0 0 16 0 8 0 661038673 632115200 62943 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 10.52
Current children cumulated vsize (KiB) 617300

[startup+6.30127 s]
/proc/loadavg: 7.52 6.60 6.97 9/409 1831
/proc/meminfo: memFree=22805968/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=617300 CPUtime=36.07 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 84159 0 0 0 3581 26 0 0 22 0 8 0 661038516 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4370266 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 154325 64666 464 189 0 148674 0
[pid=1823/tid=1825] ppid=1821 vsize=617300 CPUtime=4.21 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 5139 0 0 0 420 1 0 0 22 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=617300 CPUtime=4.29 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 7727 0 0 0 426 3 0 0 22 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476356129 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=617300 CPUtime=4.28 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 8259 0 0 0 424 4 0 0 22 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=617300 CPUtime=4.27 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 5864 0 0 0 424 3 0 0 22 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=617300 CPUtime=4.57 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 6889 0 0 0 455 2 0 0 25 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=617300 CPUtime=4.29 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 8430 0 0 0 427 2 0 0 22 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669294 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=617300 CPUtime=4.3 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 7763 0 0 0 427 3 0 0 22 0 8 0 661038673 632115200 64666 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 36.07
Current children cumulated vsize (KiB) 617300

[startup+12.701 s]
/proc/loadavg: 7.59 6.64 6.98 9/409 1831
/proc/meminfo: memFree=22794196/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=617988 CPUtime=87.18 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 87281 0 0 0 8688 30 0 0 25 0 8 0 661038516 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670180 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 154497 67788 464 189 0 148846 0
[pid=1823/tid=1825] ppid=1821 vsize=617988 CPUtime=10.52 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 5274 0 0 0 1050 2 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=617988 CPUtime=10.7 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 8444 0 0 0 1065 5 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669969 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=617988 CPUtime=10.67 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 8869 0 0 0 1063 4 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=617988 CPUtime=10.66 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 6246 0 0 0 1063 3 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=617988 CPUtime=10.97 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 6974 0 0 0 1095 2 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=617988 CPUtime=10.7 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 9028 0 0 0 1067 3 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=617988 CPUtime=10.7 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 8194 0 0 0 1067 3 0 0 25 0 8 0 661038673 632819712 67788 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668801 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 87.18
Current children cumulated vsize (KiB) 617988

[startup+25.5006 s]
/proc/loadavg: 7.66 6.69 6.99 9/409 1832
/proc/meminfo: memFree=22772604/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=619068 CPUtime=189.4 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 93179 0 0 0 18900 40 0 0 25 0 8 0 661038516 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668987 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 154767 73686 464 189 0 149116 0
[pid=1823/tid=1825] ppid=1821 vsize=619068 CPUtime=23.14 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 5476 0 0 0 2312 2 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4370259 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=619068 CPUtime=23.5 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 10057 0 0 0 2342 8 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=619068 CPUtime=23.48 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 9864 0 0 0 2342 6 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=619068 CPUtime=23.47 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 7249 0 0 0 2342 5 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669973 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=619068 CPUtime=23.77 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 7029 0 0 0 2375 2 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668592 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=619068 CPUtime=23.49 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 10264 0 0 0 2345 4 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=619068 CPUtime=23.51 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 8736 0 0 0 2346 5 0 0 25 0 8 0 661038673 633925632 73686 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 189.4
Current children cumulated vsize (KiB) 619068

[startup+51.1007 s]
/proc/loadavg: 7.85 6.81 7.02 9/409 1832
/proc/meminfo: memFree=22735276/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=626420 CPUtime=393.85 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 102605 0 0 0 39332 53 0 0 25 0 8 0 661038516 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669294 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 156605 83112 466 189 0 150954 0
[pid=1823/tid=1825] ppid=1821 vsize=626420 CPUtime=48.4 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 5788 0 0 0 4837 3 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669413 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=626420 CPUtime=49.1 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 13012 0 0 0 4898 12 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669523 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=626420 CPUtime=49.07 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 11350 0 0 0 4900 7 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=626420 CPUtime=49.05 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 8693 0 0 0 4899 6 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=626420 CPUtime=49.37 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 7120 0 0 0 4935 2 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=626420 CPUtime=49.09 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 11541 0 0 0 4903 6 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=626420 CPUtime=49.1 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 10322 0 0 0 4903 7 0 0 25 0 8 0 661038673 641454080 83112 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 393.85
Current children cumulated vsize (KiB) 626420

[startup+102.316 s]
/proc/loadavg: 8.01 7.02 7.08 9/409 1834
/proc/meminfo: memFree=22667512/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=635540 CPUtime=802.85 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 119155 0 0 0 80209 76 0 0 25 0 8 0 661038516 650792960 99662 33554432000 4194304 4964800 140733767918352 18446744073709551615 4660495 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 158885 99662 466 189 0 153234 0
[pid=1823/tid=1825] ppid=1821 vsize=635540 CPUtime=99.13 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 6261 0 0 0 9910 3 0 0 25 0 8 0 661038673 650792960 99662 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=635540 CPUtime=100.31 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 16975 0 0 0 10013 18 0 0 25 0 8 0 661038673 650792960 99662 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=635540 CPUtime=100.3 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 12958 0 0 0 10020 10 0 0 25 0 8 0 661038673 650792960 99662 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0

################
# More data... #
################

/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38438 0 0 0 465980 48 0 0 25 0 8 0 661038673 2941825024 631828 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2872876 CPUtime=4622.71 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264369 0 0 0 461904 367 0 0 25 0 8 0 661038673 2941825024 631828 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37219.1
Current children cumulated vsize (KiB) 2872876

[startup+4722.32 s]
/proc/loadavg: 8.06 8.04 8.00 9/409 1969
/proc/meminfo: memFree=20534188/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2872876 CPUtime=37698.4 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 652208 0 0 0 3768983 857 0 0 25 0 8 0 661038516 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670645 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 718219 631849 466 189 0 712568 0
[pid=1823/tid=1825] ppid=1821 vsize=2872876 CPUtime=4718.05 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 17192 0 0 0 471784 21 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669557 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=2872876 CPUtime=4720.28 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 85357 0 0 0 471902 126 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=2872876 CPUtime=4720.35 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 40843 0 0 0 471992 43 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=2872876 CPUtime=4695.68 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 79681 0 0 0 469449 119 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=2872876 CPUtime=4720.67 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 72269 0 0 0 471972 95 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=2872876 CPUtime=4720.29 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38440 0 0 0 471981 48 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4644912 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2872876 CPUtime=4681.91 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264369 0 0 0 467824 367 0 0 25 0 8 0 661038673 2941825024 631849 33554432000 4194304 4964800 140733767918352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37698.4
Current children cumulated vsize (KiB) 2872876

[startup+4782.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/409 1971
/proc/meminfo: memFree=20534048/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2872876 CPUtime=38177.4 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 652213 0 0 0 3816886 858 0 0 25 0 8 0 661038516 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4660444 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 718219 631854 466 189 0 712568 0
[pid=1823/tid=1825] ppid=1821 vsize=2872876 CPUtime=4778.03 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 17192 0 0 0 477782 21 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4632355 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=2872876 CPUtime=4780.23 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 85357 0 0 0 477897 126 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669496 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=2872876 CPUtime=4780.34 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 40846 0 0 0 477991 43 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=2872876 CPUtime=4755.67 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 79683 0 0 0 475448 119 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=2872876 CPUtime=4780.66 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 72269 0 0 0 477971 95 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=2872876 CPUtime=4780.27 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38440 0 0 0 477979 48 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2872876 CPUtime=4741.08 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264369 0 0 0 473741 367 0 0 25 0 8 0 661038673 2941825024 631854 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38177.4
Current children cumulated vsize (KiB) 2872876

[startup+4842.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/409 1972
/proc/meminfo: memFree=20533900/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2872876 CPUtime=38656.6 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 652216 0 0 0 3864802 858 0 0 25 0 8 0 661038516 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669779 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 718219 631857 466 189 0 712568 0
[pid=1823/tid=1825] ppid=1821 vsize=2872876 CPUtime=4838.02 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 17192 0 0 0 483781 21 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4362287 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=2872876 CPUtime=4840.23 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 85358 0 0 0 483897 126 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4668728 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=2872876 CPUtime=4840.34 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 40846 0 0 0 483991 43 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4320848 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=2872876 CPUtime=4815.67 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 79685 0 0 0 481448 119 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=2872876 CPUtime=4840.66 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 72269 0 0 0 483971 95 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=2872876 CPUtime=4840.27 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38440 0 0 0 483979 48 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2872876 CPUtime=4800.26 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264369 0 0 0 479659 367 0 0 25 0 8 0 661038673 2941825024 631857 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669386 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38656.6
Current children cumulated vsize (KiB) 2872876

[startup+4902.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/409 1974
/proc/meminfo: memFree=20533756/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2872876 CPUtime=39135.8 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 652219 0 0 0 3912718 858 0 0 25 0 8 0 661038516 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4338788 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 718219 631860 466 189 0 712568 0
[pid=1823/tid=1825] ppid=1821 vsize=2872876 CPUtime=4898.01 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 17192 0 0 0 489780 21 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=2872876 CPUtime=4900.23 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 85358 0 0 0 489897 126 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669523 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=2872876 CPUtime=4900.34 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 40846 0 0 0 489991 43 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=2872876 CPUtime=4875.5 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 79685 0 0 0 487431 119 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=2872876 CPUtime=4900.66 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 72271 0 0 0 489971 95 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4320919 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=2872876 CPUtime=4900.27 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38441 0 0 0 489979 48 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4320864 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2872876 CPUtime=4859.61 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264369 0 0 0 485594 367 0 0 25 0 8 0 661038673 2941825024 631860 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39135.8
Current children cumulated vsize (KiB) 2872876

[startup+4962.3 s]
/proc/loadavg: 8.06 8.06 8.00 9/409 1975
/proc/meminfo: memFree=20532988/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2872876 CPUtime=39614.9 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 652549 0 0 0 3960628 859 0 0 25 0 8 0 661038516 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670437 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 718219 631893 466 189 0 712568 0
[pid=1823/tid=1825] ppid=1821 vsize=2872876 CPUtime=4957.93 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 17521 0 0 0 495772 21 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=2872876 CPUtime=4960.23 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 85358 0 0 0 495897 126 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=2872876 CPUtime=4960.34 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 40847 0 0 0 495991 43 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=2872876 CPUtime=4934.67 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 79685 0 0 0 493348 119 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=2872876 CPUtime=4960.67 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 72271 0 0 0 495972 95 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=2872876 CPUtime=4960.27 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38441 0 0 0 495979 48 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2872876 CPUtime=4919.61 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264369 0 0 0 491594 367 0 0 25 0 8 0 661038673 2941825024 631893 33554432000 4194304 4964800 140733767918352 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39614.9
Current children cumulated vsize (KiB) 2872876



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.6 s]
/proc/loadavg: 8.07 8.06 8.00 9/409 1977
/proc/meminfo: memFree=20517344/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2889604 CPUtime=40000.6 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 656695 0 0 0 3999201 862 0 0 25 0 8 0 661038516 2958954496 636039 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1823/statm: 722401 636039 466 189 0 716750 0
[pid=1823/tid=1825] ppid=1821 vsize=2889604 CPUtime=5006.23 cores=0-7
/proc/1823/task/1825/stat : 1825 (claspmt) R 1821 1823 1407 0 -1 4202560 17521 0 0 0 500602 21 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1823/tid=1826] ppid=1821 vsize=2889604 CPUtime=5008.55 cores=0-7
/proc/1823/task/1826/stat : 1826 (claspmt) R 1821 1823 1407 0 -1 4202560 85900 0 0 0 500728 127 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1823/tid=1827] ppid=1821 vsize=2889604 CPUtime=5008.65 cores=0-7
/proc/1823/task/1827/stat : 1827 (claspmt) R 1821 1823 1407 0 -1 4202560 40851 0 0 0 500822 43 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1823/tid=1828] ppid=1821 vsize=2889604 CPUtime=4982.32 cores=0-7
/proc/1823/task/1828/stat : 1828 (claspmt) R 1821 1823 1407 0 -1 4202560 82869 0 0 0 498110 122 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1823/tid=1829] ppid=1821 vsize=2889604 CPUtime=5008.98 cores=0-7
/proc/1823/task/1829/stat : 1829 (claspmt) R 1821 1823 1407 0 -1 4202560 72402 0 0 0 500803 95 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1823/tid=1830] ppid=1821 vsize=2889604 CPUtime=5008.59 cores=0-7
/proc/1823/task/1830/stat : 1830 (claspmt) R 1821 1823 1407 0 -1 4202560 38530 0 0 0 500811 48 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1823/tid=1831] ppid=1821 vsize=2889604 CPUtime=4967.92 cores=0-7
/proc/1823/task/1831/stat : 1831 (claspmt) R 1821 1823 1407 0 -1 4202560 264566 0 0 0 496425 367 0 0 25 0 8 0 661038673 2958954496 636040 33554432000 4194304 4964800 140733767918352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 2889604

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011 s]
/proc/loadavg: 8.07 8.06 8.00 9/409 1977
/proc/meminfo: memFree=20517344/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2849316 CPUtime=40001.1 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 656707 0 0 0 3999252 863 0 0 25 0 1 0 661038516 2917699584 625979 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476343759 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1823/statm: 712329 625979 476 189 0 706678 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 2849316

[startup+5011.4 s]
/proc/loadavg: 8.07 8.06 8.00 9/409 1977
/proc/meminfo: memFree=20517344/32950928 swapFree=67056568/67111528
[pid=1823] ppid=1821 vsize=2282048 CPUtime=40001.5 cores=0-7
/proc/1823/stat : 1823 (claspmt) R 1821 1823 1407 0 -1 4202496 656707 0 0 0 3999290 864 0 0 25 0 1 0 661038516 2336817152 521373 33554432000 4194304 4964800 140733767918352 18446744073709551615 239476738407 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1823/statm: 570512 521373 476 189 0 564861 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 2282048

Child status: 0
Real time (s): 5011.5
CPU time (s): 40001.7
CPU user time (s): 39992.9
CPU system time (s): 8.74567
CPU usage (%): 798.197
Max. virtual memory (cumulated for all children) (KiB): 2889604

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.9
system time used= 8.74567
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 656708
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 33856
involuntary context switches= 189538

runsolver used 19.495 second user time and 48.8516 second system time

The end

Launcher Data 

Begin job on node148 at 2011-05-15 23:20:15
IDJOB=3380899
IDBENCH=82481
IDSOLVER=1865
FILE ID=node148/3380899-1305494414
RUNJOBID= node148-1305488799-1421
PBS_JOBID= 13325403
Free space on /tmp= 72848 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k50-pdtvisns3p00-tseitin.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380899-1305494414/watcher-3380899-1305494414 -o /tmp/evaluation-result-3380899-1305494414/solver-3380899-1305494414 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380899-1305494414.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a8ad0173824ed9404a8c153da07c5f9c
RANDOM SEED=1316984328

node148.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23060976 kB
Buffers:        346928 kB
Cached:        4999344 kB
SwapCached:       8488 kB
Active:        6181944 kB
Inactive:      3341840 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23060976 kB
SwapTotal:    67111528 kB
SwapFree:     67056568 kB
Dirty:           10512 kB
Writeback:           0 kB
AnonPages:     4176204 kB
Mapped:          32476 kB
Slab:           259348 kB
PageTables:      37176 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 105087472 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72848 MiB
End job on node148 at 2011-05-16 00:43:47