Trace number 3381382

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001 5009.62

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q505.75-n35-p30-s59841049.cnf
MD5SUMc86ce640ddb0138bf41ccc37bc6fcf28
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.156975
Satisfiable
(Un)Satisfiability was proved
Number of variables1295
Number of clauses148400
Sum of the clauses size298025
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2148365
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...514483/instance-3381382-1305514483.cnf
0.59/0.63	c Solving...
40000.30/5009.01	
40000.30/5009.01	*** INTERRUPTED! ***
40000.30/5009.01	*** Shutting down threads...
40000.93/5009.62	s UNKNOWN
40000.93/5009.62	
40000.93/5009.62	c INTERRUPTED   : 1
40000.93/5009.62	c Models        : 0+    
40000.93/5009.62	c Time          : 5009.021s (Solving: 5008.39s 1st Model: 0.00s Unsat: 0.00s)
40000.93/5009.62	c CPU Time      : 40000.350s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381382-1305514483/watcher-3381382-1305514483 -o /tmp/evaluation-result-3381382-1305514483/solver-3381382-1305514483 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381382-1305514483.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.55 7.82 7.50 1/371 19603
/proc/meminfo: memFree=19750268/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=23016 CPUtime=0.01 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 1211 0 0 0 1 0 0 0 25 0 1 0 643863882 23568384 961 33554432000 4194304 4964800 140734087512720 18446744073709551615 264726065861 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/19603/statm: 5754 962 386 189 0 628 0

[startup+0.0528761 s]
/proc/loadavg: 7.55 7.82 7.50 1/371 19603
/proc/meminfo: memFree=19750268/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=31092 CPUtime=0.04 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 3656 0 0 0 4 0 0 0 25 0 1 0 643863882 31838208 2821 33554432000 4194304 4964800 140734087512720 18446744073709551615 4607472 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/19603/statm: 7773 2821 399 189 0 2647 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 31092

[startup+0.100871 s]
/proc/loadavg: 7.55 7.82 7.50 1/371 19603
/proc/meminfo: memFree=19750268/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=31092 CPUtime=0.09 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 3656 0 0 0 9 0 0 0 25 0 1 0 643863882 31838208 2821 33554432000 4194304 4964800 140734087512720 18446744073709551615 4607575 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/19603/statm: 7773 2821 399 189 0 2647 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31092

[startup+0.300833 s]
/proc/loadavg: 7.55 7.82 7.50 1/371 19603
/proc/meminfo: memFree=19750268/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=31092 CPUtime=0.29 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 3656 0 0 0 29 0 0 0 25 0 1 0 643863882 31838208 2821 33554432000 4194304 4964800 140734087512720 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/19603/statm: 7773 2821 399 189 0 2647 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 31092

[startup+0.700753 s]
/proc/loadavg: 7.55 7.82 7.50 1/371 19603
/proc/meminfo: memFree=19750268/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=390076 CPUtime=0.7 cores=0-7
/proc/19603/stat : 19603 (claspmt) S 19601 19603 19465 0 -1 4202496 4762 0 0 0 70 0 0 0 18 0 8 0 643863882 399437824 3625 33554432000 4194304 4964800 140734087512720 18446744073709551615 264738220953 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/19603/statm: 97519 3626 435 189 0 91868 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 390076

[startup+1.50057 s]
/proc/loadavg: 7.55 7.82 7.50 9/379 19611
/proc/meminfo: memFree=19717724/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=457728 CPUtime=6.96 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 15779 0 0 0 690 6 0 0 18 0 8 0 643863882 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4364744 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 114432 14642 464 189 0 108781 0
[pid=19603/tid=19605] ppid=19601 vsize=457728 CPUtime=0.77 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 1151 0 0 0 77 0 0 0 18 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=457728 CPUtime=0.77 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 1365 0 0 0 77 0 0 0 19 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670688 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=457728 CPUtime=0.76 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 1608 0 0 0 75 1 0 0 18 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=457728 CPUtime=0.77 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 1702 0 0 0 77 0 0 0 20 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 264726066209 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=457728 CPUtime=0.86 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 615 0 0 0 86 0 0 0 19 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320911 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=457728 CPUtime=0.78 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 1911 0 0 0 78 0 0 0 18 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4332387 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=457728 CPUtime=0.77 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 1879 0 0 0 77 0 0 0 18 0 8 0 643863946 468713472 14642 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 6.96
Current children cumulated vsize (KiB) 457728

[startup+3.10126 s]
/proc/loadavg: 7.55 7.82 7.50 10/379 19611
/proc/meminfo: memFree=19667380/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=467456 CPUtime=19.72 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 32255 0 0 0 1957 15 0 0 19 0 8 0 643863882 478674944 31118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4627335 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 116864 31118 464 189 0 111213 0
[pid=19603/tid=19605] ppid=19601 vsize=467456 CPUtime=2.37 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 3151 0 0 0 236 1 0 0 19 0 8 0 643863946 478674944 31118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4689698 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=467456 CPUtime=2.37 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 2468 0 0 0 236 1 0 0 22 0 8 0 643863946 478674944 31119 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321029 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=467456 CPUtime=2.34 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 4175 0 0 0 233 1 0 0 20 0 8 0 643863946 478674944 31120 33554432000 4194304 4964800 140734087512720 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=467456 CPUtime=2.37 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 3862 0 0 0 235 2 0 0 25 0 8 0 643863946 478674944 31120 33554432000 4194304 4964800 140734087512720 18446744073709551615 264726066209 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=467456 CPUtime=2.46 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 2097 0 0 0 245 1 0 0 18 0 8 0 643863946 478674944 31122 33554432000 4194304 4964800 140734087512720 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=467456 CPUtime=2.38 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 3686 0 0 0 237 1 0 0 20 0 8 0 643863946 478674944 31122 33554432000 4194304 4964800 140734087512720 18446744073709551615 4342186 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=467456 CPUtime=2.37 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 5120 0 0 0 234 3 0 0 20 0 8 0 643863946 478674944 31122 33554432000 4194304 4964800 140734087512720 18446744073709551615 4660144 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 19.72
Current children cumulated vsize (KiB) 467456

[startup+6.30066 s]
/proc/loadavg: 7.59 7.82 7.50 9/379 19611
/proc/meminfo: memFree=19577108/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=474876 CPUtime=45.27 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 46417 0 0 0 4504 23 0 0 19 0 8 0 643863882 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4669026 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 118719 45280 466 189 0 113068 0
[pid=19603/tid=19605] ppid=19601 vsize=474876 CPUtime=5.57 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 4876 0 0 0 555 2 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=474876 CPUtime=5.57 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 3189 0 0 0 556 1 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=474876 CPUtime=5.5 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 5891 0 0 0 548 2 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=474876 CPUtime=5.57 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 4989 0 0 0 555 2 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=474876 CPUtime=5.65 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 3960 0 0 0 563 2 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321093 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=474876 CPUtime=5.58 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 5136 0 0 0 556 2 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=474876 CPUtime=5.57 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 8876 0 0 0 552 5 0 0 25 0 8 0 643863946 486273024 45280 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 45.27
Current children cumulated vsize (KiB) 474876

[startup+12.7005 s]
/proc/loadavg: 7.62 7.82 7.50 9/379 19611
/proc/meminfo: memFree=19540964/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=478844 CPUtime=96.38 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 58065 0 0 0 9608 30 0 0 25 0 8 0 643863882 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4653316 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 119711 56928 466 189 0 114060 0
[pid=19603/tid=19605] ppid=19601 vsize=478844 CPUtime=11.97 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 6582 0 0 0 1194 3 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=478844 CPUtime=11.97 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 4151 0 0 0 1195 2 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670330 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=478844 CPUtime=11.82 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 6596 0 0 0 1179 3 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=478844 CPUtime=11.97 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 5646 0 0 0 1194 3 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=478844 CPUtime=12.05 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 5503 0 0 0 1202 3 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670129 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=478844 CPUtime=11.97 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 6323 0 0 0 1195 2 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=478844 CPUtime=11.96 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 12324 0 0 0 1190 6 0 0 25 0 8 0 643863946 490336256 56928 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 96.38
Current children cumulated vsize (KiB) 478844

[startup+25.501 s]
/proc/loadavg: 7.70 7.83 7.51 9/379 19612
/proc/meminfo: memFree=19452728/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=490020 CPUtime=198.62 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 78620 0 0 0 19818 44 0 0 25 0 8 0 643863882 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 122505 77482 466 189 0 116854 0
[pid=19603/tid=19605] ppid=19601 vsize=490020 CPUtime=24.76 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 9024 0 0 0 2472 4 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4627290 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=490020 CPUtime=24.77 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 5254 0 0 0 2475 2 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=490020 CPUtime=24.47 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 9504 0 0 0 2442 5 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=490020 CPUtime=24.77 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 6711 0 0 0 2474 3 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=490020 CPUtime=24.85 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 7547 0 0 0 2481 4 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=490020 CPUtime=24.77 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 7158 0 0 0 2474 3 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=490020 CPUtime=24.77 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 19106 0 0 0 2466 11 0 0 25 0 8 0 643863946 501780480 77482 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 198.62
Current children cumulated vsize (KiB) 490020

[startup+51.1012 s]
/proc/loadavg: 7.88 7.86 7.53 9/379 19612
/proc/meminfo: memFree=19373200/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=496512 CPUtime=403.08 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 98767 0 0 0 40248 60 0 0 25 0 8 0 643863882 508428288 97605 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670321 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 124128 97605 466 189 0 118477 0
[pid=19603/tid=19605] ppid=19601 vsize=496512 CPUtime=50.35 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 11360 0 0 0 5029 6 0 0 25 0 8 0 643863946 508428288 97605 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321076 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=496512 CPUtime=50.37 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 6720 0 0 0 5034 3 0 0 25 0 8 0 643863946 508428288 97606 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=496512 CPUtime=49.75 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 11323 0 0 0 4969 6 0 0 25 0 8 0 643863946 508428288 97606 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=496512 CPUtime=50.36 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 8829 0 0 0 5032 4 0 0 25 0 8 0 643863946 508428288 97606 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=496512 CPUtime=50.44 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 8759 0 0 0 5039 5 0 0 25 0 8 0 643863946 508428288 97606 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=496512 CPUtime=50.36 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 8532 0 0 0 5032 4 0 0 25 0 8 0 643863946 508428288 97606 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=496512 CPUtime=50.37 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 23175 0 0 0 5023 14 0 0 25 0 8 0 643863946 508428288 97606 33554432000 4194304 4964800 140734087512720 18446744073709551615 264726066209 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 403.08

################
# More data... #
################

/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 64951 0 0 0 463678 85 0 0 25 0 8 0 643863946 3222401024 704177 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670313 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3146876 CPUtime=4653.23 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164671 0 0 0 465103 220 0 0 25 0 8 0 643863946 3222401024 704177 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37231.4
Current children cumulated vsize (KiB) 3146876

[startup+4722.31 s]
/proc/loadavg: 8.06 8.03 8.00 9/379 19758
/proc/meminfo: memFree=16930976/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3146876 CPUtime=37710.6 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 718284 0 0 0 3770284 780 0 0 25 0 8 0 643863882 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4362195 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 786719 705068 466 189 0 781068 0
[pid=19603/tid=19605] ppid=19601 vsize=3146876 CPUtime=4703.55 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 70782 0 0 0 470274 81 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321065 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=3146876 CPUtime=4720.56 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 55558 0 0 0 472034 22 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4644876 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=3146876 CPUtime=4712.01 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 87104 0 0 0 471092 109 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=3146876 CPUtime=4721.49 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 56086 0 0 0 472101 48 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=3146876 CPUtime=4720.92 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 50877 0 0 0 472056 36 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=3146876 CPUtime=4696.87 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 64951 0 0 0 469602 85 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320921 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3146876 CPUtime=4713.24 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164671 0 0 0 471104 220 0 0 25 0 8 0 643863946 3222401024 705068 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37710.6
Current children cumulated vsize (KiB) 3146876

[startup+4782.3 s]
/proc/loadavg: 8.08 8.04 8.00 9/379 19760
/proc/meminfo: memFree=16930180/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3148700 CPUtime=38189.7 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 719280 0 0 0 3818186 783 0 0 25 0 8 0 643863882 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 787175 705152 466 189 0 781524 0
[pid=19603/tid=19605] ppid=19601 vsize=3148700 CPUtime=4763.54 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 70787 0 0 0 476273 81 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4686992 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=3148700 CPUtime=4780.47 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 55562 0 0 0 478024 23 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=3148700 CPUtime=4772 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 87104 0 0 0 477091 109 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=3148700 CPUtime=4781.48 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 57056 0 0 0 478100 48 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4669008 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=3148700 CPUtime=4780.92 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 50877 0 0 0 478055 37 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=3148700 CPUtime=4756.11 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 64968 0 0 0 475525 86 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4343816 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3148700 CPUtime=4773.23 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164671 0 0 0 477102 221 0 0 25 0 8 0 643863946 3224268800 705152 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38189.7
Current children cumulated vsize (KiB) 3148700

[startup+4842.3 s]
/proc/loadavg: 8.03 8.03 8.00 9/379 19761
/proc/meminfo: memFree=16930020/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3148700 CPUtime=38668.7 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 719294 0 0 0 3866088 786 0 0 25 0 8 0 643863882 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4653507 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 787175 705166 466 189 0 781524 0
[pid=19603/tid=19605] ppid=19601 vsize=3148700 CPUtime=4823.53 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 70792 0 0 0 482272 81 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=3148700 CPUtime=4840.46 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 55562 0 0 0 484023 23 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=3148700 CPUtime=4831.99 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 87112 0 0 0 483089 110 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=3148700 CPUtime=4841.39 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 57057 0 0 0 484090 49 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=3148700 CPUtime=4840.91 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 50877 0 0 0 484054 37 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=3148700 CPUtime=4815.76 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 64968 0 0 0 481490 86 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3148700 CPUtime=4832.76 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164671 0 0 0 483055 221 0 0 25 0 8 0 643863946 3224268800 705166 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38668.7
Current children cumulated vsize (KiB) 3148700

[startup+4902.3 s]
/proc/loadavg: 8.01 8.02 8.00 9/379 19763
/proc/meminfo: memFree=16929848/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3148700 CPUtime=39148 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 719361 0 0 0 3914011 786 0 0 25 0 8 0 643863882 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 787175 705233 466 189 0 781524 0
[pid=19603/tid=19605] ppid=19601 vsize=3148700 CPUtime=4883.54 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 70793 0 0 0 488273 81 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=3148700 CPUtime=4900.44 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 55562 0 0 0 490021 23 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=3148700 CPUtime=4891.98 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 87148 0 0 0 489088 110 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=3148700 CPUtime=4901.39 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 57057 0 0 0 490090 49 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=3148700 CPUtime=4900.92 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 50878 0 0 0 490055 37 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=3148700 CPUtime=4875.76 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 64988 0 0 0 487490 86 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3148700 CPUtime=4892 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164680 0 0 0 488979 221 0 0 25 0 8 0 643863946 3224268800 705233 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39148
Current children cumulated vsize (KiB) 3148700

[startup+4962.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/379 19764
/proc/meminfo: memFree=16926952/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3148700 CPUtime=39627.2 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 720194 0 0 0 3961935 786 0 0 25 0 8 0 643863882 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 787175 706066 466 189 0 781524 0
[pid=19603/tid=19605] ppid=19601 vsize=3148700 CPUtime=4943.54 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 70875 0 0 0 494273 81 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=3148700 CPUtime=4960.43 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 55562 0 0 0 496020 23 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=3148700 CPUtime=4951.98 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 87148 0 0 0 495088 110 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=3148700 CPUtime=4961.39 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 57069 0 0 0 496090 49 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=3148700 CPUtime=4960.92 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 50878 0 0 0 496055 37 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=3148700 CPUtime=4935.76 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 65719 0 0 0 493490 86 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3148700 CPUtime=4951.24 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164680 0 0 0 494903 221 0 0 25 0 8 0 643863946 3224268800 706066 33554432000 4194304 4964800 140734087512720 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39627.2
Current children cumulated vsize (KiB) 3148700



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009 s]
/proc/loadavg: 8.05 8.03 8.00 9/379 19766
/proc/meminfo: memFree=16926912/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3148700 CPUtime=40000.3 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 720246 0 0 0 3999243 787 0 0 25 0 8 0 643863882 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4660123 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19603/statm: 787175 706118 466 189 0 781524 0
[pid=19603/tid=19605] ppid=19601 vsize=3148700 CPUtime=4990.25 cores=0-7
/proc/19603/task/19605/stat : 19605 (claspmt) R 19601 19603 19465 0 -1 4202560 70886 0 0 0 498944 81 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19603/tid=19606] ppid=19601 vsize=3148700 CPUtime=5007.14 cores=0-7
/proc/19603/task/19606/stat : 19606 (claspmt) R 19601 19603 19465 0 -1 4202560 55562 0 0 0 500691 23 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19603/tid=19607] ppid=19601 vsize=3148700 CPUtime=4998.7 cores=0-7
/proc/19603/task/19607/stat : 19607 (claspmt) R 19601 19603 19465 0 -1 4202560 87148 0 0 0 499760 110 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19603/tid=19608] ppid=19601 vsize=3148700 CPUtime=5008.11 cores=0-7
/proc/19603/task/19608/stat : 19608 (claspmt) R 19601 19603 19465 0 -1 4202560 57069 0 0 0 500762 49 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19603/tid=19609] ppid=19601 vsize=3148700 CPUtime=5007.63 cores=0-7
/proc/19603/task/19609/stat : 19609 (claspmt) R 19601 19603 19465 0 -1 4202560 50881 0 0 0 500726 37 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19603/tid=19610] ppid=19601 vsize=3148700 CPUtime=4982.47 cores=0-7
/proc/19603/task/19610/stat : 19610 (claspmt) R 19601 19603 19465 0 -1 4202560 65735 0 0 0 498160 87 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19603/tid=19611] ppid=19601 vsize=3148700 CPUtime=4997.34 cores=0-7
/proc/19603/task/19611/stat : 19611 (claspmt) R 19601 19603 19465 0 -1 4202560 164702 0 0 0 499513 221 0 0 25 0 8 0 643863946 3224268800 706118 33554432000 4194304 4964800 140734087512720 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 3148700

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.41 s]
/proc/loadavg: 8.05 8.03 8.00 2/373 19767
/proc/meminfo: memFree=16936632/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=3132120 CPUtime=40000.7 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 720256 0 0 0 3999287 787 0 0 25 0 1 0 643863882 3207290880 699442 33554432000 4194304 4964800 140734087512720 18446744073709551615 264726060313 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/19603/statm: 783030 699442 476 189 0 777379 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 3132120

[startup+5009.6 s]
/proc/loadavg: 8.05 8.03 8.00 2/373 19767
/proc/meminfo: memFree=16936632/32951124 swapFree=63083172/67111528
[pid=19603] ppid=19601 vsize=717676 CPUtime=40000.9 cores=0-7
/proc/19603/stat : 19603 (claspmt) R 19601 19603 19465 0 -1 4202496 720256 0 0 0 3999295 798 0 0 25 0 1 0 643863882 734900224 107981 33554432000 4194304 4964800 140734087512720 18446744073709551615 264726448487 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/19603/statm: 179419 107469 476 189 0 173768 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 717676

Child status: 0
Real time (s): 5009.62
CPU time (s): 40001
CPU user time (s): 39993
CPU system time (s): 8.00878
CPU usage (%): 798.482
Max. virtual memory (cumulated for all children) (KiB): 3148700

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993
system time used= 8.00878
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 720257
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 117830
involuntary context switches= 138736

runsolver used 17.4633 second user time and 45.856 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-16 04:54:43
IDJOB=3381382
IDBENCH=88218
IDSOLVER=1865
FILE ID=node141/3381382-1305514483
RUNJOBID= node141-1305513424-19479
PBS_JOBID= 13325539
Free space on /tmp= 73396 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q505.75-n35-p30-s59841049.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381382-1305514483/watcher-3381382-1305514483 -o /tmp/evaluation-result-3381382-1305514483/solver-3381382-1305514483 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381382-1305514483.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c86ce640ddb0138bf41ccc37bc6fcf28
RANDOM SEED=1078167787

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19750672 kB
Buffers:        283216 kB
Cached:        5389744 kB
SwapCached:      15728 kB
Active:        7647212 kB
Inactive:      5284512 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19750672 kB
SwapTotal:    67111528 kB
SwapFree:     63083172 kB
Dirty:            2740 kB
Writeback:          24 kB
AnonPages:     7250936 kB
Mapped:          15504 kB
Slab:           146392 kB
PageTables:      49392 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 92063332 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73396 MiB
End job on node141 at 2011-05-16 06:18:14