Trace number 3380875

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.1 5009.62

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k45-pdtpmspalu-tseitin.cnf
MD5SUM8dae13531986e2a9bdb6559418f57130
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark1004.79
Satisfiable
(Un)Satisfiability was proved
Number of variables89190
Number of clauses265718
Sum of the clauses size620006
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2177144
Number of clauses of size 388572
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...494103/instance-3380875-1305494103.cnf
0.78/0.86	c Solving...
40000.24/5008.81	
40000.24/5008.81	*** INTERRUPTED! ***
40000.24/5008.81	*** Shutting down threads...
40001.08/5009.61	s UNKNOWN
40001.08/5009.61	
40001.08/5009.61	c INTERRUPTED   : 1
40001.08/5009.61	c Models        : 0+    
40001.08/5009.61	c Time          : 5008.821s (Solving: 5007.96s 1st Model: 0.00s Unsat: 0.00s)
40001.08/5009.61	c CPU Time      : 40000.310s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380875-1305494103/watcher-3380875-1305494103 -o /tmp/evaluation-result-3380875-1305494103/solver-3380875-1305494103 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380875-1305494103.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 6.08 4.66 2/345 29871
/proc/meminfo: memFree=13512932/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=32180 CPUtime=0 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 3036 0 0 0 0 0 0 0 18 0 1 0 661009104 32952320 2967 33554432000 4194304 4964800 140733318544944 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/29871/statm: 8045 2974 376 189 0 2919 0

[startup+0.0658069 s]
/proc/loadavg: 8.10 6.08 4.66 2/345 29871
/proc/meminfo: memFree=13512932/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=46448 CPUtime=0.06 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 8196 0 0 0 5 1 0 0 18 0 1 0 661009104 47562752 6686 33554432000 4194304 4964800 140733318544944 18446744073709551615 235594527429 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/29871/statm: 11612 6687 386 189 0 6486 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 46448

[startup+0.100801 s]
/proc/loadavg: 8.10 6.08 4.66 2/345 29871
/proc/meminfo: memFree=13512932/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=55356 CPUtime=0.09 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 11688 0 0 0 8 1 0 0 18 0 1 0 661009104 56684544 8901 33554432000 4194304 4964800 140733318544944 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/29871/statm: 13839 8901 399 189 0 8713 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 55356

[startup+0.300766 s]
/proc/loadavg: 8.10 6.08 4.66 2/345 29871
/proc/meminfo: memFree=13512932/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=67980 CPUtime=0.29 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 14733 0 0 0 27 2 0 0 18 0 1 0 661009104 69611520 11946 33554432000 4194304 4964800 140733318544944 18446744073709551615 235594527429 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/29871/statm: 16995 11946 404 189 0 11869 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 67980

[startup+0.700695 s]
/proc/loadavg: 8.10 6.08 4.66 2/345 29871
/proc/meminfo: memFree=13512932/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=77484 CPUtime=0.68 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 17112 0 0 0 66 2 0 0 18 0 1 0 661009104 79343616 14325 33554432000 4194304 4964800 140733318544944 18446744073709551615 4601734 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/29871/statm: 19371 14325 404 189 0 14245 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 77484

[startup+1.50056 s]
/proc/loadavg: 8.10 6.08 4.66 9/353 29880
/proc/meminfo: memFree=13379084/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=531576 CPUtime=5.17 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 44015 0 0 0 504 13 0 0 19 0 8 0 661009104 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29871/statm: 132894 34922 464 189 0 127243 0
[pid=29871/tid=29874] ppid=29869 vsize=531576 CPUtime=0.53 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 3113 0 0 0 53 0 0 0 17 0 8 0 661009190 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=531576 CPUtime=0.51 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 3796 0 0 0 50 1 0 0 21 0 8 0 661009190 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=531576 CPUtime=0.53 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 4282 0 0 0 52 1 0 0 18 0 8 0 661009190 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4660384 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=531576 CPUtime=0.53 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 3572 0 0 0 51 2 0 0 17 0 8 0 661009190 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=531576 CPUtime=0.59 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 3614 0 0 0 58 1 0 0 18 0 8 0 661009190 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=531576 CPUtime=0.52 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 4163 0 0 0 51 1 0 0 18 0 8 0 661009190 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669276 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=531576 CPUtime=0.56 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 3475 0 0 0 55 1 0 0 18 0 8 0 661009191 544333824 34922 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 5.17
Current children cumulated vsize (KiB) 531576

[startup+3.10128 s]
/proc/loadavg: 8.10 6.08 4.66 9/353 29880
/proc/meminfo: memFree=13369040/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=531780 CPUtime=17.96 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 47644 0 0 0 1781 15 0 0 23 0 8 0 661009104 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29871/statm: 132945 38551 464 189 0 127294 0
[pid=29871/tid=29874] ppid=29869 vsize=531780 CPUtime=2.14 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 3254 0 0 0 213 1 0 0 20 0 8 0 661009190 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=531780 CPUtime=2.12 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 4868 0 0 0 210 2 0 0 25 0 8 0 661009190 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=531780 CPUtime=2.13 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 5163 0 0 0 212 1 0 0 20 0 8 0 661009190 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=531780 CPUtime=2.13 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 3741 0 0 0 211 2 0 0 20 0 8 0 661009190 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4372505 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=531780 CPUtime=2.19 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 3622 0 0 0 218 1 0 0 21 0 8 0 661009190 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=531780 CPUtime=2.13 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 4960 0 0 0 211 2 0 0 20 0 8 0 661009190 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668678 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=531780 CPUtime=2.15 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 3884 0 0 0 214 1 0 0 20 0 8 0 661009191 544542720 38551 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 17.96
Current children cumulated vsize (KiB) 531780

[startup+6.30072 s]
/proc/loadavg: 8.09 6.11 4.68 9/353 29880
/proc/meminfo: memFree=13334632/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=534684 CPUtime=43.52 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 56831 0 0 0 4329 23 0 0 25 0 8 0 661009104 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29871/statm: 133671 47738 464 189 0 128020 0
[pid=29871/tid=29874] ppid=29869 vsize=534684 CPUtime=5.34 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 3522 0 0 0 533 1 0 0 25 0 8 0 661009190 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370266 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=534684 CPUtime=5.32 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 6339 0 0 0 529 3 0 0 25 0 8 0 661009190 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669216 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=534684 CPUtime=5.33 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 6478 0 0 0 531 2 0 0 25 0 8 0 661009190 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670025 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=534684 CPUtime=5.33 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 4171 0 0 0 531 2 0 0 25 0 8 0 661009190 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=534684 CPUtime=5.39 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 6331 0 0 0 536 3 0 0 25 0 8 0 661009190 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4654476 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=534684 CPUtime=5.33 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 6479 0 0 0 530 3 0 0 25 0 8 0 661009190 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670674 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=534684 CPUtime=5.35 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 4958 0 0 0 533 2 0 0 25 0 8 0 661009191 547516416 47738 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 43.52
Current children cumulated vsize (KiB) 534684

[startup+12.7006 s]
/proc/loadavg: 8.08 6.14 4.70 9/353 29880
/proc/meminfo: memFree=13286692/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=542352 CPUtime=94.62 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 66615 0 0 0 9431 31 0 0 25 0 8 0 661009104 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668681 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29871/statm: 135588 57522 464 189 0 129937 0
[pid=29871/tid=29874] ppid=29869 vsize=542352 CPUtime=11.73 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 3713 0 0 0 1172 1 0 0 25 0 8 0 661009190 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669192 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=542352 CPUtime=11.72 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 8290 0 0 0 1167 5 0 0 25 0 8 0 661009190 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=542352 CPUtime=11.73 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 7200 0 0 0 1170 3 0 0 25 0 8 0 661009190 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=542352 CPUtime=11.72 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 4803 0 0 0 1170 2 0 0 25 0 8 0 661009190 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=542352 CPUtime=11.79 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 8364 0 0 0 1175 4 0 0 25 0 8 0 661009190 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669272 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=542352 CPUtime=11.71 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 8384 0 0 0 1167 4 0 0 25 0 8 0 661009190 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=542352 CPUtime=11.75 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 6674 0 0 0 1171 4 0 0 25 0 8 0 661009191 555368448 57522 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 94.62
Current children cumulated vsize (KiB) 542352

[startup+25.5003 s]
/proc/loadavg: 8.06 6.24 4.75 9/353 29880
/proc/meminfo: memFree=13249592/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=546764 CPUtime=196.85 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 77233 0 0 0 19646 39 0 0 25 0 8 0 661009104 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670273 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29871/statm: 136691 68140 466 189 0 131040 0
[pid=29871/tid=29874] ppid=29869 vsize=546764 CPUtime=24.54 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 4466 0 0 0 2452 2 0 0 25 0 8 0 661009190 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668725 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=546764 CPUtime=24.51 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 8856 0 0 0 2446 5 0 0 25 0 8 0 661009190 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=546764 CPUtime=24.53 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 8213 0 0 0 2449 4 0 0 25 0 8 0 661009190 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=546764 CPUtime=24.53 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 7004 0 0 0 2450 3 0 0 25 0 8 0 661009190 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=546764 CPUtime=24.59 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 10669 0 0 0 2454 5 0 0 25 0 8 0 661009190 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=546764 CPUtime=24.51 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 9677 0 0 0 2445 6 0 0 25 0 8 0 661009190 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=546764 CPUtime=24.55 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 8372 0 0 0 2449 6 0 0 25 0 8 0 661009191 559886336 68140 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 196.85
Current children cumulated vsize (KiB) 546764

[startup+51.1008 s]
/proc/loadavg: 8.04 6.38 4.83 9/353 29881
/proc/meminfo: memFree=13168840/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=559456 CPUtime=401.31 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 96417 0 0 0 40078 53 0 0 25 0 8 0 661009104 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29871/statm: 139864 87324 466 189 0 134213 0
[pid=29871/tid=29874] ppid=29869 vsize=559456 CPUtime=50.13 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 4857 0 0 0 5011 2 0 0 25 0 8 0 661009190 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=559456 CPUtime=50.1 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 11109 0 0 0 5003 7 0 0 25 0 8 0 661009190 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=559456 CPUtime=50.12 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 11524 0 0 0 5005 7 0 0 25 0 8 0 661009190 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=559456 CPUtime=50.13 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 9370 0 0 0 5008 5 0 0 25 0 8 0 661009190 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=559456 CPUtime=50.18 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 14558 0 0 0 5011 7 0 0 25 0 8 0 661009190 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670155 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=559456 CPUtime=50.1 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 13294 0 0 0 5003 7 0 0 25 0 8 0 661009190 572882944 87324 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=559456 CPUtime=50.08 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 10165 0 0 0 5000 8 0 0 25 0 8 0 661009191 572882944 87325 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 401.31

################
# More data... #
################

/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87505 0 0 0 466003 81 0 0 25 0 8 0 661009190 3843522560 881656 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669476 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3753440 CPUtime=4651.99 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 319925 0 0 0 464961 238 0 0 25 0 8 0 661009191 3843522560 881656 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37232.5
Current children cumulated vsize (KiB) 3753440

[startup+4722.31 s]
/proc/loadavg: 8.07 8.06 8.00 9/353 30017
/proc/meminfo: memFree=9978444/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3753440 CPUtime=37711.9 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 894801 0 0 0 3770440 745 0 0 25 0 8 0 661009104 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668987 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29871/statm: 938360 881663 466 189 0 932709 0
[pid=29871/tid=29874] ppid=29869 vsize=3753440 CPUtime=4706.41 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 27811 0 0 0 470603 38 0 0 25 0 8 0 661009190 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=3753440 CPUtime=4719.78 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 37693 0 0 0 471937 41 0 0 25 0 8 0 661009190 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=3753440 CPUtime=4697.99 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 49869 0 0 0 469749 50 0 0 25 0 8 0 661009190 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=3753440 CPUtime=4721.07 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 130080 0 0 0 472006 101 0 0 25 0 8 0 661009190 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=3753440 CPUtime=4717.93 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 123924 0 0 0 471699 94 0 0 25 0 8 0 661009190 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=3753440 CPUtime=4720.84 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87505 0 0 0 472003 81 0 0 25 0 8 0 661009190 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3753440 CPUtime=4711.29 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 319931 0 0 0 470891 238 0 0 25 0 8 0 661009191 3843522560 881663 33554432000 4194304 4964800 140733318544944 18446744073709551615 4660444 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37711.8
Current children cumulated vsize (KiB) 3753440

[startup+4782.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/353 30019
/proc/meminfo: memFree=9978760/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3753440 CPUtime=38191 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 894811 0 0 0 3818357 746 0 0 25 0 8 0 661009104 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669509 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29871/statm: 938360 881673 466 189 0 932709 0
[pid=29871/tid=29874] ppid=29869 vsize=3753440 CPUtime=4766.4 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 27811 0 0 0 476602 38 0 0 25 0 8 0 661009190 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4654540 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=3753440 CPUtime=4779.76 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 37700 0 0 0 477934 42 0 0 25 0 8 0 661009190 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=3753440 CPUtime=4757.98 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 49870 0 0 0 475748 50 0 0 25 0 8 0 661009190 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=3753440 CPUtime=4781.06 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 130080 0 0 0 478005 101 0 0 25 0 8 0 661009190 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=3753440 CPUtime=4777.91 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 123924 0 0 0 477697 94 0 0 25 0 8 0 661009190 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=3753440 CPUtime=4780.83 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87505 0 0 0 478001 82 0 0 25 0 8 0 661009190 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3753440 CPUtime=4770.58 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 319933 0 0 0 476820 238 0 0 25 0 8 0 661009191 3843522560 881673 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38191
Current children cumulated vsize (KiB) 3753440

[startup+4842.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/353 30021
/proc/meminfo: memFree=9978472/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3753440 CPUtime=38670.3 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 894849 0 0 0 3866280 747 0 0 25 0 8 0 661009104 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4325293 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29871/statm: 938360 881711 466 189 0 932709 0
[pid=29871/tid=29874] ppid=29869 vsize=3753440 CPUtime=4826.4 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 27817 0 0 0 482602 38 0 0 25 0 8 0 661009190 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=3753440 CPUtime=4839.76 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 37700 0 0 0 483934 42 0 0 25 0 8 0 661009190 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=3753440 CPUtime=4817.97 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 49870 0 0 0 481747 50 0 0 25 0 8 0 661009190 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=3753440 CPUtime=4841.04 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 130080 0 0 0 484003 101 0 0 25 0 8 0 661009190 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 235594564194 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=3753440 CPUtime=4837.89 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 123924 0 0 0 483694 95 0 0 25 0 8 0 661009190 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=3753440 CPUtime=4840.83 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87537 0 0 0 484001 82 0 0 25 0 8 0 661009190 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3753440 CPUtime=4829.86 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 319933 0 0 0 482748 238 0 0 25 0 8 0 661009191 3843522560 881711 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38670.3
Current children cumulated vsize (KiB) 3753440

[startup+4902.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/353 30022
/proc/meminfo: memFree=9978552/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3753440 CPUtime=39149.5 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 894859 0 0 0 3914206 747 0 0 25 0 8 0 661009104 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668568 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29871/statm: 938360 881721 466 189 0 932709 0
[pid=29871/tid=29874] ppid=29869 vsize=3753440 CPUtime=4886.4 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 27817 0 0 0 488602 38 0 0 25 0 8 0 661009190 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669557 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=3753440 CPUtime=4899.75 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 37700 0 0 0 489933 42 0 0 25 0 8 0 661009190 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=3753440 CPUtime=4877.97 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 49877 0 0 0 487746 51 0 0 25 0 8 0 661009190 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668777 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=3753440 CPUtime=4901.04 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 130080 0 0 0 490003 101 0 0 25 0 8 0 661009190 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4370259 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=3753440 CPUtime=4897.88 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 123924 0 0 0 489693 95 0 0 25 0 8 0 661009190 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669413 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=3753440 CPUtime=4900.83 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87539 0 0 0 490001 82 0 0 25 0 8 0 661009190 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3753440 CPUtime=4889.16 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 319934 0 0 0 488677 239 0 0 25 0 8 0 661009191 3843522560 881721 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39149.5
Current children cumulated vsize (KiB) 3753440

[startup+4962.3 s]
/proc/loadavg: 8.06 8.06 8.00 9/353 30024
/proc/meminfo: memFree=9978372/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3753440 CPUtime=39628.8 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 894860 0 0 0 3962129 748 0 0 25 0 8 0 661009104 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29871/statm: 938360 881722 466 189 0 932709 0
[pid=29871/tid=29874] ppid=29869 vsize=3753440 CPUtime=4946.37 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 27817 0 0 0 494599 38 0 0 25 0 8 0 661009190 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=3753440 CPUtime=4959.75 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 37700 0 0 0 495933 42 0 0 25 0 8 0 661009190 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4669538 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=3753440 CPUtime=4937.97 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 49877 0 0 0 493746 51 0 0 25 0 8 0 661009190 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4321272 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=3753440 CPUtime=4961.04 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 130081 0 0 0 496003 101 0 0 25 0 8 0 661009190 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=3753440 CPUtime=4957.88 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 123924 0 0 0 495693 95 0 0 25 0 8 0 661009190 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=3753440 CPUtime=4960.82 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87539 0 0 0 496000 82 0 0 25 0 8 0 661009190 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4654532 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3753440 CPUtime=4948.45 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 319934 0 0 0 494606 239 0 0 25 0 8 0 661009191 3843522560 881722 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39628.8
Current children cumulated vsize (KiB) 3753440



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.8 s]
/proc/loadavg: 8.07 8.06 8.00 9/353 30025
/proc/meminfo: memFree=9960116/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3818976 CPUtime=40000.2 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 899772 0 0 0 3999269 755 0 0 25 0 8 0 661009104 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4334309 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29871/statm: 954744 886634 466 189 0 949093 0
[pid=29871/tid=29874] ppid=29869 vsize=3818976 CPUtime=4992.88 cores=0-7
/proc/29871/task/29874/stat : 29874 (claspmt) R 29869 29871 29772 0 -1 4202560 27817 0 0 0 499250 38 0 0 25 0 8 0 661009190 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29871/tid=29875] ppid=29869 vsize=3818976 CPUtime=5006.24 cores=0-7
/proc/29871/task/29875/stat : 29875 (claspmt) R 29869 29871 29772 0 -1 4202560 37700 0 0 0 500582 42 0 0 25 0 8 0 661009190 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29871/tid=29876] ppid=29869 vsize=3818976 CPUtime=4984.47 cores=0-7
/proc/29871/task/29876/stat : 29876 (claspmt) R 29869 29871 29772 0 -1 4202560 50950 0 0 0 498395 52 0 0 25 0 8 0 661009190 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29871/tid=29877] ppid=29869 vsize=3818976 CPUtime=5007.55 cores=0-7
/proc/29871/task/29877/stat : 29877 (claspmt) R 29869 29871 29772 0 -1 4202560 130392 0 0 0 500653 102 0 0 25 0 8 0 661009190 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29871/tid=29878] ppid=29869 vsize=3818976 CPUtime=5004.36 cores=0-7
/proc/29871/task/29878/stat : 29878 (claspmt) R 29869 29871 29772 0 -1 4202560 123928 0 0 0 500341 95 0 0 25 0 8 0 661009190 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29871/tid=29879] ppid=29869 vsize=3818976 CPUtime=5007.33 cores=0-7
/proc/29871/task/29879/stat : 29879 (claspmt) R 29869 29871 29772 0 -1 4202560 87571 0 0 0 500651 82 0 0 25 0 8 0 661009190 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4670361 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29871/tid=29880] ppid=29869 vsize=3818976 CPUtime=4994.4 cores=0-7
/proc/29871/task/29880/stat : 29880 (claspmt) R 29869 29871 29772 0 -1 4202560 323426 0 0 0 499198 242 0 0 25 0 8 0 661009191 3910631424 886634 33554432000 4194304 4964800 140733318544944 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 3818976

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.41 s]
/proc/loadavg: 8.07 8.06 8.00 2/347 30026
/proc/meminfo: memFree=10024024/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=3709888 CPUtime=40000.9 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 899783 0 0 0 3999335 755 0 0 25 0 1 0 661009104 3798925312 866187 33554432000 4194304 4964800 140733318544944 18446744073709551615 235594522039 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29871/statm: 927472 866187 476 189 0 921821 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 3709888

[startup+5009.6 s]
/proc/loadavg: 8.07 8.06 8.00 2/347 30026
/proc/meminfo: memFree=10024024/32950928 swapFree=54713408/67111528
[pid=29871] ppid=29869 vsize=678892 CPUtime=40001.1 cores=0-7
/proc/29871/stat : 29871 (claspmt) R 29869 29871 29772 0 -1 4202496 899783 0 0 0 3999340 768 0 0 25 0 1 0 661009104 695185408 91745 33554432000 4194304 4964800 140733318544944 18446744073709551615 235594910055 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29871/statm: 169723 91233 476 189 0 164072 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 678892

Child status: 0
Real time (s): 5009.62
CPU time (s): 40001.1
CPU user time (s): 39993.4
CPU system time (s): 7.70583
CPU usage (%): 798.486
Max. virtual memory (cumulated for all children) (KiB): 3818976

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.4
system time used= 7.70583
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 899784
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 123059
involuntary context switches= 192995

runsolver used 16.0806 second user time and 42.0906 second system time

The end

Launcher Data 

Begin job on node120 at 2011-05-15 23:15:03
IDJOB=3380875
IDBENCH=82467
IDSOLVER=1865
FILE ID=node120/3380875-1305494103
RUNJOBID= node120-1305493750-29786
PBS_JOBID= 13325429
Free space on /tmp= 73628 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k45-pdtpmspalu-tseitin.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380875-1305494103/watcher-3380875-1305494103 -o /tmp/evaluation-result-3380875-1305494103/solver-3380875-1305494103 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380875-1305494103.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 8dae13531986e2a9bdb6559418f57130
RANDOM SEED=317932163

node120.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13513212 kB
Buffers:         67300 kB
Cached:         999736 kB
SwapCached:      31808 kB
Active:        6463372 kB
Inactive:     12762464 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13513212 kB
SwapTotal:    67111528 kB
SwapFree:     54713408 kB
Dirty:            5088 kB
Writeback:           0 kB
AnonPages:    18132992 kB
Mapped:          19912 kB
Slab:            64152 kB
PageTables:      78236 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 66190284 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73628 MiB
End job on node120 at 2011-05-16 00:38:34