Trace number 3322426

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 34711.1 5100.8

General information on the benchmark

Namecrafted/skvortsov/automata-synchronization/
crn_40_1521_s.cnf
MD5SUM4e35cb9894b1ca841d462da2d6957cc5
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables63882
Number of clauses123981
Sum of the clauses size369562
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22381
Number of clauses of size 3121600
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...385189/instance-3322426-1304385189.cnf
0.49/0.54	c Solving...
34710.27/5100.01	
34710.27/5100.01	*** INTERRUPTED! ***
34710.27/5100.01	*** Shutting down threads...
34711.00/5100.74	s UNKNOWN
34711.00/5100.74	
34711.00/5100.74	c INTERRUPTED   : 1
34711.00/5100.74	c Models        : 0+    
34711.00/5100.74	c Time          : 5100.017s (Solving: 5099.47s 1st Model: 0.00s Unsat: 0.00s)
34711.00/5100.74	c CPU Time      : 34710.310s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322426-1304385189/watcher-3322426-1304385189 -o /tmp/evaluation-result-3322426-1304385189/solver-3322426-1304385189 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322426-1304385189.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.98 9.90 10.02 3/350 28045
/proc/meminfo: memFree=25292112/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=32800 CPUtime=0 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 3862 0 0 0 0 0 0 0 22 0 1 0 530934422 33587200 3338 33554432000 4194304 4964704 140737230475184 18446744073709551615 4679014 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28045/statm: 8200 3343 378 189 0 3074 0

[startup+0.0645131 s]
/proc/loadavg: 8.98 9.90 10.02 3/350 28045
/proc/meminfo: memFree=25292112/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=46300 CPUtime=0.06 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 8283 0 0 0 5 1 0 0 22 0 1 0 530934422 47411200 6617 33554432000 4194304 4964704 140737230475184 18446744073709551615 47189147520696 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28045/statm: 11575 6618 397 189 0 6449 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 46300

[startup+0.100504 s]
/proc/loadavg: 8.98 9.90 10.02 3/350 28045
/proc/meminfo: memFree=25292112/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=49844 CPUtime=0.09 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 9157 0 0 0 8 1 0 0 22 0 1 0 530934422 51040256 7491 33554432000 4194304 4964704 140737230475184 18446744073709551615 4614542 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28045/statm: 12461 7492 400 189 0 7335 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 49844

[startup+0.300468 s]
/proc/loadavg: 8.98 9.90 10.02 3/350 28045
/proc/meminfo: memFree=25292112/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=55388 CPUtime=0.29 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 10541 0 0 0 28 1 0 0 23 0 1 0 530934422 56717312 8875 33554432000 4194304 4964704 140737230475184 18446744073709551615 4621898 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28045/statm: 13847 8875 401 189 0 8721 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 55388

[startup+0.796345 s]
/proc/loadavg: 8.98 9.90 10.02 3/350 28045
/proc/meminfo: memFree=25292112/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=519280 CPUtime=1.36 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 41678 0 0 0 122 14 0 0 17 0 8 0 530934422 531742720 35434 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370236 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28045/statm: 129820 35434 464 189 0 124169 0
Current children cumulated CPU time (s) 1.36
Current children cumulated vsize (KiB) 519280

[startup+1.50122 s]
/proc/loadavg: 8.98 9.90 10.02 11/358 28053
/proc/meminfo: memFree=25150076/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=519948 CPUtime=5.57 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 43745 0 0 0 542 15 0 0 18 0 8 0 530934422 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095118298 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28045/statm: 129987 37501 464 189 0 124336 0
[pid=28045/tid=28047] ppid=28043 vsize=519948 CPUtime=0.81 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 3549 0 0 0 80 1 0 0 18 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 4668629 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=519948 CPUtime=0.56 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 5056 0 0 0 55 1 0 0 17 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670230 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=519948 CPUtime=0.57 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 4861 0 0 0 54 3 0 0 18 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=519948 CPUtime=0.69 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 3217 0 0 0 69 0 0 0 17 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=519948 CPUtime=0.45 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 4471 0 0 0 43 2 0 0 18 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=519948 CPUtime=0.59 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 4290 0 0 0 58 1 0 0 18 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095148217 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=519948 CPUtime=0.5 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 4902 0 0 0 48 2 0 0 18 0 8 0 530934477 532426752 37501 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.57
Current children cumulated vsize (KiB) 519948

[startup+3.10092 s]
/proc/loadavg: 8.98 9.90 10.02 11/358 28053
/proc/meminfo: memFree=25135196/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=522428 CPUtime=15.15 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 49178 0 0 0 1497 18 0 0 21 0 8 0 530934422 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370229 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28045/statm: 130607 42934 464 189 0 124956 0
[pid=28045/tid=28047] ppid=28043 vsize=522428 CPUtime=2.4 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 3721 0 0 0 239 1 0 0 21 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=522428 CPUtime=1.68 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 6250 0 0 0 167 1 0 0 18 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=522428 CPUtime=1.29 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 5500 0 0 0 126 3 0 0 18 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=522428 CPUtime=1.56 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 3647 0 0 0 155 1 0 0 18 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=522428 CPUtime=2.04 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 4976 0 0 0 202 2 0 0 21 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370462 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=522428 CPUtime=1.49 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 5272 0 0 0 147 2 0 0 19 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=522428 CPUtime=1.68 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 6065 0 0 0 165 3 0 0 19 0 8 0 530934477 534966272 42934 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 15.15
Current children cumulated vsize (KiB) 522428

[startup+6.30033 s]
/proc/loadavg: 9.06 9.90 10.02 11/358 28053
/proc/meminfo: memFree=25098160/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=525252 CPUtime=34.29 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 56520 0 0 0 3407 22 0 0 19 0 8 0 530934422 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28045/statm: 131313 50276 464 189 0 125662 0
[pid=28045/tid=28047] ppid=28043 vsize=525252 CPUtime=4.7 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 3835 0 0 0 469 1 0 0 25 0 8 0 530934477 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=525252 CPUtime=3.87 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 7644 0 0 0 385 2 0 0 22 0 8 0 530934477 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=525252 CPUtime=3.2 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 6480 0 0 0 316 4 0 0 20 0 8 0 530934477 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4660874 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=525252 CPUtime=4.7 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 4333 0 0 0 469 1 0 0 25 0 8 0 530934477 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370443 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=525252 CPUtime=4.25 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 5594 0 0 0 423 2 0 0 18 0 8 0 530934477 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=525252 CPUtime=3.73 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 6612 0 0 0 370 3 0 0 22 0 8 0 530934477 537858048 50276 33554432000 4194304 4964704 140737230475184 18446744073709551615 4320911 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=525252 CPUtime=3.64 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 7364 0 0 0 360 4 0 0 24 0 8 0 530934477 537858048 50277 33554432000 4194304 4964704 140737230475184 18446744073709551615 4668531 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 34.29
Current children cumulated vsize (KiB) 525252

[startup+12.7012 s]
/proc/loadavg: 9.13 9.90 10.02 11/358 28053
/proc/meminfo: memFree=25051744/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=530592 CPUtime=72.59 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 67611 0 0 0 7232 27 0 0 25 0 8 0 530934422 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370224 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28045/statm: 132648 61367 466 189 0 126997 0
[pid=28045/tid=28047] ppid=28043 vsize=530592 CPUtime=9.29 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 4267 0 0 0 928 1 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=530592 CPUtime=7.18 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 9538 0 0 0 715 3 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=530592 CPUtime=6.96 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 7802 0 0 0 692 4 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 4660816 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=530592 CPUtime=10.76 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 5177 0 0 0 1075 1 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=530592 CPUtime=8.95 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 6346 0 0 0 893 2 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=530592 CPUtime=8.83 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 8219 0 0 0 880 3 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 4660304 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=530592 CPUtime=8.77 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 11093 0 0 0 871 6 0 0 25 0 8 0 530934477 543326208 61367 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 72.59
Current children cumulated vsize (KiB) 530592

[startup+25.5008 s]
/proc/loadavg: 9.33 9.91 10.01 11/358 28053
/proc/meminfo: memFree=24988776/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=538744 CPUtime=149.16 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 83658 0 0 0 14879 37 0 0 25 0 8 0 530934422 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370224 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28045/statm: 134686 77384 466 189 0 129035 0
[pid=28045/tid=28047] ppid=28043 vsize=538744 CPUtime=17.94 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 5107 0 0 0 1793 1 0 0 25 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=538744 CPUtime=17.21 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 12811 0 0 0 1717 4 0 0 19 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=538744 CPUtime=17.53 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 10874 0 0 0 1747 6 0 0 25 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=538744 CPUtime=21.36 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 6991 0 0 0 2134 2 0 0 25 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=538744 CPUtime=19.47 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 7454 0 0 0 1944 3 0 0 25 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=538744 CPUtime=18.01 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 11054 0 0 0 1796 5 0 0 25 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 4669196 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=538744 CPUtime=15.95 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 13163 0 0 0 1587 8 0 0 25 0 8 0 530934477 551673856 77384 33554432000 4194304 4964704 140737230475184 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 149.16
Current children cumulated vsize (KiB) 538744

[startup+51.1011 s]
/proc/loadavg: 9.62 9.93 10.02 11/358 28054
/proc/meminfo: memFree=24895248/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=559296 CPUtime=302.37 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 107626 0 0 0 30189 48 0 0 25 0 8 0 530934422 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670416 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28045/statm: 139824 101294 466 189 0 134173 0
[pid=28045/tid=28047] ppid=28043 vsize=559296 CPUtime=37.88 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 6680 0 0 0 3786 2 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=559296 CPUtime=37.22 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 17315 0 0 0 3716 6 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 4321061 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=559296 CPUtime=38.86 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 14454 0 0 0 3878 8 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=559296 CPUtime=39.6 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 8935 0 0 0 3957 3 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370352 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=559296 CPUtime=41.74 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 11652 0 0 0 4169 5 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=559296 CPUtime=36.61 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 15192 0 0 0 3654 7 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 4654493 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=559296 CPUtime=32.94 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 16062 0 0 0 3284 10 0 0 25 0 8 0 530934477 572719104 101294 33554432000 4194304 4964704 140737230475184 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 302.37

################
# More data... #
################

/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 174949 0 0 0 396883 109 0 0 25 0 8 0 530934477 4546744320 1007595 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4440180 CPUtime=3969.74 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 193230 0 0 0 396837 137 0 0 25 0 8 0 530934477 4546744320 1007595 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 32485.5
Current children cumulated vsize (KiB) 4440180

[startup+4842.3 s]
/proc/loadavg: 9.08 9.08 9.08 10/357 28695
/proc/meminfo: memFree=21321256/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4442912 CPUtime=32909.4 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1038215 0 0 0 3290277 659 0 0 25 0 8 0 530934422 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28045/statm: 1110728 1011523 466 189 0 1105077 0
[pid=28045/tid=28047] ppid=28043 vsize=4442912 CPUtime=4206.6 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 76316 0 0 0 420614 46 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=4442912 CPUtime=4179.87 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 151410 0 0 0 417909 78 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=4442912 CPUtime=3819.46 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 127358 0 0 0 381871 75 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=4442912 CPUtime=4329.89 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 110206 0 0 0 432923 66 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4320989 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=4442912 CPUtime=4267.64 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 104304 0 0 0 426698 66 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=4442912 CPUtime=4029.23 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 175339 0 0 0 402813 110 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4442912 CPUtime=4028.26 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 194462 0 0 0 402689 137 0 0 25 0 8 0 530934477 4549541888 1011523 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 32909.4
Current children cumulated vsize (KiB) 4442912

[startup+4902.3 s]
/proc/loadavg: 9.03 9.06 9.08 10/357 28696
/proc/meminfo: memFree=21302364/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4576516 CPUtime=33328.6 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1043146 0 0 0 3332200 662 0 0 25 0 8 0 530934422 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4320928 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/28045/statm: 1144129 1016415 466 189 0 1138478 0
[pid=28045/tid=28047] ppid=28043 vsize=4576516 CPUtime=4255.22 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 76595 0 0 0 425476 46 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=4576516 CPUtime=4239.82 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 152521 0 0 0 423904 78 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=4576516 CPUtime=3877.65 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 128445 0 0 0 387689 76 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=4576516 CPUtime=4368.73 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 110719 0 0 0 436806 67 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=4576516 CPUtime=4325.94 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 104709 0 0 0 432528 66 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=4576516 CPUtime=4088.53 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 176119 0 0 0 408743 110 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4576516 CPUtime=4088.26 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 194764 0 0 0 408689 137 0 0 25 0 8 0 530934477 4686352384 1016415 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670560 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 33328.6
Current children cumulated vsize (KiB) 4576516

[startup+4962.3 s]
/proc/loadavg: 9.01 9.05 9.07 10/357 28698
/proc/meminfo: memFree=21275528/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4581908 CPUtime=33747.9 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1049734 0 0 0 3374125 668 0 0 25 0 8 0 530934422 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4332409 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/28045/statm: 1145477 1022901 466 189 0 1139826 0
[pid=28045/tid=28047] ppid=28043 vsize=4581908 CPUtime=4307.62 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 77806 0 0 0 430715 47 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=4581908 CPUtime=4299.83 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 152838 0 0 0 429905 78 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=4581908 CPUtime=3937.65 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 130410 0 0 0 393688 77 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=4581908 CPUtime=4399.02 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 111390 0 0 0 439835 67 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370259 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=4581908 CPUtime=4363.24 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 104920 0 0 0 436258 66 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=4581908 CPUtime=4147.84 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 176877 0 0 0 414674 110 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4581908 CPUtime=4148.26 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 195128 0 0 0 414688 138 0 0 25 0 8 0 530934477 4691873792 1022901 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 33747.9
Current children cumulated vsize (KiB) 4581908

[startup+5022.3 s]
/proc/loadavg: 9.08 9.06 9.08 10/357 28700
/proc/meminfo: memFree=21252912/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4584172 CPUtime=34167.2 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1055802 0 0 0 3416051 671 0 0 25 0 8 0 530934422 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370269 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/28045/statm: 1146043 1028510 466 189 0 1140392 0
[pid=28045/tid=28047] ppid=28043 vsize=4584172 CPUtime=4365.03 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 78178 0 0 0 436456 47 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4669427 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=4584172 CPUtime=4359.83 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 153866 0 0 0 435905 78 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4660188 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=4584172 CPUtime=3995.65 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 131750 0 0 0 399487 78 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=4584172 CPUtime=4449.32 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 112188 0 0 0 444864 68 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=4584172 CPUtime=4413.43 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 105591 0 0 0 441276 67 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=4584172 CPUtime=4195.84 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 177550 0 0 0 419474 110 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4584172 CPUtime=4208.26 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 195958 0 0 0 420688 138 0 0 25 0 8 0 530934477 4694192128 1028510 33554432000 4194304 4964704 140737230475184 18446744073709551615 4365155 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 34167.2
Current children cumulated vsize (KiB) 4584172

[startup+5082.3 s]
/proc/loadavg: 9.06 9.06 9.07 10/357 28701
/proc/meminfo: memFree=21226856/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4594372 CPUtime=34586.5 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1062625 0 0 0 3457974 675 0 0 25 0 8 0 530934422 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4342199 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/28045/statm: 1148593 1035001 466 189 0 1142942 0
[pid=28045/tid=28047] ppid=28043 vsize=4594372 CPUtime=4423.65 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 78467 0 0 0 442317 48 0 0 25 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=4594372 CPUtime=4410.09 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 155263 0 0 0 440930 79 0 0 25 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4660310 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=4594372 CPUtime=4041.9 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 133924 0 0 0 404111 79 0 0 25 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=4594372 CPUtime=4496.44 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 112705 0 0 0 449576 68 0 0 25 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=4594372 CPUtime=4471.31 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 105724 0 0 0 447064 67 0 0 25 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=4594372 CPUtime=4255.84 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 178890 0 0 0 425473 111 0 0 25 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4594372 CPUtime=4265.08 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 196208 0 0 0 426369 139 0 0 20 0 8 0 530934477 4704636928 1035001 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34586.5
Current children cumulated vsize (KiB) 4594372



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.04 9.06 9.07 10/357 28702
/proc/meminfo: memFree=21215816/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4596420 CPUtime=34710.3 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1065347 0 0 0 3470350 677 0 0 25 0 8 0 530934422 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/28045/statm: 1149105 1037723 466 189 0 1143454 0
[pid=28045/tid=28047] ppid=28043 vsize=4596420 CPUtime=4441.36 cores=0-7
/proc/28045/task/28047/stat : 28047 (claspmt) R 28043 28045 26224 0 -1 4202560 78467 0 0 0 444088 48 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28045/tid=28048] ppid=28043 vsize=4596420 CPUtime=4427.8 cores=0-7
/proc/28045/task/28048/stat : 28048 (claspmt) R 28043 28045 26224 0 -1 4202560 155824 0 0 0 442701 79 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28045/tid=28049] ppid=28043 vsize=4596420 CPUtime=4055.2 cores=0-7
/proc/28045/task/28049/stat : 28049 (claspmt) R 28043 28045 26224 0 -1 4202560 134254 0 0 0 405441 79 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28045/tid=28050] ppid=28043 vsize=4596420 CPUtime=4510.61 cores=0-7
/proc/28045/task/28050/stat : 28050 (claspmt) R 28043 28045 26224 0 -1 4202560 113139 0 0 0 450993 68 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28051] ppid=28043 vsize=4596420 CPUtime=4480.35 cores=0-7
/proc/28045/task/28051/stat : 28051 (claspmt) R 28043 28045 26224 0 -1 4202560 105724 0 0 0 447968 67 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28045/tid=28052] ppid=28043 vsize=4596420 CPUtime=4273.55 cores=0-7
/proc/28045/task/28052/stat : 28052 (claspmt) R 28043 28045 26224 0 -1 4202560 179851 0 0 0 427244 111 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4362990 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28045/tid=28053] ppid=28043 vsize=4596420 CPUtime=4281.49 cores=0-7
/proc/28045/task/28053/stat : 28053 (claspmt) R 28043 28045 26224 0 -1 4202560 196439 0 0 0 428010 139 0 0 25 0 8 0 530934477 4706734080 1037723 33554432000 4194304 4964704 140737230475184 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34710.3
Current children cumulated vsize (KiB) 4596420

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 9.04 9.06 9.07 3/351 28703
/proc/meminfo: memFree=21287908/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=4552868 CPUtime=34710.9 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202496 1065357 0 0 0 3470413 677 0 0 25 0 1 0 530934422 4662136832 1019384 33554432000 4194304 4964704 140737230475184 18446744073709551615 228095108882 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/28045/statm: 1138217 1019384 476 189 0 1132566 0
Current children cumulated CPU time (s) 34710.9
Current children cumulated vsize (KiB) 4552868

[startup+5100.8 s]
/proc/loadavg: 9.04 9.06 9.07 3/351 28703
/proc/meminfo: memFree=21287908/32951124 swapFree=67091720/67111528
[pid=28045] ppid=28043 vsize=0 CPUtime=34711.1 cores=0-7
/proc/28045/stat : 28045 (claspmt) R 28043 28045 26224 0 -1 4202500 1065358 0 0 0 3470414 695 0 0 25 0 1 0 530934422 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/28045/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 34711.1
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 5100.8
CPU time (s): 34711.1
CPU user time (s): 34704.1
CPU system time (s): 6.95494
CPU usage (%): 680.503
Max. virtual memory (cumulated for all children) (KiB): 4596420

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 34704.1
system time used= 6.95494
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1065358
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 148477
involuntary context switches= 253786

runsolver used 16.7854 second user time and 41.4157 second system time

The end

Launcher Data 

Begin job on node142 at 2011-05-03 03:13:10
IDJOB=3322426
IDBENCH=83191
IDSOLVER=1711
FILE ID=node142/3322426-1304385189
RUNJOBID= node142-1304366464-26238
PBS_JOBID= 13181731
Free space on /tmp= 73252 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/skvortsov/automata-synchronization/crn_40_1521_s.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322426-1304385189/watcher-3322426-1304385189 -o /tmp/evaluation-result-3322426-1304385189/solver-3322426-1304385189 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322426-1304385189.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4e35cb9894b1ca841d462da2d6957cc5
RANDOM SEED=2145676541

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25292268 kB
Buffers:        358492 kB
Cached:        1485184 kB
SwapCached:       3812 kB
Active:        4562040 kB
Inactive:      2688684 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25292268 kB
SwapTotal:    67111528 kB
SwapFree:     67091720 kB
Dirty:            3644 kB
Writeback:           0 kB
AnonPages:     5403524 kB
Mapped:          28288 kB
Slab:           308268 kB
PageTables:      32004 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 77295056 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73472 MiB
End job on node142 at 2011-05-03 04:38:12