Trace number 3322786

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40000.5 5007.49

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m32-q369.75-n30-p30-s25693430.cnf
MD5SUM08b4f1e095a7a4b98cb40c2c00e78d45
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.153975
Satisfiable
(Un)Satisfiability was proved
Number of variables952
Number of clauses84809
Sum of the clauses size170510
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 284779
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 530

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...395555/instance-3322786-1304395555.cnf
0.19/0.27	c Solving...
40000.11/5007.11	
40000.11/5007.11	*** INTERRUPTED! ***
40000.11/5007.11	*** Shutting down threads...
40000.45/5007.48	s UNKNOWN
40000.45/5007.48	
40000.45/5007.48	c INTERRUPTED   : 1
40000.45/5007.48	c Models        : 0+    
40000.45/5007.48	c Time          : 5007.119s (Solving: 5006.85s 1st Model: 0.00s Unsat: 0.00s)
40000.45/5007.48	c CPU Time      : 40000.170s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322786-1304395555/watcher-3322786-1304395555 -o /tmp/evaluation-result-3322786-1304395555/solver-3322786-1304395555 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322786-1304395555.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 7.88 6.45 2/310 14780
/proc/meminfo: memFree=11271176/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=22652 CPUtime=0 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 1035 0 0 0 0 0 0 0 24 0 1 0 551157816 23195648 869 33554432000 4194304 4964704 140733927538704 18446744073709551615 212435677893 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14780/statm: 5663 869 386 189 0 537 0

[startup+0.0755199 s]
/proc/loadavg: 8.10 7.88 6.45 2/310 14780
/proc/meminfo: memFree=11271176/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=26212 CPUtime=0.06 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 2144 0 0 0 6 0 0 0 24 0 1 0 551157816 26841088 1767 33554432000 4194304 4964704 140733927538704 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14780/statm: 6553 1767 399 189 0 1427 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 26212

[startup+0.100517 s]
/proc/loadavg: 8.10 7.88 6.45 2/310 14780
/proc/meminfo: memFree=11271176/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=26212 CPUtime=0.09 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 2144 0 0 0 9 0 0 0 24 0 1 0 551157816 26841088 1767 33554432000 4194304 4964704 140733927538704 18446744073709551615 4607575 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14780/statm: 6553 1767 399 189 0 1427 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26212

[startup+0.300473 s]
/proc/loadavg: 8.10 7.88 6.45 2/310 14780
/proc/meminfo: memFree=11271176/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=385832 CPUtime=0.29 cores=0-7
/proc/14780/stat : 14780 (claspmt) S 14778 14780 14490 0 -1 4202496 2773 0 0 0 29 0 0 0 25 0 8 0 551157816 395091968 2210 33554432000 4194304 4964704 140733927538704 18446744073709551615 212447832985 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/14780/statm: 96458 2210 435 189 0 90807 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 385832

[startup+0.950808 s]
/proc/loadavg: 8.10 7.88 6.45 2/310 14780
/proc/meminfo: memFree=11271176/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=391432 CPUtime=5.55 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 14123 0 0 0 548 7 0 0 18 0 8 0 551157816 400826368 13560 33554432000 4194304 4964704 140733927538704 18446744073709551615 4380234 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 98357 16231 464 189 0 92706 0
Current children cumulated CPU time (s) 5.55
Current children cumulated vsize (KiB) 391432

[startup+1.50021 s]
/proc/loadavg: 8.18 7.90 6.46 9/318 14788
/proc/meminfo: memFree=11199720/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=394764 CPUtime=9.85 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 19322 0 0 0 976 9 0 0 18 0 8 0 551157816 404238336 18759 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670184 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 98691 18760 464 189 0 93040 0
[pid=14780/tid=14782] ppid=14778 vsize=394764 CPUtime=1.19 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 1998 0 0 0 119 0 0 0 19 0 8 0 551157844 404238336 18760 33554432000 4194304 4964704 140733927538704 18446744073709551615 4334283 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=394764 CPUtime=1.19 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 2059 0 0 0 118 1 0 0 19 0 8 0 551157844 404238336 18760 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=394764 CPUtime=1.19 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 2048 0 0 0 118 1 0 0 19 0 8 0 551157844 404238336 18760 33554432000 4194304 4964704 140733927538704 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=394764 CPUtime=1.17 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 2465 0 0 0 116 1 0 0 19 0 8 0 551157844 404238336 18761 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=394764 CPUtime=1.22 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 1459 0 0 0 122 0 0 0 19 0 8 0 551157844 404238336 18761 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=394764 CPUtime=1.18 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 2009 0 0 0 118 0 0 0 19 0 8 0 551157844 404238336 18761 33554432000 4194304 4964704 140733927538704 18446744073709551615 212436060343 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=394764 CPUtime=1.19 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 2927 0 0 0 118 1 0 0 19 0 8 0 551157844 404238336 18761 33554432000 4194304 4964704 140733927538704 18446744073709551615 4365947 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 9.85
Current children cumulated vsize (KiB) 394764

[startup+3.10091 s]
/proc/loadavg: 8.18 7.90 6.46 9/318 14788
/proc/meminfo: memFree=11157808/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=404044 CPUtime=22.55 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 31346 0 0 0 2241 14 0 0 21 0 8 0 551157816 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4343451 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 101011 30783 466 189 0 95360 0
[pid=14780/tid=14782] ppid=14778 vsize=404044 CPUtime=2.78 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 3316 0 0 0 277 1 0 0 22 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=404044 CPUtime=2.78 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 3136 0 0 0 277 1 0 0 22 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=404044 CPUtime=2.77 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 3568 0 0 0 276 1 0 0 22 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=404044 CPUtime=2.75 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 3971 0 0 0 273 2 0 0 22 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=404044 CPUtime=2.81 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 3143 0 0 0 280 1 0 0 23 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=404044 CPUtime=2.77 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 3485 0 0 0 276 1 0 0 23 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=404044 CPUtime=2.78 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 4500 0 0 0 276 2 0 0 22 0 8 0 551157844 413741056 30783 33554432000 4194304 4964704 140733927538704 18446744073709551615 4325338 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 22.55
Current children cumulated vsize (KiB) 404044

[startup+6.3013 s]
/proc/loadavg: 8.16 7.90 6.47 9/318 14788
/proc/meminfo: memFree=11110688/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=410820 CPUtime=48.11 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 42906 0 0 0 4792 19 0 0 25 0 8 0 551157816 420679680 42343 33554432000 4194304 4964704 140733927538704 18446744073709551615 4342232 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 102705 42343 466 189 0 97054 0
[pid=14780/tid=14782] ppid=14778 vsize=410820 CPUtime=5.97 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 4720 0 0 0 596 1 0 0 18 0 8 0 551157844 420679680 42343 33554432000 4194304 4964704 140733927538704 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=410820 CPUtime=5.98 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 3992 0 0 0 597 1 0 0 25 0 8 0 551157844 420679680 42343 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=410820 CPUtime=5.98 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 4966 0 0 0 596 2 0 0 25 0 8 0 551157844 420679680 42343 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=410820 CPUtime=5.91 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 5372 0 0 0 589 2 0 0 25 0 8 0 551157844 420679680 42344 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321076 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=410820 CPUtime=6.01 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 4483 0 0 0 600 1 0 0 25 0 8 0 551157844 420679680 42344 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=410820 CPUtime=5.97 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 4873 0 0 0 595 2 0 0 25 0 8 0 551157844 420679680 42344 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=410820 CPUtime=5.97 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 6117 0 0 0 595 2 0 0 20 0 8 0 551157844 420679680 42346 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321097 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 48.11
Current children cumulated vsize (KiB) 410820

[startup+12.7011 s]
/proc/loadavg: 8.15 7.90 6.48 9/318 14789
/proc/meminfo: memFree=11065484/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=414536 CPUtime=99.22 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 52836 0 0 0 9897 25 0 0 25 0 8 0 551157816 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320963 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 103634 52273 466 189 0 97983 0
[pid=14780/tid=14782] ppid=14778 vsize=414536 CPUtime=12.37 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 6128 0 0 0 1235 2 0 0 25 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670462 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=414536 CPUtime=12.38 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 4845 0 0 0 1236 2 0 0 25 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=414536 CPUtime=12.38 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 6470 0 0 0 1235 3 0 0 25 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=414536 CPUtime=12.24 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 7041 0 0 0 1221 3 0 0 25 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=414536 CPUtime=12.41 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 5810 0 0 0 1239 2 0 0 20 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4660418 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=414536 CPUtime=12.37 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 5783 0 0 0 1235 2 0 0 18 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=414536 CPUtime=12.38 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 7330 0 0 0 1234 4 0 0 25 0 8 0 551157844 424484864 52273 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 99.22
Current children cumulated vsize (KiB) 414536

[startup+25.5007 s]
/proc/loadavg: 8.12 7.91 6.49 9/318 14789
/proc/meminfo: memFree=11006796/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=426796 CPUtime=201.46 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 68191 0 0 0 20112 34 0 0 25 0 8 0 551157816 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4332488 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 106699 67625 466 189 0 101048 0
[pid=14780/tid=14782] ppid=14778 vsize=426796 CPUtime=25.17 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 8427 0 0 0 2513 4 0 0 25 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=426796 CPUtime=25.18 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 6771 0 0 0 2515 3 0 0 18 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=426796 CPUtime=25.17 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 7920 0 0 0 2514 3 0 0 25 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=426796 CPUtime=24.92 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 8633 0 0 0 2487 5 0 0 25 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=426796 CPUtime=25.2 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 7912 0 0 0 2518 2 0 0 25 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=426796 CPUtime=25.16 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 7846 0 0 0 2513 3 0 0 18 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=426796 CPUtime=25.17 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 9154 0 0 0 2512 5 0 0 25 0 8 0 551157844 437039104 67625 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 201.46
Current children cumulated vsize (KiB) 426796

[startup+51.1118 s]
/proc/loadavg: 8.13 7.93 6.54 9/318 14790
/proc/meminfo: memFree=10917724/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=509764 CPUtime=406.06 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 90371 0 0 0 40556 50 0 0 25 0 8 0 551157816 521998336 89773 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670281 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 127441 89773 466 189 0 121790 0
[pid=14780/tid=14782] ppid=14778 vsize=509764 CPUtime=50.76 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 11852 0 0 0 5069 7 0 0 25 0 8 0 551157844 521998336 89773 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=509764 CPUtime=50.78 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 8514 0 0 0 5075 3 0 0 25 0 8 0 551157844 521998336 89773 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=509764 CPUtime=50.78 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 10249 0 0 0 5073 5 0 0 25 0 8 0 551157844 521998336 89774 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=509764 CPUtime=50.26 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 11007 0 0 0 5020 6 0 0 25 0 8 0 551157844 521998336 89774 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=509764 CPUtime=50.81 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 11261 0 0 0 5076 5 0 0 25 0 8 0 551157844 521998336 89774 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=509764 CPUtime=50.76 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 10712 0 0 0 5072 4 0 0 25 0 8 0 551157844 521998336 89774 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=509764 CPUtime=50.78 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 12354 0 0 0 5070 8 0 0 25 0 8 0 551157844 521998336 89774 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.06

################
# More data... #
################

/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 54352 0 0 0 465211 75 0 0 25 0 8 0 551157844 2065944576 441613 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2017524 CPUtime=4655.08 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 65003 0 0 0 465420 88 0 0 25 0 8 0 551157844 2065944576 441613 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37245.5
Current children cumulated vsize (KiB) 2017524

[startup+4722.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/318 14926
/proc/meminfo: memFree=9467584/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2084528 CPUtime=37724.8 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 460370 0 0 0 3771976 501 0 0 25 0 8 0 551157816 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 521132 451083 466 189 0 515481 0
[pid=14780/tid=14782] ppid=14778 vsize=2084528 CPUtime=4711.11 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 55090 0 0 0 471045 66 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=2084528 CPUtime=4721.91 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 55090 0 0 0 472168 23 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=2084528 CPUtime=4721.7 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 44456 0 0 0 472116 54 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=2084528 CPUtime=4718.28 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 50797 0 0 0 471784 44 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4660369 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=2084528 CPUtime=4702.43 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 48498 0 0 0 470191 52 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=2084528 CPUtime=4712.84 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 55195 0 0 0 471209 75 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2084528 CPUtime=4714.47 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 66119 0 0 0 471358 89 0 0 25 0 8 0 551157844 2134556672 451083 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37724.8
Current children cumulated vsize (KiB) 2084528

[startup+4782.3 s]
/proc/loadavg: 8.04 8.03 8.00 9/318 14928
/proc/meminfo: memFree=9454416/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2084908 CPUtime=38204.1 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 463483 0 0 0 3819904 506 0 0 25 0 8 0 551157816 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 521227 454041 466 189 0 515576 0
[pid=14780/tid=14782] ppid=14778 vsize=2084908 CPUtime=4771.08 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 55738 0 0 0 477041 67 0 0 25 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=2084908 CPUtime=4781.92 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 55472 0 0 0 478169 23 0 0 25 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=2084908 CPUtime=4781.7 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 44536 0 0 0 478115 55 0 0 25 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=2084908 CPUtime=4778.25 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 51371 0 0 0 477781 44 0 0 18 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4644880 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=2084908 CPUtime=4762.42 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 48503 0 0 0 476189 53 0 0 25 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=2084908 CPUtime=4772.85 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 55257 0 0 0 477209 76 0 0 25 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2084908 CPUtime=4773.87 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 66119 0 0 0 477297 90 0 0 25 0 8 0 551157844 2134945792 454041 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38204.1
Current children cumulated vsize (KiB) 2084908

[startup+4842.3 s]
/proc/loadavg: 8.05 8.04 8.00 9/318 14929
/proc/meminfo: memFree=9453496/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2085156 CPUtime=38683.5 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 463868 0 0 0 3867841 507 0 0 25 0 8 0 551157816 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4627290 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 521289 454395 466 189 0 515638 0
[pid=14780/tid=14782] ppid=14778 vsize=2085156 CPUtime=4831.08 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 55876 0 0 0 483041 67 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=2085156 CPUtime=4841.92 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 55478 0 0 0 484169 23 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=2085156 CPUtime=4841.7 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 44595 0 0 0 484115 55 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=2085156 CPUtime=4838.24 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 51375 0 0 0 483779 45 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=2085156 CPUtime=4822.42 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 48660 0 0 0 482189 53 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=2085156 CPUtime=4832.84 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 55273 0 0 0 483208 76 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2085156 CPUtime=4833.27 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 66124 0 0 0 483237 90 0 0 25 0 8 0 551157844 2135199744 454395 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38683.5
Current children cumulated vsize (KiB) 2085156

[startup+4902.3 s]
/proc/loadavg: 8.13 8.06 8.01 9/318 14931
/proc/meminfo: memFree=9450620/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2086624 CPUtime=39162.8 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 464523 0 0 0 3915768 509 0 0 25 0 8 0 551157816 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 521656 455050 466 189 0 516005 0
[pid=14780/tid=14782] ppid=14778 vsize=2086624 CPUtime=4891.03 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 56090 0 0 0 489035 68 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=2086624 CPUtime=4901.89 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 55483 0 0 0 490166 23 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=2086624 CPUtime=4901.68 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 44604 0 0 0 490113 55 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=2086624 CPUtime=4898.23 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 51452 0 0 0 489778 45 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=2086624 CPUtime=4882.42 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 49000 0 0 0 488189 53 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=2086624 CPUtime=4892.84 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 55280 0 0 0 489208 76 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2086624 CPUtime=4892.66 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 66127 0 0 0 489176 90 0 0 25 0 8 0 551157844 2136702976 455050 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39162.8
Current children cumulated vsize (KiB) 2086624

[startup+4962.3 s]
/proc/loadavg: 8.12 8.06 8.00 9/318 14933
/proc/meminfo: memFree=9450092/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2086624 CPUtime=39642.1 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 464630 0 0 0 3963704 510 0 0 25 0 8 0 551157816 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4332387 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 521656 455157 466 189 0 516005 0
[pid=14780/tid=14782] ppid=14778 vsize=2086624 CPUtime=4951.03 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 56090 0 0 0 495035 68 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=2086624 CPUtime=4961.89 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 55483 0 0 0 496166 23 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=2086624 CPUtime=4961.68 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 44604 0 0 0 496113 55 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=2086624 CPUtime=4958.19 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 51456 0 0 0 495774 45 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=2086624 CPUtime=4942.42 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 49084 0 0 0 494189 53 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=2086624 CPUtime=4952.84 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 55280 0 0 0 495208 76 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2086624 CPUtime=4952.06 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 66144 0 0 0 495116 90 0 0 25 0 8 0 551157844 2136702976 455157 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39642.1
Current children cumulated vsize (KiB) 2086624



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.1 s]
/proc/loadavg: 8.05 8.05 8.00 9/318 14934
/proc/meminfo: memFree=9447704/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2087476 CPUtime=40000.1 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 465339 0 0 0 3999500 511 0 0 25 0 8 0 551157816 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320966 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/14780/statm: 521869 455711 466 189 0 516218 0
[pid=14780/tid=14782] ppid=14778 vsize=2087476 CPUtime=4995.63 cores=0-7
/proc/14780/task/14782/stat : 14782 (claspmt) R 14778 14780 14490 0 -1 4202560 56277 0 0 0 499494 69 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14780/tid=14783] ppid=14778 vsize=2087476 CPUtime=5006.71 cores=0-7
/proc/14780/task/14783/stat : 14783 (claspmt) R 14778 14780 14490 0 -1 4202560 55483 0 0 0 500648 23 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14780/tid=14784] ppid=14778 vsize=2087476 CPUtime=5006.49 cores=0-7
/proc/14780/task/14784/stat : 14784 (claspmt) R 14778 14780 14490 0 -1 4202560 44607 0 0 0 500593 56 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14780/tid=14785] ppid=14778 vsize=2087476 CPUtime=5002.99 cores=0-7
/proc/14780/task/14785/stat : 14785 (claspmt) R 14778 14780 14490 0 -1 4202560 51519 0 0 0 500254 45 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4654500 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14780/tid=14786] ppid=14778 vsize=2087476 CPUtime=4987.24 cores=0-7
/proc/14780/task/14786/stat : 14786 (claspmt) R 14778 14780 14490 0 -1 4202560 49364 0 0 0 498671 53 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14780/tid=14787] ppid=14778 vsize=2087476 CPUtime=4997.64 cores=0-7
/proc/14780/task/14787/stat : 14787 (claspmt) R 14778 14780 14490 0 -1 4202560 55280 0 0 0 499688 76 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14780/tid=14788] ppid=14778 vsize=2087476 CPUtime=4996.58 cores=0-7
/proc/14780/task/14788/stat : 14788 (claspmt) R 14778 14780 14490 0 -1 4202560 66319 0 0 0 499568 90 0 0 25 0 8 0 551157844 2137575424 455711 33554432000 4194304 4964704 140733927538704 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 2087476

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/318 14934
/proc/meminfo: memFree=9447704/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2082612 CPUtime=40000.4 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 465349 0 0 0 3999524 511 0 0 25 0 1 0 551157816 2132594688 453819 33554432000 4194304 4964704 140733927538704 18446744073709551615 4646564 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14780/statm: 520653 453819 476 189 0 515002 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 2082612

[startup+5007.4 s]
/proc/loadavg: 8.05 8.05 8.00 9/318 14934
/proc/meminfo: memFree=9447704/32951124 swapFree=67111416/67111528
[pid=14780] ppid=14778 vsize=2066100 CPUtime=40000.4 cores=0-7
/proc/14780/stat : 14780 (claspmt) R 14778 14780 14490 0 -1 4202496 465349 0 0 0 3999534 511 0 0 25 0 1 0 551157816 2115686400 446796 33554432000 4194304 4964704 140733927538704 18446744073709551615 212435674945 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14780/statm: 516525 446796 476 189 0 510874 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 2066100

Child status: 0
Real time (s): 5007.49
CPU time (s): 40000.5
CPU user time (s): 39995.4
CPU system time (s): 5.19121
CPU usage (%): 798.815
Max. virtual memory (cumulated for all children) (KiB): 2087476

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.4
system time used= 5.19121
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 465350
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 188105
involuntary context switches= 189919

runsolver used 13.5759 second user time and 36.5035 second system time

The end

Launcher Data 

Begin job on node102 at 2011-05-03 06:05:55
IDJOB=3322786
IDBENCH=88227
IDSOLVER=1711
FILE ID=node102/3322786-1304395555
RUNJOBID= node102-1304390340-14504
PBS_JOBID= 13181832
Free space on /tmp= 73580 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m32-q369.75-n30-p30-s25693430.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322786-1304395555/watcher-3322786-1304395555 -o /tmp/evaluation-result-3322786-1304395555/solver-3322786-1304395555 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322786-1304395555.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 08b4f1e095a7a4b98cb40c2c00e78d45
RANDOM SEED=2006689011

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      11271464 kB
Buffers:        680660 kB
Cached:        2068236 kB
SwapCached:          0 kB
Active:       18959992 kB
Inactive:      2244332 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      11271464 kB
SwapTotal:    67111528 kB
SwapFree:     67111416 kB
Dirty:            1972 kB
Writeback:           0 kB
AnonPages:    18456908 kB
Mapped:          27268 kB
Slab:           357068 kB
PageTables:      52576 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 59036720 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73580 MiB
End job on node102 at 2011-05-03 07:29:24