Trace number 3322621

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 35729.1 5100.11

General information on the benchmark

NameSAT04/handmade/marijn/equilarge-l/
equilarge_l2.used-as.sat04-894.cnf
MD5SUMb3ddb2ee2bb2b83e1c98b2a9018351fc
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark930.14
Satisfiable
(Un)Satisfiability was proved
Number of variables4487
Number of clauses18555
Sum of the clauses size54310
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21355
Number of clauses of size 317200
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...391748/instance-3322621-1304391748.cnf
0.02/0.04	c Solving...
35728.97/5100.01	
35728.97/5100.01	*** INTERRUPTED! ***
35728.97/5100.01	*** Shutting down threads...
35728.97/5100.10	s UNKNOWN
35728.97/5100.10	
35728.97/5100.10	c INTERRUPTED   : 1
35728.97/5100.10	c Models        : 0+    
35728.97/5100.10	c Time          : 5100.014s (Solving: 5099.97s 1st Model: 0.00s Unsat: 0.00s)
35728.97/5100.10	c CPU Time      : 35729.020s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322621-1304391748/watcher-3322621-1304391748 -o /tmp/evaluation-result-3322621-1304391748/solver-3322621-1304391748 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322621-1304391748.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.37 8.89 8.96 3/285 19384
/proc/meminfo: memFree=5070304/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=23304 CPUtime=0 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 1215 0 0 0 0 0 0 0 25 0 1 0 550777582 23863296 1030 33554432000 4194304 4964704 140734491793856 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/19384/statm: 5826 1030 397 189 0 700 0

[startup+0.0288019 s]
/proc/loadavg: 8.37 8.89 8.96 3/285 19384
/proc/meminfo: memFree=5070304/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=24272 CPUtime=0.02 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 1464 0 0 0 2 0 0 0 25 0 1 0 550777582 24854528 1279 33554432000 4194304 4964704 140734491793856 18446744073709551615 218370618053 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/19384/statm: 6068 1280 400 189 0 942 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 24272

[startup+0.132848 s]
/proc/loadavg: 8.37 8.89 8.96 3/285 19384
/proc/meminfo: memFree=5070304/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=385860 CPUtime=0.53 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 4648 0 0 0 52 1 0 0 18 0 8 0 550777582 395120640 4134 33554432000 4194304 4964704 140734491793856 18446744073709551615 4456583 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/19384/statm: 96465 4134 464 189 0 90814 0
Current children cumulated CPU time (s) 0.53
Current children cumulated vsize (KiB) 385860

[startup+0.300708 s]
/proc/loadavg: 8.37 8.89 8.96 3/285 19384
/proc/meminfo: memFree=5070304/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=386256 CPUtime=1.7 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 5380 0 0 0 169 1 0 0 18 0 8 0 550777582 395526144 4866 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670565 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/19384/statm: 96564 4867 464 189 0 90913 0
Current children cumulated CPU time (s) 1.7
Current children cumulated vsize (KiB) 386256

[startup+0.700637 s]
/proc/loadavg: 8.37 8.89 8.96 3/285 19384
/proc/meminfo: memFree=5070304/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=387336 CPUtime=4.5 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 6775 0 0 0 448 2 0 0 19 0 8 0 550777582 396632064 6261 33554432000 4194304 4964704 140734491793856 18446744073709551615 4336761 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/19384/statm: 96834 6261 464 189 0 91183 0
Current children cumulated CPU time (s) 4.5
Current children cumulated vsize (KiB) 387336

[startup+1.50049 s]
/proc/loadavg: 8.37 8.89 8.96 10/293 19392
/proc/meminfo: memFree=5047180/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=388268 CPUtime=10.08 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 8364 0 0 0 1005 3 0 0 19 0 8 0 550777582 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668632 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19384/statm: 97067 7850 464 189 0 91416 0
[pid=19384/tid=19386] ppid=19382 vsize=388268 CPUtime=1.18 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 930 0 0 0 118 0 0 0 19 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4320953 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=388268 CPUtime=1.19 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 796 0 0 0 119 0 0 0 19 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 218371062254 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=388268 CPUtime=1.42 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 934 0 0 0 142 0 0 0 19 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668816 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=388268 CPUtime=1.43 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 797 0 0 0 143 0 0 0 19 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668632 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=388268 CPUtime=1.42 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 874 0 0 0 142 0 0 0 20 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669997 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=388268 CPUtime=1.25 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 880 0 0 0 125 0 0 0 19 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=388268 CPUtime=1.41 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 1053 0 0 0 141 0 0 0 19 0 8 0 550777587 397586432 7850 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.08
Current children cumulated vsize (KiB) 388268

[startup+3.10121 s]
/proc/loadavg: 8.37 8.89 8.96 10/293 19392
/proc/meminfo: memFree=5039864/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=389276 CPUtime=21.27 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 9990 0 0 0 2123 4 0 0 24 0 8 0 550777582 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4321272 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19384/statm: 97319 9476 464 189 0 91668 0
[pid=19384/tid=19386] ppid=19382 vsize=389276 CPUtime=2.1 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 1065 0 0 0 210 0 0 0 19 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=389276 CPUtime=2.64 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 1110 0 0 0 264 0 0 0 21 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 218371062254 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=389276 CPUtime=3.02 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 1186 0 0 0 302 0 0 0 24 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4363158 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=389276 CPUtime=3.03 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 1070 0 0 0 303 0 0 0 24 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=389276 CPUtime=3.01 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 1123 0 0 0 301 0 0 0 25 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=389276 CPUtime=2.07 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 990 0 0 0 207 0 0 0 19 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=389276 CPUtime=3.01 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 1189 0 0 0 301 0 0 0 24 0 8 0 550777587 398618624 9476 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 21.27
Current children cumulated vsize (KiB) 389276

[startup+6.30064 s]
/proc/loadavg: 8.50 8.91 8.96 10/293 19392
/proc/meminfo: memFree=5027864/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=391160 CPUtime=43.62 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 13468 0 0 0 4356 6 0 0 25 0 8 0 550777582 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19384/statm: 97790 12954 464 189 0 92139 0
[pid=19384/tid=19386] ppid=19382 vsize=391160 CPUtime=4.99 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 1270 0 0 0 499 0 0 0 25 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 218371062254 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=391160 CPUtime=4.56 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 1343 0 0 0 456 0 0 0 25 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=391160 CPUtime=6.22 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 1680 0 0 0 622 0 0 0 25 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=391160 CPUtime=5.74 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 1393 0 0 0 573 1 0 0 25 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=391160 CPUtime=6.18 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 1674 0 0 0 618 0 0 0 25 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=391160 CPUtime=4.15 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 1313 0 0 0 415 0 0 0 23 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=391160 CPUtime=6.21 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 2101 0 0 0 620 1 0 0 25 0 8 0 550777587 400547840 12954 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668668 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 43.62
Current children cumulated vsize (KiB) 391160

[startup+12.7005 s]
/proc/loadavg: 8.54 8.91 8.96 10/293 19392
/proc/meminfo: memFree=5005248/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=459236 CPUtime=88.29 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 18059 0 0 0 8819 10 0 0 25 0 8 0 550777582 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4660755 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19384/statm: 114809 17545 466 189 0 109158 0
[pid=19384/tid=19386] ppid=19382 vsize=459236 CPUtime=10.6 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 1536 0 0 0 1060 0 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4362287 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=459236 CPUtime=10.95 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 2059 0 0 0 1095 0 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=459236 CPUtime=10.4 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 1986 0 0 0 1039 1 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=459236 CPUtime=11.62 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 1847 0 0 0 1161 1 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=459236 CPUtime=10.46 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 2326 0 0 0 1045 1 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=459236 CPUtime=10.53 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 1990 0 0 0 1053 0 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=459236 CPUtime=12.47 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 3012 0 0 0 1245 2 0 0 25 0 8 0 550777587 470257664 17545 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 88.29
Current children cumulated vsize (KiB) 459236

[startup+25.5012 s]
/proc/loadavg: 8.64 8.92 8.96 10/293 19392
/proc/meminfo: memFree=4986628/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=461904 CPUtime=177.74 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 22221 0 0 0 17759 15 0 0 25 0 8 0 550777582 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668505 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/19384/statm: 115476 21626 466 189 0 109825 0
[pid=19384/tid=19386] ppid=19382 vsize=461904 CPUtime=23.26 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 2032 0 0 0 2325 1 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=461904 CPUtime=23.75 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 2486 0 0 0 2374 1 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=461904 CPUtime=19.59 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 2217 0 0 0 1958 1 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=461904 CPUtime=20.05 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 2212 0 0 0 2004 1 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=461904 CPUtime=21.81 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 3354 0 0 0 2179 2 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=461904 CPUtime=23.17 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 2284 0 0 0 2316 1 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=461904 CPUtime=25.27 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 3826 0 0 0 2524 3 0 0 25 0 8 0 550777587 472989696 21626 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 177.74
Current children cumulated vsize (KiB) 461904

[startup+51.1007 s]
/proc/loadavg: 8.76 8.92 8.96 10/293 19393
/proc/meminfo: memFree=4954932/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=466180 CPUtime=356.68 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 30198 0 0 0 35646 22 0 0 25 0 8 0 550777582 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/19384/statm: 116545 29521 466 189 0 110894 0
[pid=19384/tid=19386] ppid=19382 vsize=466180 CPUtime=47.22 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 2707 0 0 0 4720 2 0 0 25 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=466180 CPUtime=49.35 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 3353 0 0 0 4934 1 0 0 25 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=466180 CPUtime=31.92 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 2400 0 0 0 3191 1 0 0 24 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4644880 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=466180 CPUtime=39.76 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 3248 0 0 0 3974 2 0 0 25 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=466180 CPUtime=47.41 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 5487 0 0 0 4737 4 0 0 25 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=466180 CPUtime=48.76 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 2865 0 0 0 4875 1 0 0 25 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670431 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=466180 CPUtime=46.87 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 5094 0 0 0 4683 4 0 0 25 0 8 0 550777587 477368320 29521 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 356.68

################
# More data... #
################


[startup+4782.3 s]
/proc/loadavg: 9.03 9.03 9.02 10/293 19534
/proc/meminfo: memFree=4553776/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=848924 CPUtime=33508 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 123417 0 0 0 3350550 249 0 0 23 0 8 0 550777582 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 218371062254 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/19384/statm: 212231 117723 466 189 0 206580 0
[pid=19384/tid=19386] ppid=19382 vsize=848924 CPUtime=3904.6 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13415 0 0 0 390434 26 0 0 25 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=848924 CPUtime=4521.52 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 10878 0 0 0 452130 22 0 0 25 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670230 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=848924 CPUtime=3971.9 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 8727 0 0 0 397166 24 0 0 25 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 4653682 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=848924 CPUtime=4244.05 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 13909 0 0 0 424377 28 0 0 25 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=848924 CPUtime=4165.25 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18601 0 0 0 416492 33 0 0 25 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 218370616052 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=848924 CPUtime=4231.87 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13885 0 0 0 423159 28 0 0 22 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=848924 CPUtime=4148.79 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25349 0 0 0 414828 51 0 0 25 0 8 0 550777587 869298176 117723 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 33508
Current children cumulated vsize (KiB) 848924

[startup+4842.3 s]
/proc/loadavg: 9.01 9.02 9.01 10/293 19536
/proc/meminfo: memFree=4552168/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=849600 CPUtime=33927.4 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 124015 0 0 0 3392491 251 0 0 25 0 8 0 550777582 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670038 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/19384/statm: 212400 118269 466 189 0 206749 0
[pid=19384/tid=19386] ppid=19382 vsize=849600 CPUtime=3956.52 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13415 0 0 0 395625 27 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=849600 CPUtime=4581.5 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 11294 0 0 0 458127 23 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4660077 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=849600 CPUtime=4031.9 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 8766 0 0 0 403166 24 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=849600 CPUtime=4296.71 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 13979 0 0 0 429643 28 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=849600 CPUtime=4225.25 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18622 0 0 0 422492 33 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=849600 CPUtime=4271.8 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13885 0 0 0 427152 28 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=849600 CPUtime=4205.78 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25399 0 0 0 420527 51 0 0 25 0 8 0 550777587 869990400 118269 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 33927.4
Current children cumulated vsize (KiB) 849600

[startup+4902.3 s]
/proc/loadavg: 9.05 9.03 9.01 10/293 19537
/proc/meminfo: memFree=4551608/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=850172 CPUtime=34346.9 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 124162 0 0 0 3434435 252 0 0 25 0 8 0 550777582 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/19384/statm: 212543 118412 466 189 0 206892 0
[pid=19384/tid=19386] ppid=19382 vsize=850172 CPUtime=3993.13 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13420 0 0 0 399286 27 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=850172 CPUtime=4641.5 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 11355 0 0 0 464127 23 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=850172 CPUtime=4068.16 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 8779 0 0 0 406792 24 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 218370654850 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=850172 CPUtime=4344.63 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 14035 0 0 0 434434 29 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=850172 CPUtime=4285.25 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18629 0 0 0 428492 33 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 218370624514 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=850172 CPUtime=4331.79 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13885 0 0 0 433151 28 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=850172 CPUtime=4264.99 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25404 0 0 0 426447 52 0 0 25 0 8 0 550777587 870576128 118412 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 34346.9
Current children cumulated vsize (KiB) 850172

[startup+4962.3 s]
/proc/loadavg: 9.08 9.04 9.01 10/293 19539
/proc/meminfo: memFree=4550376/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=850360 CPUtime=34766.3 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 124549 0 0 0 3476376 254 0 0 25 0 8 0 550777582 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4320928 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/19384/statm: 212590 118795 466 189 0 206939 0
[pid=19384/tid=19386] ppid=19382 vsize=850360 CPUtime=4039.65 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13421 0 0 0 403938 27 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=850360 CPUtime=4698.9 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 11374 0 0 0 469866 24 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=850360 CPUtime=4102.97 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 9004 0 0 0 410272 25 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=850360 CPUtime=4395.28 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 14113 0 0 0 439499 29 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=850360 CPUtime=4345.16 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18643 0 0 0 434483 33 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=850360 CPUtime=4391.79 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13896 0 0 0 439151 28 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=850360 CPUtime=4323.19 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25438 0 0 0 432267 52 0 0 25 0 8 0 550777587 870768640 118795 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 34766.3
Current children cumulated vsize (KiB) 850360

[startup+5022.3 s]
/proc/loadavg: 9.08 9.04 9.01 10/293 19541
/proc/meminfo: memFree=4549384/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=850868 CPUtime=35185.7 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 124784 0 0 0 3518318 255 0 0 25 0 8 0 550777582 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669427 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/19384/statm: 212717 119004 466 189 0 207066 0
[pid=19384/tid=19386] ppid=19382 vsize=850868 CPUtime=4087.9 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13465 0 0 0 408763 27 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=850868 CPUtime=4738.39 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 11413 0 0 0 473815 24 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=850868 CPUtime=4158.52 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 9050 0 0 0 415827 25 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=850868 CPUtime=4452.32 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 14121 0 0 0 445203 29 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=850868 CPUtime=4405.16 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18725 0 0 0 440483 33 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=850868 CPUtime=4449.58 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13896 0 0 0 444930 28 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4660075 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=850868 CPUtime=4367.86 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25443 0 0 0 436734 52 0 0 25 0 8 0 550777587 871288832 119004 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35185.7
Current children cumulated vsize (KiB) 850868

[startup+5082.3 s]
/proc/loadavg: 9.03 9.03 9.00 10/293 19542
/proc/meminfo: memFree=4547836/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=852180 CPUtime=35605.2 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 125215 0 0 0 3560262 256 0 0 25 0 8 0 550777582 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/19384/statm: 213045 119434 466 189 0 207394 0
[pid=19384/tid=19386] ppid=19382 vsize=852180 CPUtime=4147.91 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13649 0 0 0 414763 28 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4320942 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=852180 CPUtime=4772.19 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 11418 0 0 0 477195 24 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=852180 CPUtime=4216.07 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 9058 0 0 0 421582 25 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=852180 CPUtime=4482.81 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 14296 0 0 0 448252 29 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=852180 CPUtime=4465.16 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18735 0 0 0 446483 33 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=852180 CPUtime=4509.59 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13922 0 0 0 450931 28 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=852180 CPUtime=4425.46 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25447 0 0 0 442494 52 0 0 25 0 8 0 550777587 872632320 119434 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35605.2
Current children cumulated vsize (KiB) 852180



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.02 9.03 9.00 10/293 19543
/proc/meminfo: memFree=4547656/32951124 swapFree=67051472/67111528
[pid=19384] ppid=19382 vsize=852180 CPUtime=35729 cores=0-7
/proc/19384/stat : 19384 (claspmt) R 19382 19384 18668 0 -1 4202496 125232 0 0 0 3572641 256 0 0 25 0 8 0 550777582 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670243 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/19384/statm: 213045 119451 466 189 0 207394 0
[pid=19384/tid=19386] ppid=19382 vsize=852180 CPUtime=4165.62 cores=0-7
/proc/19384/task/19386/stat : 19386 (claspmt) R 19382 19384 18668 0 -1 4202560 13649 0 0 0 416534 28 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19384/tid=19387] ppid=19382 vsize=852180 CPUtime=4786.2 cores=0-7
/proc/19384/task/19387/stat : 19387 (claspmt) R 19382 19384 18668 0 -1 4202560 11418 0 0 0 478596 24 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669477 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19384/tid=19388] ppid=19382 vsize=852180 CPUtime=4229.15 cores=0-7
/proc/19384/task/19388/stat : 19388 (claspmt) R 19382 19384 18668 0 -1 4202560 9058 0 0 0 422890 25 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19384/tid=19389] ppid=19382 vsize=852180 CPUtime=4495.61 cores=0-7
/proc/19384/task/19389/stat : 19389 (claspmt) R 19382 19384 18668 0 -1 4202560 14296 0 0 0 449532 29 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19384/tid=19390] ppid=19382 vsize=852180 CPUtime=4482.87 cores=0-7
/proc/19384/task/19390/stat : 19390 (claspmt) R 19382 19384 18668 0 -1 4202560 18744 0 0 0 448254 33 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669203 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19384/tid=19391] ppid=19382 vsize=852180 CPUtime=4527.3 cores=0-7
/proc/19384/task/19391/stat : 19391 (claspmt) R 19382 19384 18668 0 -1 4202560 13922 0 0 0 452702 28 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4669349 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19384/tid=19392] ppid=19382 vsize=852180 CPUtime=4438.52 cores=0-7
/proc/19384/task/19392/stat : 19392 (claspmt) R 19382 19384 18668 0 -1 4202560 25455 0 0 0 443800 52 0 0 25 0 8 0 550777587 872632320 119451 33554432000 4194304 4964704 140734491793856 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 35729
Current children cumulated vsize (KiB) 852180

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 5100.11
CPU time (s): 35729.1
CPU user time (s): 35726.5
CPU system time (s): 2.59061
CPU usage (%): 700.556
Max. virtual memory (cumulated for all children) (KiB): 852180

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35726.5
system time used= 2.59061
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 125243
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 147657
involuntary context switches= 230318

runsolver used 12.989 second user time and 33.3909 second system time

The end

Launcher Data 

Begin job on node101 at 2011-05-03 05:02:28
IDJOB=3322621
IDBENCH=84761
IDSOLVER=1711
FILE ID=node101/3322621-1304391748
RUNJOBID= node101-1304386646-18682
PBS_JOBID= 13181808
Free space on /tmp= 73484 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT04/handmade/marijn/equilarge-l/equilarge_l2.used-as.sat04-894.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322621-1304391748/watcher-3322621-1304391748 -o /tmp/evaluation-result-3322621-1304391748/solver-3322621-1304391748 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322621-1304391748.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= b3ddb2ee2bb2b83e1c98b2a9018351fc
RANDOM SEED=778375386

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       5070460 kB
Buffers:        265220 kB
Cached:        2398724 kB
SwapCached:      23472 kB
Active:       25277956 kB
Inactive:      2254184 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       5070460 kB
SwapTotal:    67111528 kB
SwapFree:     67051472 kB
Dirty:            1188 kB
Writeback:           0 kB
AnonPages:    24866968 kB
Mapped:          22756 kB
Slab:           222032 kB
PageTables:      62680 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 42439624 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73484 MiB
End job on node101 at 2011-05-03 06:27:30