Trace number 3322778

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 35776.9 5101.78

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1504.5-n60-p30-s88600538.cnf
MD5SUM76c30bc0971bc865c2b988ddaca89385
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.464928
Satisfiable
(Un)Satisfiability was proved
Number of variables3651
Number of clauses926696
Sum of the clauses size1856923
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2926636
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...395116/instance-3322778-1304395116.cnf
8.99/9.06	c Solving...
35775.02/5100.01	
35775.02/5100.01	*** INTERRUPTED! ***
35775.02/5100.01	*** Shutting down threads...
35776.77/5101.77	s UNKNOWN
35776.77/5101.77	
35776.77/5101.77	c INTERRUPTED   : 1
35776.77/5101.77	c Models        : 0+    
35776.77/5101.77	c Time          : 5100.691s (Solving: 5091.63s 1st Model: 0.00s Unsat: 0.00s)
35776.77/5101.77	c CPU Time      : 35775.760s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322778-1304395116/watcher-3322778-1304395116 -o /tmp/evaluation-result-3322778-1304395116/solver-3322778-1304395116 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322778-1304395116.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.00 8.81 7.84 3/349 29298
/proc/meminfo: memFree=25168872/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=23200 CPUtime=0.01 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1191 0 0 0 1 0 0 0 20 0 1 0 531927076 23756800 1003 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095111877 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29298/statm: 5800 1005 386 189 0 674 0

[startup+0.113475 s]
/proc/loadavg: 9.00 8.81 7.84 3/349 29298
/proc/meminfo: memFree=25168872/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=42736 CPUtime=0.1 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 8303 0 0 0 10 0 0 0 21 0 1 0 531927076 43761664 5535 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095111877 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 10684 5536 386 189 0 5558 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 42736

[startup+0.200433 s]
/proc/loadavg: 9.00 8.81 7.84 3/349 29298
/proc/meminfo: memFree=25168872/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=58272 CPUtime=0.19 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 13644 0 0 0 18 1 0 0 21 0 1 0 531927076 59670528 9509 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095111877 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 14568 9511 386 189 0 9442 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 58272

[startup+0.300406 s]
/proc/loadavg: 9.00 8.81 7.84 3/349 29298
/proc/meminfo: memFree=25168872/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=77572 CPUtime=0.29 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 18442 0 0 0 28 1 0 0 21 0 1 0 531927076 79433728 14271 33554432000 4194304 4964704 140737466567120 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 19393 14271 399 189 0 14267 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 77572

[startup+0.70033 s]
/proc/loadavg: 9.00 8.81 7.84 3/349 29298
/proc/meminfo: memFree=25168872/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=77572 CPUtime=0.69 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 18442 0 0 0 68 1 0 0 22 0 1 0 531927076 79433728 14271 33554432000 4194304 4964704 140737466567120 18446744073709551615 4607549 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 19393 14271 399 189 0 14267 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 77572

[startup+1.50018 s]
/proc/loadavg: 8.44 8.70 7.81 3/350 29299
/proc/meminfo: memFree=25112940/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=77572 CPUtime=1.49 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 18442 0 0 0 148 1 0 0 25 0 1 0 531927076 79433728 14271 33554432000 4194304 4964704 140737466567120 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 19393 14271 399 189 0 14267 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 77572

[startup+3.10088 s]
/proc/loadavg: 8.44 8.70 7.81 3/350 29299
/proc/meminfo: memFree=25112616/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=77572 CPUtime=3.09 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 18442 0 0 0 308 1 0 0 25 0 1 0 531927076 79433728 14271 33554432000 4194304 4964704 140737466567120 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 19393 14271 399 189 0 14267 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 77572

[startup+6.3003 s]
/proc/loadavg: 7.92 8.58 7.78 3/350 29300
/proc/meminfo: memFree=25111996/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=79820 CPUtime=6.29 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 19004 0 0 0 628 1 0 0 25 0 1 0 531927076 81735680 14833 33554432000 4194304 4964704 140737466567120 18446744073709551615 4600312 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/29298/statm: 19955 14833 400 189 0 14829 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 79820

[startup+12.7011 s]
/proc/loadavg: 8.09 8.61 7.79 10/357 29307
/proc/meminfo: memFree=24998620/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=501724 CPUtime=31.35 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 54956 0 0 0 3116 19 0 0 20 0 8 0 531927076 513765376 48917 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/29298/statm: 125431 48917 464 189 0 119780 0
[pid=29298/tid=29301] ppid=29296 vsize=501724 CPUtime=0.39 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 831 0 0 0 39 0 0 0 21 0 8 0 531927983 513765376 48917 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=501724 CPUtime=3.1 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 4088 0 0 0 307 3 0 0 25 0 8 0 531927983 513765376 48917 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=501724 CPUtime=3.12 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 6082 0 0 0 310 2 0 0 18 0 8 0 531927983 513765376 48919 33554432000 4194304 4964704 140737466567120 18446744073709551615 4653275 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=501724 CPUtime=3.09 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 5544 0 0 0 306 3 0 0 20 0 8 0 531927983 513765376 48919 33554432000 4194304 4964704 140737466567120 18446744073709551615 4668946 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=501724 CPUtime=3.34 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 3188 0 0 0 333 1 0 0 25 0 8 0 531927983 513765376 48919 33554432000 4194304 4964704 140737466567120 18446744073709551615 4342945 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=501724 CPUtime=2.99 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 4861 0 0 0 298 1 0 0 19 0 8 0 531927983 513765376 48919 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=501724 CPUtime=3.1 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 6383 0 0 0 307 3 0 0 18 0 8 0 531927983 513765376 48920 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320976 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 31.35
Current children cumulated vsize (KiB) 501724

[startup+25.5007 s]
/proc/loadavg: 8.29 8.63 7.81 10/357 29307
/proc/meminfo: memFree=24644304/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=868276 CPUtime=120.48 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 138976 0 0 0 11985 63 0 0 25 0 8 0 531927076 889114624 132937 33554432000 4194304 4964704 140737466567120 18446744073709551615 4343472 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/29298/statm: 217069 132937 466 189 0 211418 0
[pid=29298/tid=29301] ppid=29296 vsize=868276 CPUtime=10.83 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 8954 0 0 0 1080 3 0 0 18 0 8 0 531927983 889114624 132937 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=868276 CPUtime=12.22 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 7458 0 0 0 1217 5 0 0 19 0 8 0 531927983 889114624 132937 33554432000 4194304 4964704 140737466567120 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=868276 CPUtime=14.55 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 16674 0 0 0 1447 8 0 0 25 0 8 0 531927983 889114624 132937 33554432000 4194304 4964704 140737466567120 18446744073709551615 4653198 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=868276 CPUtime=15.38 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 14652 0 0 0 1529 9 0 0 25 0 8 0 531927983 889114624 132938 33554432000 4194304 4964704 140737466567120 18446744073709551615 4627356 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=868276 CPUtime=13.08 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 16306 0 0 0 1301 7 0 0 25 0 8 0 531927983 889114624 132938 33554432000 4194304 4964704 140737466567120 18446744073709551615 4334346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=868276 CPUtime=15.19 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 15263 0 0 0 1512 7 0 0 19 0 8 0 531927983 889114624 132938 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=868276 CPUtime=14.5 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 21698 0 0 0 1439 11 0 0 18 0 8 0 531927983 889114624 132938 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 120.48
Current children cumulated vsize (KiB) 868276

[startup+51.101 s]
/proc/loadavg: 8.68 8.69 7.85 10/357 29308
/proc/meminfo: memFree=24210940/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=1180092 CPUtime=298.48 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 247397 0 0 0 29716 132 0 0 24 0 8 0 531927076 1208414208 241358 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29298/statm: 295023 241358 466 189 0 289372 0
[pid=29298/tid=29301] ppid=29296 vsize=1180092 CPUtime=28.71 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 22874 0 0 0 2859 12 0 0 25 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=1180092 CPUtime=35.84 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 10861 0 0 0 3577 7 0 0 25 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=1180092 CPUtime=39.17 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 28330 0 0 0 3901 16 0 0 25 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=1180092 CPUtime=40.03 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 27084 0 0 0 3989 14 0 0 25 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=1180092 CPUtime=36.33 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 36071 0 0 0 3611 22 0 0 21 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=1180092 CPUtime=38.73 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 24661 0 0 0 3859 14 0 0 23 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=1180092 CPUtime=38.02 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 42599 0 0 0 3778 24 0 0 25 0 8 0 531927983 1208414208 241359 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 298.48
Current children cumulated vsize (KiB) 1180092

[startup+102.301 s]
/proc/loadavg: 8.91 8.75 7.92 10/357 29309
/proc/meminfo: memFree=23716416/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=1741868 CPUtime=655.23 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 369560 0 0 0 65301 222 0 0 25 0 8 0 531927076 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29298/statm: 435467 363507 466 189 0 429816 0
[pid=29298/tid=29301] ppid=29296 vsize=1741868 CPUtime=70.57 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 40604 0 0 0 7032 25 0 0 20 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=1741868 CPUtime=81.73 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 15290 0 0 0 8164 9 0 0 25 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=1741868 CPUtime=87.88 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 39631 0 0 0 8765 23 0 0 25 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=1741868 CPUtime=86.41 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 39723 0 0 0 8618 23 0 0 25 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=1741868 CPUtime=82.05 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 51712 0 0 0 8172 33 0 0 25 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=1741868 CPUtime=86.36 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 37842 0 0 0 8610 26 0 0 25 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=1741868 CPUtime=78.23 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 66806 0 0 0 7781 42 0 0 25 0 8 0 531927983 1783672832 363507 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670574 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 655.23
Current children cumulated vsize (KiB) 1741868

[startup+162.301 s]
/proc/loadavg: 8.97 8.79 7.98 10/357 29320
/proc/meminfo: memFree=23198048/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=2185036 CPUtime=1072.36 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 499784 0 0 0 106917 319 0 0 25 0 8 0 531927076 2237476864 493675 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29298/statm: 546259 493675 466 189 0 540608 0
[pid=29298/tid=29301] ppid=29296 vsize=2185036 CPUtime=115.89 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 68658 0 0 0 11543 46 0 0 25 0 8 0 531927983 2237476864 493675 33554432000 4194304 4964704 140737466567120 18446744073709551615 4462582 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=2185036 CPUtime=140.61 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 18163 0 0 0 14051 10 0 0 25 0 8 0 531927983 2237476864 493675 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=2185036 CPUtime=138.69 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 60297 0 0 0 13828 41 0 0 25 0 8 0 531927983 2237476864 493675 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=2185036 CPUtime=135.73 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 49695 0 0 0 13541 32 0 0 25 0 8 0 531927983 2237476864 493675 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=2185036 CPUtime=134.76 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 72031 0 0 0 13429 47 0 0 25 0 8 0 531927983 2237476864 493675 33554432000 4194304 4964704 140737466567120 18446744073709551615 4658359 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=2185036 CPUtime=137.2 cores=0-7

################
# More data... #
################

[pid=29298] ppid=29296 vsize=7557180 CPUtime=33970.3 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1832303 0 0 0 3395313 1716 0 0 25 0 8 0 531927076 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320981 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29298/statm: 1889295 1820757 466 189 0 1883644 0
[pid=29298/tid=29301] ppid=29296 vsize=7557180 CPUtime=4103.02 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 211090 0 0 0 410086 216 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=7557180 CPUtime=4489.49 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 53389 0 0 0 448921 28 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320963 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=7557180 CPUtime=4400.73 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 230314 0 0 0 439818 255 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=7557180 CPUtime=4510.95 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 146435 0 0 0 450960 135 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=7557180 CPUtime=4242.87 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 204099 0 0 0 424099 188 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=7557180 CPUtime=4372.31 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 170133 0 0 0 437052 179 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=7557180 CPUtime=3955.43 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 426296 0 0 0 395155 388 0 0 25 0 8 0 531927983 7738552320 1820757 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 33970.3
Current children cumulated vsize (KiB) 7557180

[startup+4902.3 s]
/proc/loadavg: 9.22 9.11 9.02 10/357 29458
/proc/meminfo: memFree=17821480/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7622716 CPUtime=34389.5 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1845760 0 0 0 3437216 1735 0 0 25 0 8 0 531927076 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4332514 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29298/statm: 1905679 1834214 466 189 0 1900028 0
[pid=29298/tid=29301] ppid=29296 vsize=7622716 CPUtime=4136.32 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 211175 0 0 0 413416 216 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095148624 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=7622716 CPUtime=4540.38 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 53409 0 0 0 454010 28 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=7622716 CPUtime=4460.72 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 230493 0 0 0 445817 255 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=7622716 CPUtime=4570.94 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 146477 0 0 0 456959 135 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=7622716 CPUtime=4302.83 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 204217 0 0 0 430095 188 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=7622716 CPUtime=4432.29 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 178750 0 0 0 443038 191 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=7622716 CPUtime=4015.43 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 430692 0 0 0 401149 394 0 0 25 0 8 0 531927983 7805661184 1834214 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 34389.5
Current children cumulated vsize (KiB) 7622716

[startup+4962.3 s]
/proc/loadavg: 9.12 9.10 9.02 10/357 29460
/proc/meminfo: memFree=17803348/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7622716 CPUtime=34812.6 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1849830 0 0 0 3479518 1741 0 0 25 0 8 0 531927076 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/29298/statm: 1905679 1838284 466 189 0 1900028 0
[pid=29298/tid=29301] ppid=29296 vsize=7622716 CPUtime=4151.78 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 211175 0 0 0 414961 217 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4627281 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=7622716 CPUtime=4600.38 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 53493 0 0 0 460010 28 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=7622716 CPUtime=4518.25 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 230553 0 0 0 451570 255 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=7622716 CPUtime=4630.95 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 146704 0 0 0 462959 136 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=7622716 CPUtime=4362.82 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 205095 0 0 0 436093 189 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=7622716 CPUtime=4492.3 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 178794 0 0 0 449039 191 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=7622716 CPUtime=4075.42 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 431954 0 0 0 407147 395 0 0 25 0 8 0 531927983 7805661184 1838284 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 34812.6
Current children cumulated vsize (KiB) 7622716

[startup+5022.3 s]
/proc/loadavg: 9.12 9.11 9.02 10/357 29461
/proc/meminfo: memFree=17802960/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7622716 CPUtime=35231.9 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1849927 0 0 0 3521451 1741 0 0 25 0 8 0 531927076 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4342157 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/29298/statm: 1905679 1838381 466 189 0 1900028 0
[pid=29298/tid=29301] ppid=29296 vsize=7622716 CPUtime=4182.79 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 211175 0 0 0 418062 217 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=7622716 CPUtime=4660.39 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 53499 0 0 0 466011 28 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=7622716 CPUtime=4577.58 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 230553 0 0 0 457503 255 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=7622716 CPUtime=4690.95 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 146704 0 0 0 468959 136 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=7622716 CPUtime=4422.81 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 205122 0 0 0 442092 189 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=7622716 CPUtime=4534.7 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 178858 0 0 0 453279 191 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=7622716 CPUtime=4130.01 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 431954 0 0 0 412606 395 0 0 25 0 8 0 531927983 7805661184 1838381 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 35231.9
Current children cumulated vsize (KiB) 7622716

[startup+5082.3 s]
/proc/loadavg: 9.08 9.10 9.02 10/357 29463
/proc/meminfo: memFree=17785824/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7688252 CPUtime=35651.2 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1854190 0 0 0 3563377 1748 0 0 25 0 8 0 531927076 7872770048 1842644 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/29298/statm: 1922063 1842644 466 189 0 1916412 0
[pid=29298/tid=29301] ppid=29296 vsize=7688252 CPUtime=4218.09 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 211181 0 0 0 421592 217 0 0 25 0 8 0 531927983 7872770048 1842644 33554432000 4194304 4964704 140737466567120 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=7688252 CPUtime=4720.39 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 53505 0 0 0 472011 28 0 0 25 0 8 0 531927983 7872770048 1842644 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=7688252 CPUtime=4636.92 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 230554 0 0 0 463437 255 0 0 25 0 8 0 531927983 7872770048 1842645 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=7688252 CPUtime=4750.95 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 149835 0 0 0 474954 141 0 0 25 0 8 0 531927983 7872770048 1842645 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=7688252 CPUtime=4479.3 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 206192 0 0 0 447740 190 0 0 25 0 8 0 531927983 7872770048 1842645 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=7688252 CPUtime=4592.13 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 178908 0 0 0 459021 192 0 0 25 0 8 0 531927983 7872770048 1842645 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=7688252 CPUtime=4160.79 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 431954 0 0 0 415684 395 0 0 25 0 8 0 531927983 7872770048 1842645 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 35651.2
Current children cumulated vsize (KiB) 7688252



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.13 9.11 9.03 10/357 29463
/proc/meminfo: memFree=17785316/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7688252 CPUtime=35775 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1854298 0 0 0 3575753 1749 0 0 25 0 8 0 531927076 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/29298/statm: 1922063 1842752 466 189 0 1916412 0
[pid=29298/tid=29301] ppid=29296 vsize=7688252 CPUtime=4235.81 cores=0-7
/proc/29298/task/29301/stat : 29301 (claspmt) R 29296 29298 28725 0 -1 4202560 211181 0 0 0 423364 217 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=29298/tid=29302] ppid=29296 vsize=7688252 CPUtime=4738.1 cores=0-7
/proc/29298/task/29302/stat : 29302 (claspmt) R 29296 29298 28725 0 -1 4202560 53506 0 0 0 473782 28 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29298/tid=29303] ppid=29296 vsize=7688252 CPUtime=4654.41 cores=0-7
/proc/29298/task/29303/stat : 29303 (claspmt) R 29296 29298 28725 0 -1 4202560 230574 0 0 0 465186 255 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320875 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29298/tid=29304] ppid=29296 vsize=7688252 CPUtime=4768.66 cores=0-7
/proc/29298/task/29304/stat : 29304 (claspmt) R 29296 29298 28725 0 -1 4202560 149838 0 0 0 476725 141 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29298/tid=29305] ppid=29296 vsize=7688252 CPUtime=4496.61 cores=0-7
/proc/29298/task/29305/stat : 29305 (claspmt) R 29296 29298 28725 0 -1 4202560 206275 0 0 0 449471 190 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29306] ppid=29296 vsize=7688252 CPUtime=4596.9 cores=0-7
/proc/29298/task/29306/stat : 29306 (claspmt) R 29296 29298 28725 0 -1 4202560 178908 0 0 0 459498 192 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4641858 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29298/tid=29307] ppid=29296 vsize=7688252 CPUtime=4174.13 cores=0-7
/proc/29298/task/29307/stat : 29307 (claspmt) R 29296 29298 28725 0 -1 4202560 431954 0 0 0 417018 395 0 0 25 0 8 0 531927983 7872770048 1842752 33554432000 4194304 4964704 140737466567120 18446744073709551615 4320963 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 35775
Current children cumulated vsize (KiB) 7688252

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.7 s]
/proc/loadavg: 9.13 9.11 9.03 4/351 29464
/proc/meminfo: memFree=17785364/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7687792 CPUtime=35775.8 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1854308 0 0 0 3575829 1749 0 0 19 0 1 0 531927076 7872299008 1842647 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095106251 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29298/statm: 1921948 1842647 476 189 0 1916297 0
Current children cumulated CPU time (s) 35775.8
Current children cumulated vsize (KiB) 7687792

[startup+5101.5 s]
/proc/loadavg: 9.13 9.11 9.03 4/351 29464
/proc/meminfo: memFree=17785364/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=7282272 CPUtime=35776.6 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1854308 0 0 0 3575906 1751 0 0 20 0 1 0 531927076 7457046528 1723370 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095494279 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29298/statm: 1820568 1722863 476 189 0 1814917 0
Current children cumulated CPU time (s) 35776.6
Current children cumulated vsize (KiB) 7282272

[startup+5101.7 s]
/proc/loadavg: 9.13 9.11 9.03 4/351 29464
/proc/meminfo: memFree=17785364/32951124 swapFree=67091720/67111528
[pid=29298] ppid=29296 vsize=2270816 CPUtime=35776.8 cores=0-7
/proc/29298/stat : 29298 (claspmt) R 29296 29298 28725 0 -1 4202496 1854308 0 0 0 3575906 1771 0 0 20 0 1 0 531927076 2325315584 483961 33554432000 4194304 4964704 140737466567120 18446744073709551615 228095494279 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29298/statm: 567704 483961 476 189 0 562053 0
Current children cumulated CPU time (s) 35776.8
Current children cumulated vsize (KiB) 2270816

Child status: 0
Real time (s): 5101.78
CPU time (s): 35776.9
CPU user time (s): 35759.1
CPU system time (s): 17.7933
CPU usage (%): 701.262
Max. virtual memory (cumulated for all children) (KiB): 7688252

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35759.1
system time used= 17.7933
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1854309
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 86359
involuntary context switches= 245467

runsolver used 16.4095 second user time and 42.2946 second system time

The end

Launcher Data 

Begin job on node142 at 2011-05-03 05:58:36
IDJOB=3322778
IDBENCH=88219
IDSOLVER=1711
FILE ID=node142/3322778-1304395116
RUNJOBID= node142-1304390297-28739
PBS_JOBID= 13181831
Free space on /tmp= 73464 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1504.5-n60-p30-s88600538.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322778-1304395116/watcher-3322778-1304395116 -o /tmp/evaluation-result-3322778-1304395116/solver-3322778-1304395116 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322778-1304395116.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 76c30bc0971bc865c2b988ddaca89385
RANDOM SEED=1099497613

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25169020 kB
Buffers:        365344 kB
Cached:        1693504 kB
SwapCached:       3844 kB
Active:        4489868 kB
Inactive:      2885824 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25169020 kB
SwapTotal:    67111528 kB
SwapFree:     67091720 kB
Dirty:           13108 kB
Writeback:           0 kB
AnonPages:     5314076 kB
Mapped:          28304 kB
Slab:           308364 kB
PageTables:      31600 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 77195688 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73464 MiB
End job on node142 at 2011-05-03 07:23:38