Trace number 3322771

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 28451.4 5100.53

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
MD5SUM1bac7b96ea6eb843b1de7405a721cd87
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.089986
Satisfiable
(Un)Satisfiability was proved
Number of variables1244
Number of clauses150604
Sum of the clauses size302382
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2150569
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...394184/instance-3322771-1304394184.cnf
0.59/0.64	c Solving...
28450.85/5100.02	
28450.85/5100.02	*** INTERRUPTED! ***
28450.85/5100.02	*** Shutting down threads...
28451.37/5100.52	s UNKNOWN
28451.37/5100.52	
28451.37/5100.52	c INTERRUPTED   : 1
28451.37/5100.52	c Models        : 0+    
28451.37/5100.52	c Time          : 5100.020s (Solving: 5099.37s 1st Model: 0.00s Unsat: 0.00s)
28451.37/5100.52	c CPU Time      : 28450.890s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322771-1304394184/watcher-3322771-1304394184 -o /tmp/evaluation-result-3322771-1304394184/solver-3322771-1304394184 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322771-1304394184.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.50 10.97 11.03 5/409 26284
/proc/meminfo: memFree=18817412/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=23400 CPUtime=0.01 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 1295 0 0 0 1 0 0 0 25 0 1 0 629649182 23961600 1045 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988253381 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/26284/statm: 5850 1046 386 189 0 724 0

[startup+0.0226691 s]
/proc/loadavg: 10.50 10.97 11.03 5/409 26284
/proc/meminfo: memFree=18817412/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=24696 CPUtime=0.01 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 1762 0 0 0 1 0 0 0 25 0 1 0 629649182 25288704 1391 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988253381 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/26284/statm: 6174 1392 386 189 0 1048 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 24696

[startup+0.100656 s]
/proc/loadavg: 10.50 10.97 11.03 5/409 26284
/proc/meminfo: memFree=18817412/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=31188 CPUtime=0.09 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 3687 0 0 0 9 0 0 0 25 0 1 0 629649182 31936512 2853 33554432000 4194304 4964704 140736391750288 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/26284/statm: 7797 2853 399 189 0 2671 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31188

[startup+0.300621 s]
/proc/loadavg: 10.50 10.97 11.03 5/409 26284
/proc/meminfo: memFree=18817412/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=31188 CPUtime=0.29 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 3687 0 0 0 29 0 0 0 25 0 1 0 629649182 31936512 2853 33554432000 4194304 4964704 140736391750288 18446744073709551615 4607513 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/26284/statm: 7797 2853 399 189 0 2671 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 31188

[startup+0.700551 s]
/proc/loadavg: 10.50 10.97 11.03 5/409 26284
/proc/meminfo: memFree=18817412/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=193564 CPUtime=0.69 cores=0-7
/proc/26284/stat : 26284 (claspmt) S 26282 26284 26072 0 -1 4202496 4201 0 0 0 69 0 0 0 25 0 8 0 629649182 198209536 3069 33554432000 4194304 4964704 140736391750288 18446744073709551615 216000408473 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/26284/statm: 48391 3069 435 189 0 42740 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 193564

[startup+1.50042 s]
/proc/loadavg: 10.50 10.97 11.03 12/417 26292
/proc/meminfo: memFree=18795160/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=457724 CPUtime=4.35 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 11947 0 0 0 432 3 0 0 25 0 8 0 629649182 468709376 10815 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320966 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 114431 10816 464 189 0 108780 0
[pid=26284/tid=26286] ppid=26282 vsize=457724 CPUtime=0.32 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 853 0 0 0 32 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=457724 CPUtime=0.59 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 1001 0 0 0 59 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=457724 CPUtime=0.32 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 795 0 0 0 32 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=457724 CPUtime=0.46 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 1408 0 0 0 46 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 4668942 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=457724 CPUtime=0.72 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 559 0 0 0 72 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=457724 CPUtime=0.36 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 962 0 0 0 36 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=457724 CPUtime=0.3 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 1058 0 0 0 30 0 0 0 25 0 8 0 629649247 468709376 10816 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4.35
Current children cumulated vsize (KiB) 457724

[startup+3.10114 s]
/proc/loadavg: 10.50 10.97 11.03 12/417 26292
/proc/meminfo: memFree=18754612/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=529516 CPUtime=11.62 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 22800 0 0 0 1154 8 0 0 18 0 8 0 629649182 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/26284/statm: 132379 21668 464 189 0 126728 0
[pid=26284/tid=26286] ppid=26282 vsize=529516 CPUtime=1.01 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 1894 0 0 0 101 0 0 0 25 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=529516 CPUtime=1.81 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 1749 0 0 0 181 0 0 0 25 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=529516 CPUtime=0.97 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 1823 0 0 0 96 1 0 0 25 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 4654508 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=529516 CPUtime=1.41 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 3246 0 0 0 140 1 0 0 25 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=529516 CPUtime=1.55 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 2617 0 0 0 155 0 0 0 25 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 4343224 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=529516 CPUtime=1.28 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 1985 0 0 0 128 0 0 0 18 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=529516 CPUtime=1.34 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 2717 0 0 0 133 1 0 0 25 0 8 0 629649247 542224384 21668 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.62
Current children cumulated vsize (KiB) 529516

[startup+6.30058 s]
/proc/loadavg: 10.54 10.97 11.03 13/417 26292
/proc/meminfo: memFree=18685360/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=533468 CPUtime=27.29 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 37931 0 0 0 2713 16 0 0 21 0 8 0 629649182 546271232 36799 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/26284/statm: 133367 36800 464 189 0 127716 0
[pid=26284/tid=26286] ppid=26282 vsize=533468 CPUtime=1.99 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 3011 0 0 0 198 1 0 0 25 0 8 0 629649247 546271232 36801 33554432000 4194304 4964704 140736391750288 18446744073709551615 4332519 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=533468 CPUtime=4.37 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 2692 0 0 0 436 1 0 0 25 0 8 0 629649247 546271232 36801 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988253729 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=533468 CPUtime=2.75 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 3196 0 0 0 274 1 0 0 25 0 8 0 629649247 546271232 36801 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=533468 CPUtime=2.56 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 4903 0 0 0 254 2 0 0 25 0 8 0 629649247 546271232 36801 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=533468 CPUtime=3.58 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 5002 0 0 0 356 2 0 0 25 0 8 0 629649247 546271232 36802 33554432000 4194304 4964704 140736391750288 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=533468 CPUtime=4.15 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 4299 0 0 0 414 1 0 0 20 0 8 0 629649247 546271232 36802 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320872 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=533468 CPUtime=3.14 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 5481 0 0 0 311 3 0 0 25 0 8 0 629649247 546271232 36802 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 27.29
Current children cumulated vsize (KiB) 533468

[startup+12.7005 s]
/proc/loadavg: 10.57 10.97 11.03 12/417 26292
/proc/meminfo: memFree=18601808/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=540832 CPUtime=58.01 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 55306 0 0 0 5777 24 0 0 25 0 8 0 629649182 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 135208 54174 466 189 0 129557 0
[pid=26284/tid=26286] ppid=26282 vsize=540832 CPUtime=4.94 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 4726 0 0 0 493 1 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670304 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=540832 CPUtime=9 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 3916 0 0 0 899 1 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=540832 CPUtime=6.97 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 4841 0 0 0 695 2 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=540832 CPUtime=6.77 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 7682 0 0 0 674 3 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=540832 CPUtime=8.23 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 7314 0 0 0 821 2 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4359840 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=540832 CPUtime=7.85 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 5655 0 0 0 783 2 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=540832 CPUtime=5.94 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 8561 0 0 0 590 4 0 0 25 0 8 0 629649247 553811968 54174 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 58.01
Current children cumulated vsize (KiB) 540832

[startup+25.5012 s]
/proc/loadavg: 10.59 10.95 11.03 12/417 26293
/proc/meminfo: memFree=18533420/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=550816 CPUtime=120.17 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 73066 0 0 0 11981 36 0 0 25 0 8 0 629649182 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/26284/statm: 137704 71934 466 189 0 132053 0
[pid=26284/tid=26286] ppid=26282 vsize=550816 CPUtime=11.31 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 6672 0 0 0 1128 3 0 0 24 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=550816 CPUtime=19.11 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 4871 0 0 0 1910 1 0 0 25 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988697582 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=550816 CPUtime=17.2 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 6176 0 0 0 1717 3 0 0 21 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4627290 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=550816 CPUtime=15.16 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 11896 0 0 0 1510 6 0 0 25 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=550816 CPUtime=16.25 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 9987 0 0 0 1621 4 0 0 25 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=550816 CPUtime=14.95 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 6926 0 0 0 1492 3 0 0 25 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4660144 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=550816 CPUtime=12.34 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 10217 0 0 0 1229 5 0 0 25 0 8 0 629649247 564035584 71934 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 120.17
Current children cumulated vsize (KiB) 550816

[startup+51.1007 s]
/proc/loadavg: 10.73 10.95 11.02 12/417 26293
/proc/meminfo: memFree=18468052/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=554496 CPUtime=254.76 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 88913 0 0 0 25429 47 0 0 25 0 8 0 629649182 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/26284/statm: 138624 87777 466 189 0 132973 0
[pid=26284/tid=26286] ppid=26282 vsize=554496 CPUtime=25.32 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 7549 0 0 0 2529 3 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4627309 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=554496 CPUtime=41.85 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 5887 0 0 0 4184 1 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=554496 CPUtime=31.09 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 8416 0 0 0 3104 5 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=554496 CPUtime=34.89 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 13344 0 0 0 3481 8 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670458 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=554496 CPUtime=32.49 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 11920 0 0 0 3243 6 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=554496 CPUtime=32.58 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 9684 0 0 0 3252 6 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=554496 CPUtime=25.08 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 11878 0 0 0 2502 6 0 0 25 0 8 0 629649247 567803904 87777 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670281 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 254.76

################
# More data... #
################

[pid=26284] ppid=26282 vsize=2981036 CPUtime=26912.5 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 695221 0 0 0 2690610 636 0 0 25 0 8 0 629649182 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 745259 684535 466 189 0 739608 0
[pid=26284/tid=26286] ppid=26282 vsize=2981036 CPUtime=3475.67 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 64309 0 0 0 347517 50 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=2981036 CPUtime=3890.25 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 63642 0 0 0 389005 20 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=2981036 CPUtime=3306.59 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 89200 0 0 0 330556 103 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=2981036 CPUtime=3424.65 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 52285 0 0 0 342423 42 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670318 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=2981036 CPUtime=3640.96 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 61726 0 0 0 364047 49 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=2981036 CPUtime=3227.38 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 94769 0 0 0 322621 117 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=2981036 CPUtime=2972.34 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 148915 0 0 0 297091 143 0 0 25 0 8 0 629649247 3052580864 684535 33554432000 4194304 4964704 140736391750288 18446744073709551615 4628021 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 26912.5
Current children cumulated vsize (KiB) 2981036

[startup+4902.3 s]
/proc/loadavg: 11.07 11.06 11.00 12/417 26444
/proc/meminfo: memFree=16073100/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2981036 CPUtime=27287.9 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 695434 0 0 0 2728157 637 0 0 25 0 8 0 629649182 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 745259 684748 466 189 0 739608 0
[pid=26284/tid=26286] ppid=26282 vsize=2981036 CPUtime=3516.78 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 64309 0 0 0 351628 50 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=2981036 CPUtime=3950.25 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 63642 0 0 0 395005 20 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=2981036 CPUtime=3341.39 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 89394 0 0 0 334035 104 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=2981036 CPUtime=3472.15 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 52285 0 0 0 347173 42 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=2981036 CPUtime=3683.18 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 61732 0 0 0 368268 50 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=2981036 CPUtime=3281.37 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 94778 0 0 0 328020 117 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=2981036 CPUtime=3025.61 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 148915 0 0 0 302418 143 0 0 25 0 8 0 629649247 3052580864 684748 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 27287.9
Current children cumulated vsize (KiB) 2981036

[startup+4962.3 s]
/proc/loadavg: 11.07 11.06 11.00 12/417 26445
/proc/meminfo: memFree=16072928/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2981036 CPUtime=27636.2 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 695602 0 0 0 2762984 637 0 0 25 0 8 0 629649182 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320921 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 745259 684916 466 189 0 739608 0
[pid=26284/tid=26286] ppid=26282 vsize=2981036 CPUtime=3546.52 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 64309 0 0 0 354602 50 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=2981036 CPUtime=4010.25 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 63642 0 0 0 401005 20 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=2981036 CPUtime=3371.36 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 89394 0 0 0 337032 104 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=2981036 CPUtime=3527.55 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 52285 0 0 0 352713 42 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=2981036 CPUtime=3723.76 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 61845 0 0 0 372326 50 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4325338 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=2981036 CPUtime=3311.89 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 94780 0 0 0 331072 117 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=2981036 CPUtime=3074.47 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 148917 0 0 0 307304 143 0 0 25 0 8 0 629649247 3052580864 684916 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 27636.2
Current children cumulated vsize (KiB) 2981036

[startup+5022.3 s]
/proc/loadavg: 11.06 11.06 11.00 12/417 26447
/proc/meminfo: memFree=16072256/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2981036 CPUtime=27986.7 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 695880 0 0 0 2798034 637 0 0 25 0 8 0 629649182 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/26284/statm: 745259 685194 466 189 0 739608 0
[pid=26284/tid=26286] ppid=26282 vsize=2981036 CPUtime=3596.82 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 64310 0 0 0 359632 50 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=2981036 CPUtime=4070.25 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 63642 0 0 0 407005 20 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=2981036 CPUtime=3401.37 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 89403 0 0 0 340033 104 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=2981036 CPUtime=3587.53 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 52534 0 0 0 358711 42 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=2981036 CPUtime=3762.15 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 61856 0 0 0 376165 50 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=2981036 CPUtime=3342.88 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 94780 0 0 0 334171 117 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=2981036 CPUtime=3104.48 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 148922 0 0 0 310305 143 0 0 25 0 8 0 629649247 3052580864 685194 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 27986.7
Current children cumulated vsize (KiB) 2981036

[startup+5082.3 s]
/proc/loadavg: 11.10 11.08 11.01 12/417 26448
/proc/meminfo: memFree=16071096/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2981036 CPUtime=28344.8 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 696159 0 0 0 2833838 637 0 0 25 0 8 0 629649182 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320955 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 745259 685473 466 189 0 739608 0
[pid=26284/tid=26286] ppid=26282 vsize=2981036 CPUtime=3656.83 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 64315 0 0 0 365633 50 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=2981036 CPUtime=4130.26 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 63642 0 0 0 413006 20 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=2981036 CPUtime=3430.28 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 89406 0 0 0 342924 104 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=2981036 CPUtime=3647.53 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 52792 0 0 0 364710 43 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=2981036 CPUtime=3792.12 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 61867 0 0 0 379162 50 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4321465 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=2981036 CPUtime=3383.58 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 94782 0 0 0 338241 117 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=2981036 CPUtime=3141.76 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 148922 0 0 0 314033 143 0 0 25 0 8 0 629649247 3052580864 685473 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 28344.8
Current children cumulated vsize (KiB) 2981036



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 11.20 11.10 11.02 12/417 26449
/proc/meminfo: memFree=16060032/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2981036 CPUtime=28450.8 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 698883 0 0 0 2844444 641 0 0 25 0 8 0 629649182 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4321038 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/26284/statm: 745259 688197 466 189 0 739608 0
[pid=26284/tid=26286] ppid=26282 vsize=2981036 CPUtime=3674.54 cores=0-7
/proc/26284/task/26286/stat : 26286 (claspmt) R 26282 26284 26072 0 -1 4202560 64315 0 0 0 367404 50 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=26284/tid=26287] ppid=26282 vsize=2981036 CPUtime=4147.97 cores=0-7
/proc/26284/task/26287/stat : 26287 (claspmt) R 26282 26284 26072 0 -1 4202560 63642 0 0 0 414777 20 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=26284/tid=26288] ppid=26282 vsize=2981036 CPUtime=3443.69 cores=0-7
/proc/26284/task/26288/stat : 26288 (claspmt) R 26282 26284 26072 0 -1 4202560 89413 0 0 0 344265 104 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=26284/tid=26289] ppid=26282 vsize=2981036 CPUtime=3665.24 cores=0-7
/proc/26284/task/26289/stat : 26289 (claspmt) R 26282 26284 26072 0 -1 4202560 53111 0 0 0 366481 43 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=26284/tid=26290] ppid=26282 vsize=2981036 CPUtime=3800.98 cores=0-7
/proc/26284/task/26290/stat : 26290 (claspmt) R 26282 26284 26072 0 -1 4202560 61890 0 0 0 380048 50 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=26284/tid=26291] ppid=26282 vsize=2981036 CPUtime=3396.54 cores=0-7
/proc/26284/task/26291/stat : 26291 (claspmt) R 26282 26284 26072 0 -1 4202560 94782 0 0 0 339537 117 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=26284/tid=26292] ppid=26282 vsize=2981036 CPUtime=3150.67 cores=0-7
/proc/26284/task/26292/stat : 26292 (claspmt) R 26282 26284 26072 0 -1 4202560 151297 0 0 0 314920 147 0 0 25 0 8 0 629649247 3052580864 688197 33554432000 4194304 4964704 140736391750288 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 28450.8
Current children cumulated vsize (KiB) 2981036

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.2 s]
/proc/loadavg: 11.20 11.10 11.02 12/417 26449
/proc/meminfo: memFree=16060032/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2975408 CPUtime=28451.1 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 698893 0 0 0 2844466 641 0 0 25 0 1 0 629649182 3046817792 684444 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988247905 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/26284/statm: 743852 684444 476 189 0 738201 0
Current children cumulated CPU time (s) 28451.1
Current children cumulated vsize (KiB) 2975408

[startup+5100.4 s]
/proc/loadavg: 11.20 11.10 11.02 12/417 26449
/proc/meminfo: memFree=16060032/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=2964464 CPUtime=28451.3 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 698893 0 0 0 2844486 641 0 0 25 0 1 0 629649182 3035611136 677879 33554432000 4194304 4964704 140736391750288 18446744073709551615 4646575 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/26284/statm: 741116 677879 476 189 0 735465 0
Current children cumulated CPU time (s) 28451.3
Current children cumulated vsize (KiB) 2964464

[startup+5100.5 s]
/proc/loadavg: 11.20 11.10 11.02 12/417 26449
/proc/meminfo: memFree=16060032/32950940 swapFree=67111364/67111528
[pid=26284] ppid=26282 vsize=985248 CPUtime=28451.4 cores=0-7
/proc/26284/stat : 26284 (claspmt) R 26282 26284 26072 0 -1 4202496 698893 0 0 0 2844487 650 0 0 25 0 1 0 629649182 1008893952 146866 33554432000 4194304 4964704 140736391750288 18446744073709551615 215988635783 0 0 4096 18946 0 0 0 17 5 0 0 0
/proc/26284/statm: 246312 146849 476 189 0 240661 0
Current children cumulated CPU time (s) 28451.4
Current children cumulated vsize (KiB) 985248

Child status: 0
Real time (s): 5100.53
CPU time (s): 28451.4
CPU user time (s): 28444.9
CPU system time (s): 6.53101
CPU usage (%): 557.813
Max. virtual memory (cumulated for all children) (KiB): 2981036

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 28444.9
system time used= 6.53101
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 698894
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 71587
involuntary context switches= 257673

runsolver used 19.3771 second user time and 50.1994 second system time

The end

Launcher Data 

Begin job on node117 at 2011-05-03 05:43:04
IDJOB=3322771
IDBENCH=88212
IDSOLVER=1711
FILE ID=node117/3322771-1304394184
RUNJOBID= node117-1304389082-26086
PBS_JOBID= 13181825
Free space on /tmp= 72628 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322771-1304394184/watcher-3322771-1304394184 -o /tmp/evaluation-result-3322771-1304394184/solver-3322771-1304394184 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322771-1304394184.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 1bac7b96ea6eb843b1de7405a721cd87
RANDOM SEED=236311863

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      18818188 kB
Buffers:       1036556 kB
Cached:        4623204 kB
SwapCached:        160 kB
Active:        8470940 kB
Inactive:      5134772 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      18818188 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:            2780 kB
Writeback:           0 kB
AnonPages:     7945648 kB
Mapped:          32056 kB
Slab:           416548 kB
PageTables:      44700 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS: 101188632 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72628 MiB
End job on node117 at 2011-05-03 07:08:06