Trace number 3381376

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.9 5005.52

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
MD5SUM1bac7b96ea6eb843b1de7405a721cd87
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.089986
Satisfiable
(Un)Satisfiability was proved
Number of variables1244
Number of clauses150604
Sum of the clauses size302382
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2150569
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...667198/instance-3381376-1307667198.cnf
0.59/0.64	c Solving...
40000.08/5004.81	
40000.08/5004.81	*** INTERRUPTED! ***
40000.08/5004.81	*** Shutting down threads...
40000.84/5005.52	s UNKNOWN
40000.84/5005.52	
40000.84/5005.52	c INTERRUPTED   : 1
40000.84/5005.52	c Models        : 0+    
40000.84/5005.52	c Time          : 5004.828s (Solving: 5004.19s 1st Model: 0.00s Unsat: 0.00s)
40000.84/5005.52	c CPU Time      : 40000.170s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381376-1307667198/watcher-3381376-1307667198 -o /tmp/evaluation-result-3381376-1307667198/solver-3381376-1307667198 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381376-1307667198.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 6.53 3.82 2/170 1698
/proc/meminfo: memFree=32296080/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=20872 CPUtime=0 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 487 0 0 0 0 0 0 0 23 0 1 0 859135890 21372928 416 33554432000 4194304 4964800 140734158358352 18446744073709551615 255106466336 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/1698/statm: 5218 416 353 189 0 92 0

[startup+0.0680441 s]
/proc/loadavg: 8.11 6.53 3.82 2/170 1698
/proc/meminfo: memFree=32296080/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=31188 CPUtime=0.06 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 3688 0 0 0 6 0 0 0 23 0 1 0 859135890 31936512 2853 33554432000 4194304 4964800 140734158358352 18446744073709551615 4607562 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1698/statm: 7797 2853 399 189 0 2671 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 31188

[startup+0.101039 s]
/proc/loadavg: 8.11 6.53 3.82 2/170 1698
/proc/meminfo: memFree=32296080/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=31188 CPUtime=0.09 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 3688 0 0 0 9 0 0 0 23 0 1 0 859135890 31936512 2853 33554432000 4194304 4964800 140734158358352 18446744073709551615 4607486 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1698/statm: 7797 2853 399 189 0 2671 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31188

[startup+0.301002 s]
/proc/loadavg: 8.11 6.53 3.82 2/170 1698
/proc/meminfo: memFree=32296080/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=31188 CPUtime=0.29 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 3688 0 0 0 29 0 0 0 25 0 1 0 859135890 31936512 2853 33554432000 4194304 4964800 140734158358352 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/1698/statm: 7797 2853 399 189 0 2671 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 31188

[startup+0.700931 s]
/proc/loadavg: 8.11 6.53 3.82 2/170 1698
/proc/meminfo: memFree=32296080/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=259100 CPUtime=0.7 cores=0-7
/proc/1698/stat : 1698 (claspmt) S 1696 1698 1559 0 -1 4202496 4479 0 0 0 70 0 0 0 25 0 8 0 859135890 265318400 3329 33554432000 4194304 4964800 140734158358352 18446744073709551615 255118584729 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 64775 3329 435 189 0 59124 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 259100

[startup+1.50075 s]
/proc/loadavg: 8.11 6.53 3.82 10/178 1706
/proc/meminfo: memFree=32267768/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=394592 CPUtime=6.8 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 15637 0 0 0 674 6 0 0 18 0 8 0 859135890 404062208 14486 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627309 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 98648 14486 464 189 0 92997 0
[pid=1698/tid=1700] ppid=1696 vsize=394592 CPUtime=0.76 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 1563 0 0 0 76 0 0 0 18 0 8 0 859135955 404062208 14487 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=394592 CPUtime=0.75 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 1197 0 0 0 75 0 0 0 18 0 8 0 859135955 404062208 14487 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=394592 CPUtime=0.76 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 1415 0 0 0 75 1 0 0 19 0 8 0 859135955 404062208 14488 33554432000 4194304 4964800 140734158358352 18446744073709551615 4325305 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=394592 CPUtime=0.74 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 1702 0 0 0 73 1 0 0 18 0 8 0 859135955 404062208 14488 33554432000 4194304 4964800 140734158358352 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=394592 CPUtime=0.84 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 996 0 0 0 84 0 0 0 25 0 8 0 859135955 404062208 14488 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=394592 CPUtime=0.74 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 1645 0 0 0 73 1 0 0 18 0 8 0 859135955 404062208 14488 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=394592 CPUtime=0.76 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 1737 0 0 0 76 0 0 0 18 0 8 0 859135955 404062208 14488 33554432000 4194304 4964800 140734158358352 18446744073709551615 4321143 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 6.8
Current children cumulated vsize (KiB) 394592

[startup+3.10047 s]
/proc/loadavg: 8.11 6.53 3.82 9/178 1706
/proc/meminfo: memFree=32213964/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=405080 CPUtime=19.57 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 30628 0 0 0 1945 12 0 0 22 0 8 0 859135890 414801920 29477 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 101270 29477 464 189 0 95619 0
[pid=1698/tid=1700] ppid=1696 vsize=405080 CPUtime=2.36 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 3403 0 0 0 234 2 0 0 22 0 8 0 859135955 414801920 29477 33554432000 4194304 4964800 140734158358352 18446744073709551615 255106429985 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=405080 CPUtime=2.35 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 2235 0 0 0 235 0 0 0 20 0 8 0 859135955 414801920 29478 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=405080 CPUtime=2.35 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 3145 0 0 0 234 1 0 0 22 0 8 0 859135955 414801920 29478 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=405080 CPUtime=2.33 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 4154 0 0 0 231 2 0 0 22 0 8 0 859135955 414801920 29478 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627168 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=405080 CPUtime=2.43 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 3249 0 0 0 243 0 0 0 25 0 8 0 859135955 414801920 29478 33554432000 4194304 4964800 140734158358352 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=405080 CPUtime=2.34 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 2963 0 0 0 232 2 0 0 22 0 8 0 859135955 414801920 29478 33554432000 4194304 4964800 140734158358352 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=405080 CPUtime=2.36 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 3975 0 0 0 235 1 0 0 21 0 8 0 859135955 414801920 29478 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670330 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 19.57
Current children cumulated vsize (KiB) 405080

[startup+6.30091 s]
/proc/loadavg: 8.10 6.56 3.85 9/178 1706
/proc/meminfo: memFree=32120680/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=417356 CPUtime=45.15 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 49472 0 0 0 4494 21 0 0 25 0 8 0 859135890 427372544 48321 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320966 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 104339 48321 466 189 0 98688 0
[pid=1698/tid=1700] ppid=1696 vsize=417356 CPUtime=5.56 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 6072 0 0 0 553 3 0 0 20 0 8 0 859135955 427372544 48321 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=417356 CPUtime=5.55 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 3640 0 0 0 554 1 0 0 19 0 8 0 859135955 427372544 48322 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=417356 CPUtime=5.55 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 5170 0 0 0 553 2 0 0 25 0 8 0 859135955 427372544 48322 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=417356 CPUtime=5.51 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 6692 0 0 0 548 3 0 0 25 0 8 0 859135955 427372544 48322 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=417356 CPUtime=5.63 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 5287 0 0 0 562 1 0 0 25 0 8 0 859135955 427372544 48322 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=417356 CPUtime=5.54 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 4919 0 0 0 552 2 0 0 25 0 8 0 859135955 427372544 48323 33554432000 4194304 4964800 140734158358352 18446744073709551615 255106429985 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=417356 CPUtime=5.55 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 6917 0 0 0 553 2 0 0 19 0 8 0 859135955 427372544 48323 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 45.15
Current children cumulated vsize (KiB) 417356

[startup+12.7008 s]
/proc/loadavg: 8.09 6.58 3.87 9/178 1706
/proc/meminfo: memFree=32079116/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=419796 CPUtime=96.28 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 57536 0 0 0 9601 27 0 0 25 0 8 0 859135890 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4364704 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 104949 56385 466 189 0 99298 0
[pid=1698/tid=1700] ppid=1696 vsize=419796 CPUtime=11.94 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 7205 0 0 0 1191 3 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=419796 CPUtime=11.95 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 4823 0 0 0 1194 1 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=419796 CPUtime=11.95 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 5978 0 0 0 1192 3 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=419796 CPUtime=11.87 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 7385 0 0 0 1183 4 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=419796 CPUtime=12.03 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 6329 0 0 0 1201 2 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=419796 CPUtime=11.93 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 5373 0 0 0 1190 3 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=419796 CPUtime=11.95 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 8491 0 0 0 1192 3 0 0 25 0 8 0 859135955 429871104 56385 33554432000 4194304 4964800 140734158358352 18446744073709551615 4342186 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 96.28
Current children cumulated vsize (KiB) 419796

[startup+25.5005 s]
/proc/loadavg: 8.07 6.65 3.93 9/178 1707
/proc/meminfo: memFree=32004136/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=433068 CPUtime=198.59 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 75150 0 0 0 19821 38 0 0 25 0 8 0 859135890 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670323 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 108267 73999 466 189 0 102616 0
[pid=1698/tid=1700] ppid=1696 vsize=433068 CPUtime=24.74 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 9848 0 0 0 2468 6 0 0 19 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4362234 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=433068 CPUtime=24.76 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 6128 0 0 0 2474 2 0 0 25 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=433068 CPUtime=24.75 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 7772 0 0 0 2470 5 0 0 25 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=433068 CPUtime=24.6 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 9293 0 0 0 2454 6 0 0 25 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627424 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=433068 CPUtime=24.84 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 7982 0 0 0 2481 3 0 0 25 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=433068 CPUtime=24.73 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 7263 0 0 0 2469 4 0 0 25 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4653296 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=433068 CPUtime=24.75 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 11451 0 0 0 2470 5 0 0 25 0 8 0 859135955 443461632 73999 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 198.59
Current children cumulated vsize (KiB) 433068

[startup+51.101 s]
/proc/loadavg: 8.11 6.77 4.04 9/178 1707
/proc/meminfo: memFree=31934140/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=440996 CPUtime=403.21 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 92816 0 0 0 40269 52 0 0 25 0 8 0 859135890 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4321079 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 110249 91647 466 189 0 104598 0
[pid=1698/tid=1700] ppid=1696 vsize=440996 CPUtime=50.33 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 11458 0 0 0 5026 7 0 0 20 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=440996 CPUtime=50.35 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 7147 0 0 0 5033 2 0 0 25 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=440996 CPUtime=50.35 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 9548 0 0 0 5029 6 0 0 25 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=440996 CPUtime=50.05 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 11379 0 0 0 4998 7 0 0 25 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=440996 CPUtime=50.43 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 9823 0 0 0 5039 4 0 0 25 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=440996 CPUtime=50.33 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 9575 0 0 0 5026 7 0 0 25 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=440996 CPUtime=50.34 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 15880 0 0 0 5026 8 0 0 18 0 8 0 859135955 451579904 91647 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 403.21

################
# More data... #
################

[pid=1698] ppid=1696 vsize=2900972 CPUtime=37741.8 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 665228 0 0 0 3773486 694 0 0 25 0 8 0 859135890 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320981 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 725243 652581 466 189 0 719592 0
[pid=1698/tid=1700] ppid=1696 vsize=2900972 CPUtime=4717.75 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 50860 0 0 0 471714 61 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=2900972 CPUtime=4721.42 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 67408 0 0 0 472120 22 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=2900972 CPUtime=4720.99 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 52552 0 0 0 472035 64 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=2900972 CPUtime=4712.69 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 64596 0 0 0 471204 65 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=2900972 CPUtime=4720.06 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 58556 0 0 0 471955 51 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=2900972 CPUtime=4718.67 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 76454 0 0 0 471762 105 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=2900972 CPUtime=4709.23 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 137249 0 0 0 470751 172 0 0 25 0 8 0 859135955 2970595328 652581 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37741.8
Current children cumulated vsize (KiB) 2900972

[startup+4782.3 s]
/proc/loadavg: 8.08 8.08 8.01 9/178 2358
/proc/meminfo: memFree=29675496/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2900972 CPUtime=38221.4 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 665704 0 0 0 3821447 695 0 0 25 0 8 0 859135890 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4653368 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 725243 653017 466 189 0 719592 0
[pid=1698/tid=1700] ppid=1696 vsize=2900972 CPUtime=4777.75 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 50938 0 0 0 477714 61 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=2900972 CPUtime=4781.42 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 67486 0 0 0 478120 22 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=2900972 CPUtime=4780.99 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 52641 0 0 0 478035 64 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=2900972 CPUtime=4772.68 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 64681 0 0 0 477203 65 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=2900972 CPUtime=4780.05 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 58702 0 0 0 477954 51 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=2900972 CPUtime=4778.66 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 76454 0 0 0 477761 105 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=2900972 CPUtime=4768.91 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 137249 0 0 0 476718 173 0 0 25 0 8 0 859135955 2970595328 653017 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38221.4
Current children cumulated vsize (KiB) 2900972

[startup+4842.3 s]
/proc/loadavg: 8.06 8.08 8.01 9/178 2359
/proc/meminfo: memFree=29666664/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2900972 CPUtime=38701 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 668168 0 0 0 3869404 699 0 0 25 0 8 0 859135890 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670499 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 725243 655104 466 189 0 719592 0
[pid=1698/tid=1700] ppid=1696 vsize=2900972 CPUtime=4837.71 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 51819 0 0 0 483709 62 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=2900972 CPUtime=4841.42 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 67486 0 0 0 484120 22 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=2900972 CPUtime=4840.98 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 52658 0 0 0 484034 64 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=2900972 CPUtime=4832.68 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 64682 0 0 0 483203 65 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=2900972 CPUtime=4840.05 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 59036 0 0 0 483953 52 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=2900972 CPUtime=4838.67 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 76464 0 0 0 483761 106 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=2900972 CPUtime=4828.57 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 137287 0 0 0 482684 173 0 0 25 0 8 0 859135955 2970595328 655104 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38701
Current children cumulated vsize (KiB) 2900972

[startup+4902.3 s]
/proc/loadavg: 8.06 8.08 8.01 9/178 2361
/proc/meminfo: memFree=29663788/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2900972 CPUtime=39180.7 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 669122 0 0 0 3917368 700 0 0 25 0 8 0 859135890 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 725243 656058 466 189 0 719592 0
[pid=1698/tid=1700] ppid=1696 vsize=2900972 CPUtime=4897.71 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 51992 0 0 0 489709 62 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=2900972 CPUtime=4901.42 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 67652 0 0 0 490120 22 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=2900972 CPUtime=4900.97 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 52658 0 0 0 490033 64 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=2900972 CPUtime=4892.68 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 64694 0 0 0 489203 65 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=2900972 CPUtime=4900.06 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 59504 0 0 0 489954 52 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=2900972 CPUtime=4898.66 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 76507 0 0 0 489760 106 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=2900972 CPUtime=4888.24 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 137379 0 0 0 488651 173 0 0 25 0 8 0 859135955 2970595328 656058 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39180.7
Current children cumulated vsize (KiB) 2900972

[startup+4962.3 s]
/proc/loadavg: 8.06 8.07 8.00 9/178 2362
/proc/meminfo: memFree=29660156/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2900972 CPUtime=39660.3 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 670022 0 0 0 3965329 702 0 0 25 0 8 0 859135890 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4658556 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 725243 656684 466 189 0 719592 0
[pid=1698/tid=1700] ppid=1696 vsize=2900972 CPUtime=4957.72 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 52152 0 0 0 495709 63 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=2900972 CPUtime=4961.43 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 68202 0 0 0 496120 23 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=2900972 CPUtime=4960.97 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 52664 0 0 0 496033 64 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=2900972 CPUtime=4952.68 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 64750 0 0 0 495203 65 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=2900972 CPUtime=4960.02 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 59629 0 0 0 495949 53 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=2900972 CPUtime=4958.66 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 76507 0 0 0 495760 106 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=2900972 CPUtime=4947.9 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 137382 0 0 0 494617 173 0 0 25 0 8 0 859135955 2970595328 656684 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39660.3
Current children cumulated vsize (KiB) 2900972



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5004.8 s]
/proc/loadavg: 8.07 8.08 8.01 9/178 2364
/proc/meminfo: memFree=29658028/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2902412 CPUtime=40000.1 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 670590 0 0 0 3999305 703 0 0 25 0 8 0 859135890 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1698/statm: 725603 657252 466 189 0 719952 0
[pid=1698/tid=1700] ppid=1696 vsize=2902412 CPUtime=5000.22 cores=0-7
/proc/1698/task/1700/stat : 1700 (claspmt) R 1696 1698 1559 0 -1 4202560 52412 0 0 0 499959 63 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1698/tid=1701] ppid=1696 vsize=2902412 CPUtime=5003.93 cores=0-7
/proc/1698/task/1701/stat : 1701 (claspmt) R 1696 1698 1559 0 -1 4202560 68298 0 0 0 500370 23 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4321029 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1698/tid=1702] ppid=1696 vsize=2902412 CPUtime=5003.48 cores=0-7
/proc/1698/task/1702/stat : 1702 (claspmt) R 1696 1698 1559 0 -1 4202560 52664 0 0 0 500283 65 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1698/tid=1703] ppid=1696 vsize=2902412 CPUtime=4995.19 cores=0-7
/proc/1698/task/1703/stat : 1703 (claspmt) R 1696 1698 1559 0 -1 4202560 64826 0 0 0 499453 66 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1698/tid=1704] ppid=1696 vsize=2902412 CPUtime=5002.53 cores=0-7
/proc/1698/task/1704/stat : 1704 (claspmt) R 1696 1698 1559 0 -1 4202560 59629 0 0 0 500200 53 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1698/tid=1705] ppid=1696 vsize=2902412 CPUtime=5001.16 cores=0-7
/proc/1698/task/1705/stat : 1705 (claspmt) R 1696 1698 1559 0 -1 4202560 76570 0 0 0 500010 106 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1698/tid=1706] ppid=1696 vsize=2902412 CPUtime=4990.17 cores=0-7
/proc/1698/task/1706/stat : 1706 (claspmt) R 1696 1698 1559 0 -1 4202560 137455 0 0 0 498844 173 0 0 25 0 8 0 859135955 2972069888 657252 33554432000 4194304 4964800 140734158358352 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 2902412

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5005.01 s]
/proc/loadavg: 8.07 8.08 8.01 2/172 2366
/proc/meminfo: memFree=29658572/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2896784 CPUtime=40000.3 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 670600 0 0 0 3999331 703 0 0 25 0 1 0 859135890 2966306816 655823 33554432000 4194304 4964800 140734158358352 18446744073709551615 255106424987 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/1698/statm: 724196 655823 476 189 0 718545 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 2896784

[startup+5005.4 s]
/proc/loadavg: 8.07 8.08 8.01 2/172 2366
/proc/meminfo: memFree=29658572/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=2882192 CPUtime=40000.7 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 670600 0 0 0 3999371 703 0 0 25 0 1 0 859135890 2951364608 644628 33554432000 4194304 4964800 140734158358352 18446744073709551615 255106425393 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/1698/statm: 720548 644628 476 189 0 714897 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 2882192

[startup+5005.5 s]
/proc/loadavg: 8.07 8.08 8.01 2/172 2366
/proc/meminfo: memFree=29658572/32951124 swapFree=67088052/67111528
[pid=1698] ppid=1696 vsize=717808 CPUtime=40000.8 cores=0-7
/proc/1698/stat : 1698 (claspmt) R 1696 1698 1559 0 -1 4202496 670600 0 0 0 3999371 713 0 0 25 0 1 0 859135890 735035392 109924 33554432000 4194304 4964800 140734158358352 18446744073709551615 255106812039 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/1698/statm: 179452 109421 476 189 0 173801 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 717808

Child status: 0
Real time (s): 5005.52
CPU time (s): 40000.9
CPU user time (s): 39993.7
CPU system time (s): 7.15291
CPU usage (%): 799.134
Max. virtual memory (cumulated for all children) (KiB): 2902412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.7
system time used= 7.15291
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 670601
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 148911
involuntary context switches= 191281

runsolver used 8.00678 second user time and 19.3091 second system time

The end

Launcher Data 

Begin job on node144 at 2011-06-10 02:53:18
IDJOB=3381376
IDBENCH=88212
IDSOLVER=1865
FILE ID=node144/3381376-1307667198
RUNJOBID= node144-1307666743-1573
PBS_JOBID= 13549483
Free space on /tmp= 74116 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381376-1307667198/watcher-3381376-1307667198 -o /tmp/evaluation-result-3381376-1307667198/solver-3381376-1307667198 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381376-1307667198.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 1bac7b96ea6eb843b1de7405a721cd87
RANDOM SEED=757973733

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      32296252 kB
Buffers:        116912 kB
Cached:         322960 kB
SwapCached:       1556 kB
Active:         187380 kB
Inactive:       295460 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      32296252 kB
SwapTotal:    67111528 kB
SwapFree:     67088052 kB
Dirty:            2728 kB
Writeback:           0 kB
AnonPages:       41028 kB
Mapped:          13964 kB
Slab:           105564 kB
PageTables:       4412 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   181360 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74192 MiB
End job on node144 at 2011-06-10 04:16:44