Trace number 3322612

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 35794.4 5100.4

General information on the benchmark

NameSAT04/handmade/kexu/frb-0.8/
frb59-26-1.used-as.sat04-891.cnf
MD5SUM5877850d9f604b6973676132cbba5c90
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark502.369
Satisfiable
(Un)Satisfiability was proved
Number of variables1534
Number of clauses132295
Sum of the clauses size266006
Maximum clause length26
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2132236
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 559

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...391388/instance-3322612-1304391388.cnf
0.39/0.48	c Solving...
35793.93/5100.01	
35793.93/5100.01	*** INTERRUPTED! ***
35793.93/5100.01	*** Shutting down threads...
35794.26/5100.39	s UNKNOWN
35794.26/5100.39	
35794.26/5100.39	c INTERRUPTED   : 1
35794.26/5100.39	c Models        : 0+    
35794.26/5100.39	c Time          : 5100.019s (Solving: 5099.54s 1st Model: 0.00s Unsat: 0.00s)
35794.26/5100.39	c CPU Time      : 35793.990s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322612-1304391388/watcher-3322612-1304391388 -o /tmp/evaluation-result-3322612-1304391388/solver-3322612-1304391388 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322612-1304391388.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.09 9.06 9.01 2/232 31201
/proc/meminfo: memFree=14173124/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=22004 CPUtime=0 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 843 0 0 0 0 0 0 0 24 0 1 0 550739533 22532096 725 33554432000 4194304 4964704 140736338714208 18446744073709551615 222921437893 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/31201/statm: 5501 726 386 189 0 375 0

[startup+0.083182 s]
/proc/loadavg: 9.09 9.06 9.01 2/232 31201
/proc/meminfo: memFree=14173124/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=29228 CPUtime=0.07 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 3082 0 0 0 7 0 0 0 24 0 1 0 550739533 29929472 2513 33554432000 4194304 4964704 140736338714208 18446744073709551615 4607641 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/31201/statm: 7307 2513 399 189 0 2181 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 29228

[startup+0.100177 s]
/proc/loadavg: 9.09 9.06 9.01 2/232 31201
/proc/meminfo: memFree=14173124/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=29228 CPUtime=0.09 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 3082 0 0 0 9 0 0 0 24 0 1 0 550739533 29929472 2513 33554432000 4194304 4964704 140736338714208 18446744073709551615 4607641 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/31201/statm: 7307 2513 399 189 0 2181 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 29228

[startup+0.300142 s]
/proc/loadavg: 9.09 9.06 9.01 2/232 31201
/proc/meminfo: memFree=14173124/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=29756 CPUtime=0.29 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 3219 0 0 0 29 0 0 0 25 0 1 0 550739533 30470144 2650 33554432000 4194304 4964704 140736338714208 18446744073709551615 4623021 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/31201/statm: 7439 2650 400 189 0 2313 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 29756

[startup+0.701034 s]
/proc/loadavg: 9.09 9.06 9.01 2/232 31201
/proc/meminfo: memFree=14173124/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=389600 CPUtime=1.43 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 6130 0 0 0 141 2 0 0 25 0 8 0 550739533 398950400 5286 33554432000 4194304 4964704 140736338714208 18446744073709551615 4652238 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31201/statm: 97400 5286 464 189 0 91749 0
Current children cumulated CPU time (s) 1.43
Current children cumulated vsize (KiB) 389600

[startup+1.50089 s]
/proc/loadavg: 8.53 8.94 8.97 11/240 31209
/proc/meminfo: memFree=14144672/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=390392 CPUtime=7.03 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 10886 0 0 0 698 5 0 0 25 0 8 0 550739533 399761408 10042 33554432000 4194304 4964704 140736338714208 18446744073709551615 4374314 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31201/statm: 97598 10043 464 189 0 91947 0
[pid=31201/tid=31203] ppid=31199 vsize=390392 CPUtime=0.92 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 925 0 0 0 92 0 0 0 25 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=390392 CPUtime=0.91 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 1039 0 0 0 91 0 0 0 18 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=390392 CPUtime=0.91 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 1023 0 0 0 91 0 0 0 18 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 4653154 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=390392 CPUtime=0.92 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 982 0 0 0 91 1 0 0 18 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=390392 CPUtime=0.99 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 528 0 0 0 99 0 0 0 25 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=390392 CPUtime=0.84 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 932 0 0 0 84 0 0 0 19 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=390392 CPUtime=0.91 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 1783 0 0 0 91 0 0 0 18 0 8 0 550739582 399761408 10043 33554432000 4194304 4964704 140736338714208 18446744073709551615 222921437893 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 7.03
Current children cumulated vsize (KiB) 390392

[startup+3.1006 s]
/proc/loadavg: 8.53 8.94 8.97 10/240 31209
/proc/meminfo: memFree=14122588/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=395804 CPUtime=18.19 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 18323 0 0 0 1811 8 0 0 25 0 8 0 550739533 405303296 17479 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627281 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/31201/statm: 98951 17479 464 189 0 93300 0
[pid=31201/tid=31203] ppid=31199 vsize=395804 CPUtime=2.52 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 2106 0 0 0 252 0 0 0 25 0 8 0 550739582 405303296 17479 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670281 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=395804 CPUtime=2.51 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 2272 0 0 0 250 1 0 0 20 0 8 0 550739582 405303296 17480 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=395804 CPUtime=2.51 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 1715 0 0 0 250 1 0 0 20 0 8 0 550739582 405303296 17480 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=395804 CPUtime=2.5 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 2130 0 0 0 249 1 0 0 20 0 8 0 550739582 405303296 17480 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=395804 CPUtime=2.59 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 712 0 0 0 259 0 0 0 25 0 8 0 550739582 405303296 17480 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=395804 CPUtime=2.32 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 1599 0 0 0 232 0 0 0 22 0 8 0 550739582 405303296 17480 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=395804 CPUtime=2.27 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 3801 0 0 0 225 2 0 0 20 0 8 0 550739582 405303296 17480 33554432000 4194304 4964704 140736338714208 18446744073709551615 222921438241 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 18.19
Current children cumulated vsize (KiB) 395804

[startup+6.30101 s]
/proc/loadavg: 8.53 8.94 8.97 10/240 31209
/proc/meminfo: memFree=14068288/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=402956 CPUtime=40.54 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 28594 0 0 0 4040 14 0 0 25 0 8 0 550739533 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627263 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/31201/statm: 100739 27750 466 189 0 95088 0
[pid=31201/tid=31203] ppid=31199 vsize=402956 CPUtime=5.71 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 3348 0 0 0 570 1 0 0 25 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4684659 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=402956 CPUtime=5.64 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 3697 0 0 0 562 2 0 0 25 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=402956 CPUtime=5.66 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 2885 0 0 0 565 1 0 0 19 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=402956 CPUtime=5.59 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 3435 0 0 0 557 2 0 0 25 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=402956 CPUtime=5.75 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 1976 0 0 0 575 0 0 0 25 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=402956 CPUtime=5.49 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 2572 0 0 0 548 1 0 0 25 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=402956 CPUtime=4.05 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 6145 0 0 0 402 3 0 0 25 0 8 0 550739582 412626944 27750 33554432000 4194304 4964704 140736338714208 18446744073709551615 4668907 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40.54
Current children cumulated vsize (KiB) 402956

[startup+12.7008 s]
/proc/loadavg: 8.60 8.94 8.97 10/240 31210
/proc/meminfo: memFree=14053012/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=403352 CPUtime=85.28 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 31483 0 0 0 8512 16 0 0 25 0 8 0 550739533 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670215 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/31201/statm: 100838 30639 466 189 0 95187 0
[pid=31201/tid=31203] ppid=31199 vsize=403352 CPUtime=10.3 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 3714 0 0 0 1029 1 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4654527 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=403352 CPUtime=12.04 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 3918 0 0 0 1202 2 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=403352 CPUtime=10.88 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 3104 0 0 0 1087 1 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=403352 CPUtime=9.8 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 3583 0 0 0 978 2 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=403352 CPUtime=12.16 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 2268 0 0 0 1215 1 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=403352 CPUtime=11.84 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 2925 0 0 0 1183 1 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4632289 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=403352 CPUtime=9.21 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 6938 0 0 0 918 3 0 0 25 0 8 0 550739582 413032448 30639 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 85.28
Current children cumulated vsize (KiB) 403352

[startup+25.5005 s]
/proc/loadavg: 8.66 8.94 8.97 10/240 31210
/proc/meminfo: memFree=14010200/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=412228 CPUtime=174.73 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 41894 0 0 0 17452 21 0 0 25 0 8 0 550739533 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670184 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/31201/statm: 103057 41050 466 189 0 97406 0
[pid=31201/tid=31203] ppid=31199 vsize=412228 CPUtime=18.61 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 4454 0 0 0 1860 1 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670462 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=412228 CPUtime=24.82 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 5261 0 0 0 2480 2 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=412228 CPUtime=18.89 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 3778 0 0 0 1887 2 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320848 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=412228 CPUtime=21.76 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 4325 0 0 0 2174 2 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670281 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=412228 CPUtime=24.95 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 3315 0 0 0 2493 2 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=412228 CPUtime=24.54 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 4017 0 0 0 2452 2 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4660292 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=412228 CPUtime=19.32 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 10320 0 0 0 1927 5 0 0 25 0 8 0 550739582 422121472 41050 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 174.73
Current children cumulated vsize (KiB) 412228

[startup+51.1008 s]
/proc/loadavg: 8.79 8.95 8.97 10/240 31211
/proc/meminfo: memFree=13949772/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=421348 CPUtime=353.66 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 56540 0 0 0 35336 30 0 0 25 0 8 0 550739533 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321016 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/31201/statm: 105337 55690 466 189 0 99686 0
[pid=31201/tid=31203] ppid=31199 vsize=421348 CPUtime=40 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 6491 0 0 0 3997 3 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4683962 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=421348 CPUtime=50.12 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 7268 0 0 0 5009 3 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=421348 CPUtime=41.56 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 5207 0 0 0 4153 3 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=421348 CPUtime=42.23 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 4981 0 0 0 4220 3 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=421348 CPUtime=46.18 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 4659 0 0 0 4616 2 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=421348 CPUtime=49.94 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 5297 0 0 0 4991 3 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=421348 CPUtime=41.54 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 14741 0 0 0 4147 7 0 0 25 0 8 0 550739582 431460352 55690 33554432000 4194304 4964704 140736338714208 18446744073709551615 222921438241 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 353.66

################
# More data... #
################

/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21077 0 0 0 410000 28 0 0 25 0 8 0 550739582 1633210368 323201 33554432000 4194304 4964704 140736338714208 18446744073709551615 4342199 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1594932 CPUtime=4218.71 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 133403 0 0 0 421720 151 0 0 25 0 8 0 550739582 1633210368 323201 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670230 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33572.4
Current children cumulated vsize (KiB) 1594932

[startup+4842.3 s]
/proc/loadavg: 9.01 9.03 9.00 10/240 31362
/proc/meminfo: memFree=12130264/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1601800 CPUtime=33991.9 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 333786 0 0 0 3398834 359 0 0 25 0 8 0 550739533 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/31201/statm: 400450 326490 466 189 0 394799 0
[pid=31201/tid=31203] ppid=31199 vsize=1601800 CPUtime=4280.19 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 35964 0 0 0 427985 34 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=1601800 CPUtime=4640.27 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 36114 0 0 0 464001 26 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4644868 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=1601800 CPUtime=3977.2 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 20196 0 0 0 397694 26 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670114 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=1601800 CPUtime=4345.85 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 22892 0 0 0 434564 21 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=1601800 CPUtime=4485.2 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 22838 0 0 0 448498 22 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=1601800 CPUtime=4159.84 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21146 0 0 0 415956 28 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1601800 CPUtime=4278.71 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 135186 0 0 0 427718 153 0 0 25 0 8 0 550739582 1640243200 326490 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33991.9
Current children cumulated vsize (KiB) 1601800

[startup+4902.3 s]
/proc/loadavg: 9.00 9.02 9.00 10/240 31364
/proc/meminfo: memFree=12118300/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1602228 CPUtime=34411.5 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 334187 0 0 0 3440785 362 0 0 25 0 8 0 550739533 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321079 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/31201/statm: 400557 326881 466 189 0 394906 0
[pid=31201/tid=31203] ppid=31199 vsize=1602228 CPUtime=4330.95 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 35967 0 0 0 433060 35 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=1602228 CPUtime=4671.18 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 36114 0 0 0 467092 26 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=1602228 CPUtime=4037.2 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 20305 0 0 0 403694 26 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=1602228 CPUtime=4405.85 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 23070 0 0 0 440564 21 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=1602228 CPUtime=4544.45 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 22860 0 0 0 454422 23 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=1602228 CPUtime=4219.41 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21219 0 0 0 421912 29 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1602228 CPUtime=4336.86 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 135202 0 0 0 433532 154 0 0 25 0 8 0 550739582 1640681472 326881 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34411.5
Current children cumulated vsize (KiB) 1602228

[startup+4962.3 s]
/proc/loadavg: 9.00 9.02 9.00 10/240 31366
/proc/meminfo: memFree=12106108/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1602228 CPUtime=34831 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 334785 0 0 0 3482741 362 0 0 25 0 8 0 550739533 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/31201/statm: 400557 327463 466 189 0 394906 0
[pid=31201/tid=31203] ppid=31199 vsize=1602228 CPUtime=4390.95 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 35967 0 0 0 439060 35 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=1602228 CPUtime=4731.18 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 36295 0 0 0 473092 26 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=1602228 CPUtime=4097.2 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 20351 0 0 0 409694 26 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=1602228 CPUtime=4465.85 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 23279 0 0 0 446564 21 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=1602228 CPUtime=4586.51 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 22861 0 0 0 458628 23 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670271 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=1602228 CPUtime=4278.97 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21268 0 0 0 427868 29 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1602228 CPUtime=4374.95 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 135314 0 0 0 437341 154 0 0 25 0 8 0 550739582 1640681472 327463 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670288 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 34831
Current children cumulated vsize (KiB) 1602228

[startup+5022.3 s]
/proc/loadavg: 9.05 9.03 9.00 10/240 31367
/proc/meminfo: memFree=12095148/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1603432 CPUtime=35250.6 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 335241 0 0 0 3524694 363 0 0 25 0 8 0 550739533 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670263 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/31201/statm: 400858 327919 466 189 0 395207 0
[pid=31201/tid=31203] ppid=31199 vsize=1603432 CPUtime=4447.22 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 36268 0 0 0 444687 35 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=1603432 CPUtime=4791.18 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 36299 0 0 0 479092 26 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4660362 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=1603432 CPUtime=4157.19 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 20351 0 0 0 415693 26 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=1603432 CPUtime=4525.85 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 23286 0 0 0 452564 21 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=1603432 CPUtime=4646.51 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 22953 0 0 0 464628 23 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320953 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=1603432 CPUtime=4316.64 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21268 0 0 0 431635 29 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1603432 CPUtime=4429.81 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 135366 0 0 0 442827 154 0 0 25 0 8 0 550739582 1641914368 327919 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35250.6
Current children cumulated vsize (KiB) 1603432

[startup+5082.3 s]
/proc/loadavg: 9.06 9.03 9.00 10/240 31369
/proc/meminfo: memFree=12085196/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1603432 CPUtime=35670.1 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 335575 0 0 0 3566647 364 0 0 25 0 8 0 550739533 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31201/statm: 400858 328238 466 189 0 395207 0
[pid=31201/tid=31203] ppid=31199 vsize=1603432 CPUtime=4507.01 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 36268 0 0 0 450666 35 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=1603432 CPUtime=4851.18 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 36451 0 0 0 485092 26 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320934 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=1603432 CPUtime=4217.19 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 20376 0 0 0 421693 26 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=1603432 CPUtime=4563.96 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 23286 0 0 0 456375 21 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=1603432 CPUtime=4696.41 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 22989 0 0 0 469618 23 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=1603432 CPUtime=4372.68 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21364 0 0 0 437238 30 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1603432 CPUtime=4489.81 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 135366 0 0 0 448827 154 0 0 25 0 8 0 550739582 1641914368 328238 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35670.1
Current children cumulated vsize (KiB) 1603432



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.04 9.03 9.00 10/240 31369
/proc/meminfo: memFree=12082580/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1603432 CPUtime=35793.9 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 335578 0 0 0 3579028 365 0 0 25 0 8 0 550739533 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31201/statm: 400858 328241 466 189 0 395207 0
[pid=31201/tid=31203] ppid=31199 vsize=1603432 CPUtime=4524.72 cores=0-7
/proc/31201/task/31203/stat : 31203 (claspmt) R 31199 31201 31067 0 -1 4202560 36268 0 0 0 452437 35 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31201/tid=31204] ppid=31199 vsize=1603432 CPUtime=4868.89 cores=0-7
/proc/31201/task/31204/stat : 31204 (claspmt) R 31199 31201 31067 0 -1 4202560 36454 0 0 0 486863 26 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31201/tid=31205] ppid=31199 vsize=1603432 CPUtime=4234.89 cores=0-7
/proc/31201/task/31205/stat : 31205 (claspmt) R 31199 31201 31067 0 -1 4202560 20376 0 0 0 423463 26 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31201/tid=31206] ppid=31199 vsize=1603432 CPUtime=4581.66 cores=0-7
/proc/31201/task/31206/stat : 31206 (claspmt) R 31199 31201 31067 0 -1 4202560 23286 0 0 0 458145 21 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31201/tid=31207] ppid=31199 vsize=1603432 CPUtime=4712.48 cores=0-7
/proc/31201/task/31207/stat : 31207 (claspmt) R 31199 31201 31067 0 -1 4202560 22989 0 0 0 471225 23 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31201/tid=31208] ppid=31199 vsize=1603432 CPUtime=4381.78 cores=0-7
/proc/31201/task/31208/stat : 31208 (claspmt) R 31199 31201 31067 0 -1 4202560 21364 0 0 0 438148 30 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31201/tid=31209] ppid=31199 vsize=1603432 CPUtime=4499.91 cores=0-7
/proc/31201/task/31209/stat : 31209 (claspmt) R 31199 31201 31067 0 -1 4202560 135366 0 0 0 449837 154 0 0 25 0 8 0 550739582 1641914368 328241 33554432000 4194304 4964704 140736338714208 18446744073709551615 4321029 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 35793.9
Current children cumulated vsize (KiB) 1603432

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.2 s]
/proc/loadavg: 9.04 9.03 9.00 10/240 31369
/proc/meminfo: memFree=12082580/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1595944 CPUtime=35794.2 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 335588 0 0 0 3579052 365 0 0 25 0 1 0 550739533 1634246656 326539 33554432000 4194304 4964704 140736338714208 18446744073709551615 222921433659 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/31201/statm: 398986 326539 476 189 0 393335 0
Current children cumulated CPU time (s) 35794.2
Current children cumulated vsize (KiB) 1595944

[startup+5100.3 s]
/proc/loadavg: 9.04 9.03 9.00 10/240 31369
/proc/meminfo: memFree=12082580/32951124 swapFree=67110188/67111528
[pid=31201] ppid=31199 vsize=1592296 CPUtime=35794.3 cores=0-7
/proc/31201/stat : 31201 (claspmt) R 31199 31201 31067 0 -1 4202496 335588 0 0 0 3579061 365 0 0 25 0 1 0 550739533 1630511104 324076 33554432000 4194304 4964704 140736338714208 18446744073709551615 222921433649 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/31201/statm: 398074 324076 476 189 0 392423 0
Current children cumulated CPU time (s) 35794.3
Current children cumulated vsize (KiB) 1592296

Child status: 0
Real time (s): 5100.4
CPU time (s): 35794.4
CPU user time (s): 35790.7
CPU system time (s): 3.69944
CPU usage (%): 701.796
Max. virtual memory (cumulated for all children) (KiB): 1603432

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35790.7
system time used= 3.69944
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 335589
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 60376
involuntary context switches= 235107

runsolver used 9.75252 second user time and 27.3548 second system time

The end

Launcher Data 

Begin job on node109 at 2011-05-03 04:56:28
IDJOB=3322612
IDBENCH=84568
IDSOLVER=1711
FILE ID=node109/3322612-1304391388
RUNJOBID= node109-1304388597-31081
PBS_JOBID= 13181822
Free space on /tmp= 73324 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT04/handmade/kexu/frb-0.8/frb59-26-1.used-as.sat04-891.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322612-1304391388/watcher-3322612-1304391388 -o /tmp/evaluation-result-3322612-1304391388/solver-3322612-1304391388 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322612-1304391388.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 5877850d9f604b6973676132cbba5c90
RANDOM SEED=581409239

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14173280 kB
Buffers:        873256 kB
Cached:        2805648 kB
SwapCached:        712 kB
Active:       15062980 kB
Inactive:      3282020 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14173280 kB
SwapTotal:    67111528 kB
SwapFree:     67110188 kB
Dirty:            2380 kB
Writeback:           8 kB
AnonPages:    14665296 kB
Mapped:          21716 kB
Slab:           333132 kB
PageTables:      37072 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 21521752 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73324 MiB
End job on node109 at 2011-05-03 06:21:30