Trace number 3380986

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.6 5012.43

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n220-s24359701-sat.cnf
MD5SUMceff918324677440f95867755c4e7133
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark6.70998
Satisfiable
(Un)Satisfiability was proved
Number of variables220
Number of clauses528
Sum of the clauses size1320
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2440
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 588
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...499042/instance-3380986-1305499042.cnf
0.00/0.00	c Solving...
40000.52/5012.41	
40000.52/5012.41	*** INTERRUPTED! ***
40000.52/5012.41	*** Shutting down threads...
40000.52/5012.42	s UNKNOWN
40000.52/5012.42	
40000.52/5012.42	c INTERRUPTED   : 1
40000.52/5012.42	c Models        : 0+    
40000.52/5012.42	c Time          : 5012.416s (Solving: 5012.41s 1st Model: 0.00s Unsat: 0.00s)
40000.52/5012.42	c CPU Time      : 40000.580s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380986-1305499042/watcher-3380986-1305499042 -o /tmp/evaluation-result-3380986-1305499042/solver-3380986-1305499042 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380986-1305499042.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.01 0.00 2/363 3991
/proc/meminfo: memFree=22759872/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=119016 CPUtime=0.01 cores=0-7
/proc/3991/stat : 3991 (claspmt) S 3989 3991 3561 0 -1 4202496 846 0 0 0 1 0 0 0 24 0 8 0 642319550 121872384 768 33554432000 4194304 4964800 140737070213648 18446744073709551615 254273432473 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 29754 768 459 189 0 24103 0
[pid=3991/tid=3992] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 41 0 0 0 0 0 0 0 18 0 8 0 642319551 189116416 830 33554432000 4194304 4964800 140737070213648 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 8 0 642319551 189116416 830 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261277381 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 21 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 830 33554432000 4194304 4964800 140737070213648 18446744073709551615 4465565 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 26 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 831 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721582 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 25 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 831 33554432000 4194304 4964800 140737070213648 18446744073709551615 254273442485 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 21 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 831 33554432000 4194304 4964800 140737070213648 18446744073709551615 4467677 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 92 0 0 0 0 0 0 0 18 0 8 0 642319551 189116416 831 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 2 0 0 0

[startup+0.023088 s]
/proc/loadavg: 0.00 0.01 0.00 2/363 3991
/proc/meminfo: memFree=22759872/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=184684 CPUtime=0.06 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 987 0 0 0 6 0 0 0 25 0 8 0 642319550 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 46171 909 463 189 0 40520 0
[pid=3991/tid=3992] ppid=3989 vsize=184684 CPUtime=0.01 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 55 0 0 0 1 0 0 0 18 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=184684 CPUtime=0.01 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 63 0 0 0 1 0 0 0 18 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 4653193 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 21 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 4379076 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 26 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 25 0 0 0 0 0 0 0 25 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 4668564 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=184684 CPUtime=0 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 28 0 0 0 0 0 0 0 18 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=184684 CPUtime=0.01 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 112 0 0 0 1 0 0 0 18 0 8 0 642319551 189116416 909 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 184684

[startup+0.104042 s]
/proc/loadavg: 0.00 0.01 0.00 2/363 3991
/proc/meminfo: memFree=22759872/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=250484 CPUtime=0.72 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 1362 0 0 0 71 1 0 0 18 0 8 0 642319550 256495616 1284 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 62654 1312 464 189 0 57003 0
[pid=3991/tid=3992] ppid=3989 vsize=250616 CPUtime=0.11 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 110 0 0 0 11 0 0 0 18 0 8 0 642319551 256630784 1315 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=250616 CPUtime=0.12 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 162 0 0 0 12 0 0 0 18 0 8 0 642319551 256630784 1358 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=316152 CPUtime=0.14 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 78 0 0 0 14 0 0 0 18 0 8 0 642319551 323739648 1498 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=316152 CPUtime=0.18 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) S 3989 3991 3561 0 -1 4202560 114 0 0 0 18 0 0 0 18 0 8 0 642319551 323739648 1574 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721582 0 0 4096 18946 18446604450725534720 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=316152 CPUtime=0.19 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 83 0 0 0 19 0 0 0 18 0 8 0 642319551 323739648 1591 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=316416 CPUtime=0.26 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 98 0 0 0 26 0 0 0 18 0 8 0 642319551 324009984 1728 33554432000 4194304 4964800 140737070213648 18446744073709551615 4668564 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=316416 CPUtime=0.31 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 263 0 0 0 31 0 0 0 18 0 8 0 642319551 324009984 1758 33554432000 4194304 4964800 140737070213648 18446744073709551615 4365162 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 0.72
Current children cumulated vsize (KiB) 250484

[startup+0.612848 s]
/proc/loadavg: 0.00 0.01 0.00 2/363 3991
/proc/meminfo: memFree=22759872/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=316548 CPUtime=4.94 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 2158 0 0 0 492 2 0 0 18 0 8 0 642319550 324145152 2080 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 79170 2098 464 189 0 73519 0
[pid=3991/tid=3992] ppid=3989 vsize=316680 CPUtime=0.68 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 217 0 0 0 68 0 0 0 18 0 8 0 642319551 324280320 2099 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=316680 CPUtime=0.71 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 317 0 0 0 71 0 0 0 18 0 8 0 642319551 324280320 2163 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261313291 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=316680 CPUtime=0.76 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 119 0 0 0 76 0 0 0 18 0 8 0 642319551 324280320 2216 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=316680 CPUtime=0.81 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) S 3989 3991 3561 0 -1 4202560 231 0 0 0 81 0 0 0 18 0 8 0 642319551 324280320 2259 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721582 0 0 4096 18946 18446604450725534720 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=316808 CPUtime=0.82 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 119 0 0 0 82 0 0 0 19 0 8 0 642319551 324411392 2271 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=316808 CPUtime=0.84 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 142 0 0 0 84 0 0 0 19 0 8 0 642319551 324411392 2274 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=316808 CPUtime=0.92 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 335 0 0 0 92 0 0 0 19 0 8 0 642319551 324411392 2318 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670282 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.94
Current children cumulated vsize (KiB) 316548

[startup+1.22355 s]
/proc/loadavg: 0.00 0.01 0.00 2/363 3991
/proc/meminfo: memFree=22759872/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=317060 CPUtime=9.7 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 2579 0 0 0 966 4 0 0 19 0 8 0 642319550 324669440 2501 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 79265 2547 464 189 0 73614 0
[pid=3991/tid=3992] ppid=3989 vsize=317060 CPUtime=1.34 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 296 0 0 0 134 0 0 0 18 0 8 0 642319551 324669440 2570 33554432000 4194304 4964800 140737070213648 18446744073709551615 4362301 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=317060 CPUtime=1.35 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 387 0 0 0 135 0 0 0 18 0 8 0 642319551 324669440 2588 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=317060 CPUtime=1.37 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 141 0 0 0 137 0 0 0 19 0 8 0 642319551 324669440 2595 33554432000 4194304 4964800 140737070213648 18446744073709551615 4363270 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=317060 CPUtime=1.44 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) S 3989 3991 3561 0 -1 4202560 300 0 0 0 144 0 0 0 18 0 8 0 642319551 324669440 2617 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721582 0 0 4096 18946 18446604450725534720 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=317060 CPUtime=1.42 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 139 0 0 0 142 0 0 0 18 0 8 0 642319551 324669440 2617 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=317060 CPUtime=1.48 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 168 0 0 0 148 0 0 0 20 0 8 0 642319551 324669440 2630 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=317060 CPUtime=1.53 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 363 0 0 0 153 0 0 0 20 0 8 0 642319551 324669440 2630 33554432000 4194304 4964800 140737070213648 18446744073709551615 254261721631 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 9.7
Current children cumulated vsize (KiB) 317060

[startup+1.92935 s]
/proc/loadavg: 0.00 0.01 0.00 2/363 3991
/proc/meminfo: memFree=22759872/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=317192 CPUtime=15.17 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 2846 0 0 0 1511 6 0 0 20 0 8 0 642319550 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 79298 2768 464 189 0 73647 0
[pid=3991/tid=3992] ppid=3989 vsize=317192 CPUtime=1.91 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 336 0 0 0 191 0 0 0 19 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=317192 CPUtime=1.89 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 429 0 0 0 189 0 0 0 19 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4324775 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=317192 CPUtime=1.87 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 153 0 0 0 187 0 0 0 20 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670368 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=317192 CPUtime=1.89 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 317 0 0 0 189 0 0 0 19 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4688702 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=317192 CPUtime=1.88 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 144 0 0 0 187 1 0 0 19 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660444 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=317192 CPUtime=1.87 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 175 0 0 0 187 0 0 0 21 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=317192 CPUtime=1.91 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 376 0 0 0 191 0 0 0 21 0 8 0 642319551 324804608 2768 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 15.17
Current children cumulated vsize (KiB) 317192

[startup+3.10028 s]
/proc/loadavg: 0.72 0.16 0.05 9/371 3999
/proc/meminfo: memFree=22751012/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=317456 CPUtime=24.51 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 3202 0 0 0 2443 8 0 0 25 0 8 0 642319550 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660449 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 79364 3124 464 189 0 73713 0
[pid=3991/tid=3992] ppid=3989 vsize=317456 CPUtime=3.07 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 409 0 0 0 307 0 0 0 22 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=317456 CPUtime=3.06 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 474 0 0 0 305 1 0 0 21 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=317456 CPUtime=3.04 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 176 0 0 0 304 0 0 0 24 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660172 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=317456 CPUtime=3.04 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 385 0 0 0 304 0 0 0 21 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4362472 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=317456 CPUtime=3.05 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 176 0 0 0 304 1 0 0 18 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=317456 CPUtime=3.05 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 194 0 0 0 304 1 0 0 19 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=317456 CPUtime=3.08 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 425 0 0 0 307 1 0 0 25 0 8 0 642319551 325074944 3124 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.51
Current children cumulated vsize (KiB) 317456

[startup+6.30072 s]
/proc/loadavg: 1.30 0.29 0.09 9/371 3999
/proc/meminfo: memFree=22748260/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=317860 CPUtime=50.02 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 3787 0 0 0 4987 15 0 0 25 0 8 0 642319550 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660298 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 79465 3709 464 189 0 73814 0
[pid=3991/tid=3992] ppid=3989 vsize=317860 CPUtime=6.27 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 547 0 0 0 626 1 0 0 25 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670361 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=317860 CPUtime=6.26 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 560 0 0 0 624 2 0 0 25 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=317860 CPUtime=6.24 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 206 0 0 0 623 1 0 0 19 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660163 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=317860 CPUtime=6.21 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 472 0 0 0 619 2 0 0 18 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=317860 CPUtime=6.24 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 231 0 0 0 622 2 0 0 18 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=317860 CPUtime=6.24 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 241 0 0 0 622 2 0 0 23 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=317860 CPUtime=6.28 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 495 0 0 0 626 2 0 0 25 0 8 0 642319551 325488640 3709 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670321 0 0 4096 18946 0 0 0 -1 2 0 0 0

################
# More data... #
################


[startup+4722.3 s]
/proc/loadavg: 8.10 8.02 7.94 9/371 4137
/proc/meminfo: memFree=22726796/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=37685.2 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9233 0 0 0 3763703 4819 0 0 25 0 8 0 642319550 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4683996 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9150 464 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4703.34 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1554 0 0 0 469675 659 0 0 18 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4653230 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=4715.73 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1392 0 0 0 470947 626 0 0 25 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660168 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4709.01 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 708 0 0 0 470355 546 0 0 18 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=4711.55 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1200 0 0 0 470496 659 0 0 21 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=4713.2 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 470658 662 0 0 19 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4707.43 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 705 0 0 0 470148 595 0 0 19 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=4713.08 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1032 0 0 0 470770 538 0 0 25 0 8 0 642319551 396058624 9150 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37685.2
Current children cumulated vsize (KiB) 386776

[startup+4782.3 s]
/proc/loadavg: 8.11 8.03 7.94 9/371 4138
/proc/meminfo: memFree=22726900/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=38164.1 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9235 0 0 0 3811524 4882 0 0 18 0 8 0 642319550 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4362240 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9152 464 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4763.29 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1554 0 0 0 475663 666 0 0 25 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4319827 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=4775.66 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1392 0 0 0 476932 634 0 0 25 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627540 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4768.24 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 708 0 0 0 476270 554 0 0 19 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=4771.47 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1200 0 0 0 476480 667 0 0 25 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660157 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=4773.14 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 476644 670 0 0 25 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4767.37 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 706 0 0 0 476135 602 0 0 25 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320976 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=4773.03 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1033 0 0 0 476759 544 0 0 25 0 8 0 642319551 396058624 9152 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38164.1
Current children cumulated vsize (KiB) 386776

[startup+4842.3 s]
/proc/loadavg: 8.04 8.02 7.94 9/371 4140
/proc/meminfo: memFree=22726764/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=38642.9 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9241 0 0 0 3859348 4943 0 0 25 0 8 0 642319550 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4332581 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9158 468 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4823.24 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1556 0 0 0 481652 672 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320848 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=4835.6 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1392 0 0 0 482917 643 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4654549 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4827.5 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 708 0 0 0 482188 562 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627451 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=4831.39 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1200 0 0 0 482463 676 0 0 19 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321093 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=4833.08 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 482629 679 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4827.31 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 482121 610 0 0 22 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320883 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=4832.99 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1033 0 0 0 482747 552 0 0 18 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38642.9
Current children cumulated vsize (KiB) 386776

[startup+4902.3 s]
/proc/loadavg: 8.01 8.02 7.94 9/371 4141
/proc/meminfo: memFree=22726864/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=39121.8 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9241 0 0 0 3907175 5001 0 0 25 0 8 0 642319550 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670295 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9158 468 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4883.18 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1556 0 0 0 487640 678 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4684040 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=4895.51 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1392 0 0 0 488901 650 0 0 22 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4886.75 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 708 0 0 0 488105 570 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627290 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=4891.34 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1200 0 0 0 488449 685 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=4893.02 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 488615 687 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660447 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4887.27 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 488111 616 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660148 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=4892.95 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1033 0 0 0 488736 559 0 0 25 0 8 0 642319551 396058624 9158 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39121.8
Current children cumulated vsize (KiB) 386776

[startup+4962.3 s]
/proc/loadavg: 8.00 8.01 7.94 9/371 4143
/proc/meminfo: memFree=22726716/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=39600.6 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9248 0 0 0 3955006 5054 0 0 25 0 8 0 642319550 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321002 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9165 468 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4943.14 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1556 0 0 0 493627 687 0 0 25 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4362211 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=4955.42 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1393 0 0 0 494886 656 0 0 25 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4946.64 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 708 0 0 0 494087 577 0 0 25 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=4951.04 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1206 0 0 0 494411 693 0 0 18 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=4952.58 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 494564 694 0 0 18 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627960 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4947.21 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 494099 622 0 0 18 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=4952.89 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1033 0 0 0 494725 564 0 0 25 0 8 0 642319551 396058624 9165 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39600.6
Current children cumulated vsize (KiB) 386776



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5012.4 s]
/proc/loadavg: 8.00 8.01 7.94 9/371 4144
/proc/meminfo: memFree=22726700/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=40000.5 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9252 0 0 0 3994945 5107 0 0 25 0 8 0 642319550 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4362287 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9169 468 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4993.22 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1558 0 0 0 498630 692 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=5005.3 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1393 0 0 0 499866 664 0 0 18 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4996.7 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 499087 583 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4653558 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=5001.11 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1206 0 0 0 499410 701 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660172 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=5002.2 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 499518 702 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670415 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4997.28 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 499100 628 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=5002.97 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1033 0 0 0 499727 570 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 386776

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5012.4 s]
/proc/loadavg: 8.00 8.01 7.94 9/371 4144
/proc/meminfo: memFree=22726700/32951124 swapFree=67057896/67111528
[pid=3991] ppid=3989 vsize=386776 CPUtime=40000.5 cores=0-7
/proc/3991/stat : 3991 (claspmt) R 3989 3991 3561 0 -1 4202496 9252 0 0 0 3994945 5107 0 0 25 0 8 0 642319550 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4362287 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/3991/statm: 96694 9169 468 189 0 91043 0
[pid=3991/tid=3992] ppid=3989 vsize=386776 CPUtime=4993.22 cores=0-7
/proc/3991/task/3992/stat : 3992 (claspmt) R 3989 3991 3561 0 -1 4202560 1558 0 0 0 498630 692 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3991/tid=3993] ppid=3989 vsize=386776 CPUtime=5005.3 cores=0-7
/proc/3991/task/3993/stat : 3993 (claspmt) R 3989 3991 3561 0 -1 4202560 1393 0 0 0 499866 664 0 0 18 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3991/tid=3994] ppid=3989 vsize=386776 CPUtime=4996.7 cores=0-7
/proc/3991/task/3994/stat : 3994 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 499087 583 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4653558 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3991/tid=3995] ppid=3989 vsize=386776 CPUtime=5001.11 cores=0-7
/proc/3991/task/3995/stat : 3995 (claspmt) R 3989 3991 3561 0 -1 4202560 1206 0 0 0 499410 701 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4660172 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3991/tid=3996] ppid=3989 vsize=386776 CPUtime=5002.2 cores=0-7
/proc/3991/task/3996/stat : 3996 (claspmt) R 3989 3991 3561 0 -1 4202560 953 0 0 0 499518 702 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670415 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3991/tid=3997] ppid=3989 vsize=386776 CPUtime=4997.28 cores=0-7
/proc/3991/task/3997/stat : 3997 (claspmt) R 3989 3991 3561 0 -1 4202560 710 0 0 0 499100 628 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3991/tid=3998] ppid=3989 vsize=386776 CPUtime=5002.97 cores=0-7
/proc/3991/task/3998/stat : 3998 (claspmt) R 3989 3991 3561 0 -1 4202560 1033 0 0 0 499727 570 0 0 25 0 8 0 642319551 396058624 9169 33554432000 4194304 4964800 140737070213648 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 386776

Child status: 0
Real time (s): 5012.43
CPU time (s): 40000.6
CPU user time (s): 39949.5
CPU system time (s): 51.0762
CPU usage (%): 798.029
Max. virtual memory (cumulated for all children) (KiB): 386776

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39949.5
system time used= 51.0762
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 9269
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8357270
involuntary context switches= 188176

runsolver used 17.9883 second user time and 41.2497 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-16 00:37:22
IDJOB=3380986
IDBENCH=83146
IDSOLVER=1865
FILE ID=node140/3380986-1305499042
RUNJOBID= node140-1305488928-3578
PBS_JOBID= 13325404
Free space on /tmp= 73456 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n220-s24359701-sat.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380986-1305499042/watcher-3380986-1305499042 -o /tmp/evaluation-result-3380986-1305499042/solver-3380986-1305499042 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380986-1305499042.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ceff918324677440f95867755c4e7133
RANDOM SEED=326286883

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      22760028 kB
Buffers:        325540 kB
Cached:        4476544 kB
SwapCached:       8088 kB
Active:        6710024 kB
Inactive:      3147380 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      22760028 kB
SwapTotal:    67111528 kB
SwapFree:     67057896 kB
Dirty:             844 kB
Writeback:           0 kB
AnonPages:     5053176 kB
Mapped:          29624 kB
Slab:           227548 kB
PageTables:      36024 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 91922504 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73456 MiB
End job on node140 at 2011-05-16 02:00:57