Trace number 3381306

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.5 5011.54

General information on the benchmark

NameSAT02/handmade/pehoushek/plainoldcnf/
5cnf_3900_3900_220.shuffled.cnf
MD5SUMf0b7e0675915478b15cab3695fd0fd62
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.033994
Satisfiable
(Un)Satisfiability was proved
Number of variables220
Number of clauses3432
Sum of the clauses size17000
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32
Number of clauses of size 4156
Number of clauses of size 53274
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...511844/instance-3381306-1305511844.cnf
0.00/0.00	c Solving...
40000.41/5011.52	
40000.41/5011.52	*** INTERRUPTED! ***
40000.41/5011.52	*** Shutting down threads...
40000.41/5011.54	s UNKNOWN
40000.41/5011.54	
40000.41/5011.54	c INTERRUPTED   : 1
40000.41/5011.54	c Models        : 0+    
40000.41/5011.54	c Time          : 5011.520s (Solving: 5011.51s 1st Model: 0.00s Unsat: 0.00s)
40000.41/5011.54	c CPU Time      : 40000.450s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381306-1305511844/watcher-3381306-1305511844 -o /tmp/evaluation-result-3381306-1305511844/solver-3381306-1305511844 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381306-1305511844.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.35 8.66 8.22 2/466 16640
/proc/meminfo: memFree=29319952/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=251428 CPUtime=0.02 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 1323 0 0 0 2 0 0 0 25 0 8 0 662783123 257462272 1246 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345198 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/16640/statm: 62857 1246 453 189 0 57206 0
[pid=16640/tid=16641] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 71 0 0 0 0 0 0 0 25 0 8 0 662783124 257462272 1250 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 81 0 0 0 0 0 0 0 25 0 8 0 662783124 257462272 1251 33554432000 4194304 4964800 140734293911952 18446744073709551615 251813901345 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 93 0 0 0 0 0 0 0 25 0 8 0 662783124 257462272 1251 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 76 0 0 0 0 0 0 0 25 0 8 0 662783124 257462272 1251 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 82 0 0 0 0 0 0 0 25 0 8 0 662783124 257462272 1251 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670282 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 98 0 0 0 0 0 0 0 25 0 8 0 662783124 257462272 1252 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=251428 CPUtime=0 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 75 0 0 0 0 0 0 0 18 0 8 0 662783124 257462272 1253 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 4 0 0 0

[startup+0.094508 s]
/proc/loadavg: 8.35 8.66 8.22 2/466 16640
/proc/meminfo: memFree=29319952/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=382632 CPUtime=0.59 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 2012 0 0 0 58 1 0 0 18 0 8 0 662783123 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 251813898996 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/16640/statm: 95658 1935 463 189 0 90007 0
[pid=16640/tid=16641] ppid=16638 vsize=382632 CPUtime=0.08 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 171 0 0 0 8 0 0 0 25 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=382632 CPUtime=0.06 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 142 0 0 0 6 0 0 0 18 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=382632 CPUtime=0.06 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 155 0 0 0 6 0 0 0 25 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=382632 CPUtime=0.06 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 149 0 0 0 6 0 0 0 25 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345198 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=382632 CPUtime=0.08 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 160 0 0 0 8 0 0 0 25 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 4660425 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=382632 CPUtime=0.07 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 207 0 0 0 7 0 0 0 25 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 4325245 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=382632 CPUtime=0.07 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 192 0 0 0 7 0 0 0 18 0 8 0 662783124 391815168 1935 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670160 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 382632

[startup+0.118932 s]
/proc/loadavg: 8.35 8.66 8.22 2/466 16640
/proc/meminfo: memFree=29319952/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=382632 CPUtime=0.78 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 2107 0 0 0 77 1 0 0 18 0 8 0 662783123 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/16640/statm: 95658 2030 463 189 0 90007 0
[pid=16640/tid=16641] ppid=16638 vsize=382632 CPUtime=0.1 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 195 0 0 0 10 0 0 0 25 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 4653219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=382632 CPUtime=0.08 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 151 0 0 0 8 0 0 0 18 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=382632 CPUtime=0.09 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 163 0 0 0 9 0 0 0 25 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=382632 CPUtime=0.08 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 171 0 0 0 8 0 0 0 25 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345198 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=382632 CPUtime=0.1 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 176 0 0 0 10 0 0 0 25 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=382632 CPUtime=0.09 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 212 0 0 0 9 0 0 0 25 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=382632 CPUtime=0.1 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 199 0 0 0 10 0 0 0 18 0 8 0 662783124 391815168 2030 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.78
Current children cumulated vsize (KiB) 382632

[startup+0.300344 s]
/proc/loadavg: 8.35 8.66 8.22 2/466 16640
/proc/meminfo: memFree=29319952/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=383028 CPUtime=2.23 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 2799 0 0 0 222 1 0 0 18 0 8 0 662783123 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/16640/statm: 95757 2722 463 189 0 90106 0
[pid=16640/tid=16641] ppid=16638 vsize=383028 CPUtime=0.28 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 309 0 0 0 28 0 0 0 25 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=383028 CPUtime=0.26 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 249 0 0 0 26 0 0 0 18 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=383028 CPUtime=0.27 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 219 0 0 0 27 0 0 0 25 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=383028 CPUtime=0.26 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 239 0 0 0 26 0 0 0 25 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 4362383 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=383028 CPUtime=0.28 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 270 0 0 0 28 0 0 0 25 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=383028 CPUtime=0.27 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 279 0 0 0 27 0 0 0 18 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 4642509 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=383028 CPUtime=0.28 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 313 0 0 0 28 0 0 0 18 0 8 0 662783124 392220672 2722 33554432000 4194304 4964800 140734293911952 18446744073709551615 251814345247 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2.23
Current children cumulated vsize (KiB) 383028

[startup+1.43416 s]
/proc/loadavg: 8.35 8.66 8.22 2/466 16640
/proc/meminfo: memFree=29319952/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=385108 CPUtime=11.29 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 5301 0 0 0 1125 4 0 0 18 0 8 0 662783123 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670437 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/16640/statm: 96277 5224 464 189 0 90626 0
[pid=16640/tid=16641] ppid=16638 vsize=385108 CPUtime=1.4 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 706 0 0 0 140 0 0 0 25 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4334340 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=385108 CPUtime=1.4 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 633 0 0 0 140 0 0 0 19 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4653161 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=385108 CPUtime=1.4 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 537 0 0 0 140 0 0 0 25 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=385108 CPUtime=1.4 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 524 0 0 0 140 0 0 0 19 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=385108 CPUtime=1.41 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 509 0 0 0 141 0 0 0 25 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4362354 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=385108 CPUtime=1.4 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 518 0 0 0 140 0 0 0 19 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4332205 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=385108 CPUtime=1.41 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 668 0 0 0 141 0 0 0 20 0 8 0 662783124 394350592 5224 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321065 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.29
Current children cumulated vsize (KiB) 385108

[startup+1.50114 s]
/proc/loadavg: 8.35 8.66 8.22 2/466 16640
/proc/meminfo: memFree=29319952/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=385108 CPUtime=11.83 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 5307 0 0 0 1179 4 0 0 18 0 8 0 662783123 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321054 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/16640/statm: 96277 5230 464 189 0 90626 0
[pid=16640/tid=16641] ppid=16638 vsize=385108 CPUtime=1.47 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 708 0 0 0 147 0 0 0 25 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670546 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=385108 CPUtime=1.46 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 633 0 0 0 146 0 0 0 20 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=385108 CPUtime=1.46 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 537 0 0 0 146 0 0 0 25 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=385108 CPUtime=1.46 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 524 0 0 0 146 0 0 0 19 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=385108 CPUtime=1.48 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 510 0 0 0 148 0 0 0 25 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4324936 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=385108 CPUtime=1.47 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 519 0 0 0 147 0 0 0 20 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4642509 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=385108 CPUtime=1.47 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 670 0 0 0 147 0 0 0 20 0 8 0 662783124 394350592 5230 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.83
Current children cumulated vsize (KiB) 385108

[startup+3.10087 s]
/proc/loadavg: 8.35 8.66 8.22 9/474 16648
/proc/meminfo: memFree=29296096/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=386040 CPUtime=24.58 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 6653 0 0 0 2451 7 0 0 21 0 8 0 662783123 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/16640/statm: 96510 6576 464 189 0 90859 0
[pid=16640/tid=16641] ppid=16638 vsize=386040 CPUtime=3.07 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 901 0 0 0 306 1 0 0 19 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=386040 CPUtime=3.06 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 806 0 0 0 306 0 0 0 25 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=386040 CPUtime=3.06 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 671 0 0 0 306 0 0 0 25 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4363370 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=386040 CPUtime=3.03 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 727 0 0 0 303 0 0 0 19 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=386040 CPUtime=3.07 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 666 0 0 0 307 0 0 0 25 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4660449 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=386040 CPUtime=3.07 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 669 0 0 0 306 1 0 0 25 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=386040 CPUtime=3.06 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 872 0 0 0 306 0 0 0 19 0 8 0 662783124 395304960 6576 33554432000 4194304 4964800 140734293911952 18446744073709551615 4365162 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.58
Current children cumulated vsize (KiB) 386040

[startup+6.30134 s]
/proc/loadavg: 8.32 8.65 8.22 9/474 16648
/proc/meminfo: memFree=29289664/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=386700 CPUtime=50.12 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 8245 0 0 0 5001 11 0 0 25 0 8 0 662783123 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4668568 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/16640/statm: 96675 8168 464 189 0 91024 0
[pid=16640/tid=16641] ppid=16638 vsize=386700 CPUtime=6.27 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 1161 0 0 0 626 1 0 0 25 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4379880 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=386700 CPUtime=6.26 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 987 0 0 0 626 0 0 0 25 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=386700 CPUtime=6.27 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 853 0 0 0 626 1 0 0 25 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=386700 CPUtime=6.18 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 1023 0 0 0 617 1 0 0 20 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=386700 CPUtime=6.28 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 782 0 0 0 627 1 0 0 22 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321182 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=386700 CPUtime=6.26 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 806 0 0 0 625 1 0 0 25 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=386700 CPUtime=6.27 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 1118 0 0 0 626 1 0 0 25 0 8 0 662783124 395980800 8168 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 4 0 0 0

################
# More data... #
################


[startup+4662.3 s]
/proc/loadavg: 8.09 8.08 8.05 9/474 16784
/proc/meminfo: memFree=29129716/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=37212.9 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47243 0 0 0 3719006 2289 0 0 25 0 8 0 662783123 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4653216 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/16640/statm: 103653 47166 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=4655.8 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5859 0 0 0 465257 323 0 0 25 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=4655.66 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7377 0 0 0 465276 290 0 0 18 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4630.56 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4270 0 0 0 462771 285 0 0 25 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=4656.36 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6475 0 0 0 465378 258 0 0 25 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 251813907418 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=4656.56 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6740 0 0 0 465376 280 0 0 25 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=4657.22 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5215 0 0 0 465435 287 0 0 25 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4660268 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=4652.78 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6730 0 0 0 464995 283 0 0 25 0 8 0 662783124 424562688 47166 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37212.9
Current children cumulated vsize (KiB) 414612

[startup+4722.3 s]
/proc/loadavg: 8.06 8.07 8.05 9/474 16786
/proc/meminfo: memFree=29129804/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=37691.9 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47294 0 0 0 3766875 2313 0 0 25 0 8 0 662783123 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670542 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/16640/statm: 103653 47217 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=4715.77 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5869 0 0 0 471251 326 0 0 25 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=4715.64 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7390 0 0 0 471271 293 0 0 25 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4690.54 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4293 0 0 0 468767 287 0 0 21 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4361315 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=4715.41 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6475 0 0 0 471279 262 0 0 25 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4683952 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=4716.54 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6741 0 0 0 471371 283 0 0 25 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=4717.2 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5217 0 0 0 471430 290 0 0 25 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4660080 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=4712.77 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6732 0 0 0 470992 285 0 0 25 0 8 0 662783124 424562688 47217 33554432000 4194304 4964800 140734293911952 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37691.9
Current children cumulated vsize (KiB) 414612

[startup+4782.3 s]
/proc/loadavg: 8.02 8.06 8.04 9/474 16788
/proc/meminfo: memFree=29129772/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=38170.9 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47329 0 0 0 3814753 2341 0 0 20 0 8 0 662783123 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670437 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/16640/statm: 103653 47252 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=4775.77 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5869 0 0 0 477246 331 0 0 21 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=4775.65 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7390 0 0 0 477269 296 0 0 25 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4750.56 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4315 0 0 0 474766 290 0 0 25 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=4774.48 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6475 0 0 0 477183 265 0 0 25 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=4776.52 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6742 0 0 0 477365 287 0 0 25 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=4777.2 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5218 0 0 0 477427 293 0 0 21 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=4772.78 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6742 0 0 0 476989 289 0 0 25 0 8 0 662783124 424562688 47252 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670373 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38170.9
Current children cumulated vsize (KiB) 414612

[startup+4842.3 s]
/proc/loadavg: 8.04 8.06 8.04 9/474 16789
/proc/meminfo: memFree=29129628/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=38649.7 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47346 0 0 0 3862601 2371 0 0 25 0 8 0 662783123 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670538 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/16640/statm: 103653 47269 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=4835.73 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5870 0 0 0 483238 335 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=4835.62 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7394 0 0 0 483261 301 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4810.43 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4315 0 0 0 480749 294 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4320890 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=4833.61 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6476 0 0 0 483093 268 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=4836.46 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6742 0 0 0 483356 290 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4660263 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=4837.17 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5218 0 0 0 483420 297 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=4832.75 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6753 0 0 0 482983 292 0 0 25 0 8 0 662783124 424562688 47269 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321243 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38649.7
Current children cumulated vsize (KiB) 414612

[startup+4902.32 s]
/proc/loadavg: 8.01 8.04 8.04 9/474 16791
/proc/meminfo: memFree=29129356/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=39128.8 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47427 0 0 0 3910477 2400 0 0 25 0 8 0 662783123 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4374902 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/16640/statm: 103653 47350 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=4895.71 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5873 0 0 0 489232 339 0 0 25 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=4895.61 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7395 0 0 0 489257 304 0 0 25 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4869.5 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4351 0 0 0 486652 298 0 0 20 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=4893.61 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6488 0 0 0 489088 273 0 0 20 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=4896.45 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6755 0 0 0 489351 294 0 0 25 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=4897.16 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5219 0 0 0 489417 299 0 0 19 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4627341 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=4892.75 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6768 0 0 0 488980 295 0 0 25 0 8 0 662783124 424562688 47350 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39128.8
Current children cumulated vsize (KiB) 414612

[startup+4962.3 s]
/proc/loadavg: 8.12 8.07 8.04 9/474 16792
/proc/meminfo: memFree=29129208/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=39607.6 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47471 0 0 0 3958332 2425 0 0 25 0 8 0 662783123 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670328 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/16640/statm: 103653 47394 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=4955.66 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5873 0 0 0 495224 342 0 0 25 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=4955.58 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7406 0 0 0 495251 307 0 0 25 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4929.27 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4352 0 0 0 492626 301 0 0 25 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4660104 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=4953.57 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6489 0 0 0 495082 275 0 0 25 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670313 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=4956.4 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6770 0 0 0 495343 297 0 0 19 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=4957.13 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5223 0 0 0 495411 302 0 0 25 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=4952.72 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6780 0 0 0 494974 298 0 0 25 0 8 0 662783124 424562688 47394 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670520 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39607.6
Current children cumulated vsize (KiB) 414612



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.5 s]
/proc/loadavg: 8.12 8.07 8.04 9/474 16794
/proc/meminfo: memFree=29128956/32950928 swapFree=41934240/67111528
[pid=16640] ppid=16638 vsize=414612 CPUtime=40000.4 cores=0-7
/proc/16640/stat : 16640 (claspmt) R 16638 16640 15355 0 -1 4202496 47519 0 0 0 3997593 2448 0 0 25 0 8 0 662783123 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670534 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/16640/statm: 103653 47442 464 189 0 98002 0
[pid=16640/tid=16641] ppid=16638 vsize=414612 CPUtime=5004.87 cores=0-7
/proc/16640/task/16641/stat : 16641 (claspmt) R 16638 16640 15355 0 -1 4202560 5885 0 0 0 500142 345 0 0 22 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=16640/tid=16642] ppid=16638 vsize=414612 CPUtime=5004.78 cores=0-7
/proc/16640/task/16642/stat : 16642 (claspmt) R 16638 16640 15355 0 -1 4202560 7408 0 0 0 500169 309 0 0 25 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=16640/tid=16643] ppid=16638 vsize=414612 CPUtime=4978.48 cores=0-7
/proc/16640/task/16643/stat : 16643 (claspmt) R 16638 16640 15355 0 -1 4202560 4363 0 0 0 497543 305 0 0 25 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=16640/tid=16644] ppid=16638 vsize=414612 CPUtime=5002.79 cores=0-7
/proc/16640/task/16644/stat : 16644 (claspmt) R 16638 16640 15355 0 -1 4202560 6489 0 0 0 500000 279 0 0 20 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 251813895521 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=16640/tid=16645] ppid=16638 vsize=414612 CPUtime=5005.6 cores=0-7
/proc/16640/task/16645/stat : 16645 (claspmt) R 16638 16640 15355 0 -1 4202560 6781 0 0 0 500259 301 0 0 20 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=16640/tid=16646] ppid=16638 vsize=414612 CPUtime=5006.32 cores=0-7
/proc/16640/task/16646/stat : 16646 (claspmt) R 16638 16640 15355 0 -1 4202560 5224 0 0 0 500328 304 0 0 19 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=16640/tid=16647] ppid=16638 vsize=414612 CPUtime=5001.23 cores=0-7
/proc/16640/task/16647/stat : 16647 (claspmt) R 16638 16640 15355 0 -1 4202560 6780 0 0 0 499822 301 0 0 19 0 8 0 662783124 424562688 47442 33554432000 4194304 4964800 140734293911952 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 414612

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 5011.54
CPU time (s): 40000.5
CPU user time (s): 39976
CPU system time (s): 24.4903
CPU usage (%): 798.167
Max. virtual memory (cumulated for all children) (KiB): 414612

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39976
system time used= 24.4903
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 47531
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3201485
involuntary context switches= 191774

runsolver used 23.1645 second user time and 54.2957 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-16 04:10:44
IDJOB=3381306
IDBENCH=87453
IDSOLVER=1865
FILE ID=node126/3381306-1305511844
RUNJOBID= node126-1305511372-15370
PBS_JOBID= 13325530
Free space on /tmp= 73404 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT02/handmade/pehoushek/plainoldcnf/5cnf_3900_3900_220.shuffled.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381306-1305511844/watcher-3381306-1305511844 -o /tmp/evaluation-result-3381306-1305511844/solver-3381306-1305511844 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381306-1305511844.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f0b7e0675915478b15cab3695fd0fd62
RANDOM SEED=1863748104

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      29320356 kB
Buffers:        208352 kB
Cached:        1954656 kB
SwapCached:      49688 kB
Active:         993528 kB
Inactive:      2365596 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      29320356 kB
SwapTotal:    67111528 kB
SwapFree:     41934240 kB
Dirty:             788 kB
Writeback:           0 kB
AnonPages:     1159384 kB
Mapped:          17448 kB
Slab:           114152 kB
PageTables:      86908 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 133950028 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73400 MiB
End job on node126 at 2011-05-16 05:34:18