Trace number 3322413

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 34314.5 5100.35

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-6x40.cnf
MD5SUM3a7a68cedc6ab84c0df9c27d166b0b6b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables960
Number of clauses48480
Sum of the clauses size191040
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21440
Number of clauses of size 30
Number of clauses of size 447040
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...384364/instance-3322413-1304384364.cnf
0.79/0.83	c Solving...
34313.54/5100.11	
34313.54/5100.11	*** INTERRUPTED! ***
34313.54/5100.11	*** Shutting down threads...
34314.40/5100.34	s UNKNOWN
34314.40/5100.34	
34314.40/5100.34	c INTERRUPTED   : 1
34314.40/5100.34	c Models        : 0+    
34314.40/5100.34	c Time          : 5100.107s (Solving: 5099.28s 1st Model: 0.00s Unsat: 0.00s)
34314.40/5100.34	c CPU Time      : 34314.210s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322413-1304384364/watcher-3322413-1304384364 -o /tmp/evaluation-result-3322413-1304384364/solver-3322413-1304384364 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322413-1304384364.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.06 10.08 10.02 4/349 30836
/proc/meminfo: memFree=19742196/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=22888 CPUtime=0 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 1085 0 0 0 0 0 0 0 23 0 1 0 550034143 23437312 917 33554432000 4194304 4964704 140737028370784 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30836/statm: 5722 918 386 189 0 596 0

[startup+0.0405909 s]
/proc/loadavg: 10.06 10.08 10.02 4/349 30836
/proc/meminfo: memFree=19742196/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=25316 CPUtime=0.03 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 1769 0 0 0 3 0 0 0 23 0 1 0 550034143 25923584 1512 33554432000 4194304 4964704 140737028370784 18446744073709551615 4607554 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30836/statm: 6329 1512 399 189 0 1203 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 25316

[startup+0.100579 s]
/proc/loadavg: 10.06 10.08 10.02 4/349 30836
/proc/meminfo: memFree=19742196/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=25316 CPUtime=0.09 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 1769 0 0 0 9 0 0 0 23 0 1 0 550034143 25923584 1512 33554432000 4194304 4964704 140737028370784 18446744073709551615 4607486 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30836/statm: 6329 1512 399 189 0 1203 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 25316

[startup+0.300542 s]
/proc/loadavg: 10.06 10.08 10.02 4/349 30836
/proc/meminfo: memFree=19742196/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=27032 CPUtime=0.29 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 2205 0 0 0 29 0 0 0 25 0 1 0 550034143 27680768 1948 33554432000 4194304 4964704 140737028370784 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30836/statm: 6758 1948 400 189 0 1632 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 27032

[startup+0.700471 s]
/proc/loadavg: 10.06 10.08 10.02 4/349 30836
/proc/meminfo: memFree=19742196/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=28880 CPUtime=0.69 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 2670 0 0 0 69 0 0 0 25 0 1 0 550034143 29573120 2413 33554432000 4194304 4964704 140737028370784 18446744073709551615 4607585 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/30836/statm: 7220 2413 400 189 0 2094 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 28880

[startup+1.50129 s]
/proc/loadavg: 10.06 10.08 10.02 11/357 30844
/proc/meminfo: memFree=19702328/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=527008 CPUtime=4.57 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 17602 0 0 0 448 9 0 0 18 0 8 0 550034143 539656192 17208 33554432000 4194304 4964704 140737028370784 18446744073709551615 4660059 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 131752 17209 464 189 0 126101 0
[pid=30836/tid=30838] ppid=30834 vsize=527008 CPUtime=0.51 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 1857 0 0 0 50 1 0 0 18 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=527008 CPUtime=0.57 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 2393 0 0 0 56 1 0 0 17 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905392161 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=527008 CPUtime=0.44 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 1651 0 0 0 44 0 0 0 18 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=527008 CPUtime=0.47 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 2089 0 0 0 46 1 0 0 18 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=527008 CPUtime=0.49 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 1614 0 0 0 48 1 0 0 18 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=527008 CPUtime=0.36 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 1648 0 0 0 35 1 0 0 18 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=527008 CPUtime=0.37 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 1953 0 0 0 37 0 0 0 17 0 8 0 550034227 539656192 17209 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.57
Current children cumulated vsize (KiB) 527008

[startup+3.10101 s]
/proc/loadavg: 9.98 10.06 10.01 11/357 30844
/proc/meminfo: memFree=19662400/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=530032 CPUtime=14.07 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 24305 0 0 0 1395 12 0 0 19 0 8 0 550034143 542752768 23911 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 132508 23911 466 189 0 126857 0
[pid=30836/tid=30838] ppid=30834 vsize=530032 CPUtime=1.83 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 2570 0 0 0 182 1 0 0 19 0 8 0 550034227 542752768 23911 33554432000 4194304 4964704 140737028370784 18446744073709551615 4660310 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=530032 CPUtime=1.85 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 3317 0 0 0 184 1 0 0 18 0 8 0 550034227 542752768 23911 33554432000 4194304 4964704 140737028370784 18446744073709551615 4321099 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=530032 CPUtime=1.93 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 2530 0 0 0 192 1 0 0 19 0 8 0 550034227 542752768 23911 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=530032 CPUtime=1.35 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 2599 0 0 0 134 1 0 0 19 0 8 0 550034227 542752768 23912 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905392161 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=530032 CPUtime=1.59 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 2053 0 0 0 158 1 0 0 19 0 8 0 550034227 542752768 23912 33554432000 4194304 4964704 140737028370784 18446744073709551615 4658341 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=530032 CPUtime=1.63 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 2624 0 0 0 162 1 0 0 18 0 8 0 550034227 542752768 23912 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=530032 CPUtime=1.37 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 3807 0 0 0 135 2 0 0 19 0 8 0 550034227 542752768 23912 33554432000 4194304 4964704 140737028370784 18446744073709551615 4627601 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 14.07
Current children cumulated vsize (KiB) 530032

[startup+6.30043 s]
/proc/loadavg: 9.98 10.06 10.01 11/357 30844
/proc/meminfo: memFree=19623568/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=531508 CPUtime=33.2 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 31764 0 0 0 3303 17 0 0 24 0 8 0 550034143 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 132877 31370 466 189 0 127226 0
[pid=30836/tid=30838] ppid=30834 vsize=531508 CPUtime=3.63 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 3428 0 0 0 362 1 0 0 23 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=531508 CPUtime=3.82 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 3923 0 0 0 380 2 0 0 22 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4668850 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=531508 CPUtime=4.54 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 4318 0 0 0 452 2 0 0 25 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=531508 CPUtime=4.48 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 3931 0 0 0 447 1 0 0 25 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=531508 CPUtime=3.94 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 2267 0 0 0 393 1 0 0 19 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905392161 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=531508 CPUtime=4.23 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 3267 0 0 0 422 1 0 0 25 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4321272 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=531508 CPUtime=3.99 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 5330 0 0 0 396 3 0 0 25 0 8 0 550034227 544264192 31370 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 33.2
Current children cumulated vsize (KiB) 531508

[startup+12.7013 s]
/proc/loadavg: 9.98 10.06 10.01 11/357 30844
/proc/meminfo: memFree=19570556/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=537516 CPUtime=71.38 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 45393 0 0 0 7114 24 0 0 25 0 8 0 550034143 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30836/statm: 134379 44999 466 189 0 128728 0
[pid=30836/tid=30838] ppid=30834 vsize=537516 CPUtime=6.14 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 4645 0 0 0 612 2 0 0 25 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=537516 CPUtime=7.84 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 5037 0 0 0 782 2 0 0 25 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=537516 CPUtime=10.86 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 5772 0 0 0 1083 3 0 0 19 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=537516 CPUtime=10.38 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 5198 0 0 0 1036 2 0 0 19 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=537516 CPUtime=9.13 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 4101 0 0 0 911 2 0 0 25 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905836014 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=537516 CPUtime=8.09 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 5456 0 0 0 806 3 0 0 25 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4658344 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=537516 CPUtime=10.08 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 8377 0 0 0 1003 5 0 0 21 0 8 0 550034227 550416384 44999 33554432000 4194304 4964704 140737028370784 18446744073709551615 4365094 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 71.38
Current children cumulated vsize (KiB) 537516

[startup+25.501 s]
/proc/loadavg: 10.06 10.07 10.01 11/357 30845
/proc/meminfo: memFree=19483228/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=551952 CPUtime=147.76 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 66702 0 0 0 14741 35 0 0 25 0 8 0 550034143 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670458 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 137988 65856 466 189 0 132337 0
[pid=30836/tid=30838] ppid=30834 vsize=551952 CPUtime=16.05 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 8097 0 0 0 1601 4 0 0 25 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 4627006 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=551952 CPUtime=16.01 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 7867 0 0 0 1597 4 0 0 25 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=551952 CPUtime=22.06 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 7884 0 0 0 2202 4 0 0 25 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905428622 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=551952 CPUtime=20.1 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 8442 0 0 0 2007 3 0 0 25 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 4362375 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=551952 CPUtime=17.87 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 7658 0 0 0 1784 3 0 0 25 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 4627168 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=551952 CPUtime=16.12 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 8383 0 0 0 1607 5 0 0 25 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905392161 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=551952 CPUtime=19.39 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 10594 0 0 0 1933 6 0 0 24 0 8 0 550034227 565198848 65856 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 147.76
Current children cumulated vsize (KiB) 551952

[startup+51.1003 s]
/proc/loadavg: 10.12 10.08 10.02 11/357 30845
/proc/meminfo: memFree=19420348/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=556540 CPUtime=301.01 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 82033 0 0 0 30055 46 0 0 19 0 8 0 550034143 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670215 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 139135 81168 466 189 0 133484 0
[pid=30836/tid=30838] ppid=30834 vsize=556540 CPUtime=32.43 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 8938 0 0 0 3239 4 0 0 25 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=556540 CPUtime=31.51 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 9425 0 0 0 3146 5 0 0 25 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4361097 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=556540 CPUtime=47.2 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 9113 0 0 0 4715 5 0 0 25 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=556540 CPUtime=40.17 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 10147 0 0 0 4013 4 0 0 25 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=556540 CPUtime=40.13 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 10086 0 0 0 4008 5 0 0 19 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320916 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=556540 CPUtime=31.94 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 9404 0 0 0 3189 5 0 0 25 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=556540 CPUtime=37 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 14159 0 0 0 3691 9 0 0 19 0 8 0 550034227 569896960 81168 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 301.01

################
# More data... #
################

/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 50818 0 0 0 386945 67 0 0 25 0 8 0 550034227 1885200384 382278 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1841016 CPUtime=4106.57 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 59410 0 0 0 410587 70 0 0 25 0 8 0 550034227 1885200384 382278 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 32093.5
Current children cumulated vsize (KiB) 1841016

[startup+4842.3 s]
/proc/loadavg: 9.04 9.06 9.07 10/356 31495
/proc/meminfo: memFree=18283308/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1906552 CPUtime=32512.8 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 397449 0 0 0 3250784 491 0 0 25 0 8 0 550034143 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670193 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/30836/statm: 476638 385006 466 189 0 470987 0
[pid=30836/tid=30838] ppid=30834 vsize=1906552 CPUtime=3767.08 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 35039 0 0 0 376660 48 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=1906552 CPUtime=4159.28 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 84862 0 0 0 415840 88 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=1906552 CPUtime=4130.81 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 41014 0 0 0 413022 59 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=1906552 CPUtime=4389.6 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 38149 0 0 0 438918 42 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=1906552 CPUtime=4194.94 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 39484 0 0 0 419444 50 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=1906552 CPUtime=3930.11 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 52019 0 0 0 392943 68 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1906552 CPUtime=4166.55 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 59507 0 0 0 416585 70 0 0 25 0 8 0 550034227 1952309248 385006 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 32512.8
Current children cumulated vsize (KiB) 1906552

[startup+4902.3 s]
/proc/loadavg: 9.06 9.06 9.07 10/356 31497
/proc/meminfo: memFree=18281788/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1906552 CPUtime=32932 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 397851 0 0 0 3292707 493 0 0 25 0 8 0 550034143 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/30836/statm: 476638 385334 466 189 0 470987 0
[pid=30836/tid=30838] ppid=30834 vsize=1906552 CPUtime=3805.36 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 35113 0 0 0 380488 48 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=1906552 CPUtime=4219.02 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 84862 0 0 0 421814 88 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=1906552 CPUtime=4190.81 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 41095 0 0 0 419022 59 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=1906552 CPUtime=4446.79 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 38335 0 0 0 444637 42 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=1906552 CPUtime=4250.54 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 39484 0 0 0 425004 50 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=1906552 CPUtime=3981.97 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 52020 0 0 0 398128 69 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1906552 CPUtime=4217.63 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 59560 0 0 0 421693 70 0 0 25 0 8 0 550034227 1952309248 385334 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 32932
Current children cumulated vsize (KiB) 1906552

[startup+4962.3 s]
/proc/loadavg: 9.02 9.05 9.06 10/356 31498
/proc/meminfo: memFree=18281140/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1907244 CPUtime=33351.2 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 398040 0 0 0 3334629 496 0 0 25 0 8 0 550034143 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/30836/statm: 476811 385484 466 189 0 471160 0
[pid=30836/tid=30838] ppid=30834 vsize=1907244 CPUtime=3850.94 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 35114 0 0 0 385046 48 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=1907244 CPUtime=4276.99 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 84902 0 0 0 427610 89 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=1907244 CPUtime=4246.45 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 41095 0 0 0 424586 59 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=1907244 CPUtime=4499.88 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 38473 0 0 0 449945 43 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=1907244 CPUtime=4310.53 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 39494 0 0 0 431003 50 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=1907244 CPUtime=4029.17 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 52020 0 0 0 402848 69 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670431 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1907244 CPUtime=4265.92 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 59560 0 0 0 426521 71 0 0 25 0 8 0 550034227 1953017856 385484 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 33351.2
Current children cumulated vsize (KiB) 1907244

[startup+5022.3 s]
/proc/loadavg: 9.04 9.05 9.06 10/356 31500
/proc/meminfo: memFree=18270556/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1907244 CPUtime=33770.5 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 400759 0 0 0 3376550 502 0 0 25 0 8 0 550034143 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670215 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 476811 388129 466 189 0 471160 0
[pid=30836/tid=30838] ppid=30834 vsize=1907244 CPUtime=3887.1 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 35231 0 0 0 388661 49 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=1907244 CPUtime=4336.49 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 86055 0 0 0 433558 91 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=1907244 CPUtime=4301.28 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 41730 0 0 0 430067 61 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=1907244 CPUtime=4553.2 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 38511 0 0 0 455277 43 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=1907244 CPUtime=4370.54 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 39691 0 0 0 437003 51 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=1907244 CPUtime=4089.16 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 52099 0 0 0 408847 69 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1907244 CPUtime=4325.9 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 60046 0 0 0 432519 71 0 0 25 0 8 0 550034227 1953017856 388129 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 33770.5
Current children cumulated vsize (KiB) 1907244

[startup+5082.3 s]
/proc/loadavg: 9.05 9.05 9.06 10/356 31501
/proc/meminfo: memFree=18267932/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1907244 CPUtime=34189.8 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 401487 0 0 0 3418471 507 0 0 25 0 8 0 550034143 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4659143 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/30836/statm: 476811 388783 466 189 0 471160 0
[pid=30836/tid=30838] ppid=30834 vsize=1907244 CPUtime=3942.07 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 35330 0 0 0 394158 49 0 0 25 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=1907244 CPUtime=4382.06 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 86071 0 0 0 438115 91 0 0 25 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=1907244 CPUtime=4359.93 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 41743 0 0 0 435932 61 0 0 25 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=1907244 CPUtime=4612.85 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 38511 0 0 0 461242 43 0 0 25 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=1907244 CPUtime=4429.44 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 40140 0 0 0 442892 52 0 0 20 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=1907244 CPUtime=4125.07 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 52099 0 0 0 412437 70 0 0 25 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1907244 CPUtime=4376.32 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 60062 0 0 0 437560 72 0 0 25 0 8 0 550034227 1953017856 388783 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 34189.8
Current children cumulated vsize (KiB) 1907244



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.10 9.07 9.06 10/356 31502
/proc/meminfo: memFree=18267928/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1907244 CPUtime=34313.5 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 401496 0 0 0 3430847 507 0 0 25 0 8 0 550034143 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4659143 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/30836/statm: 476811 388792 466 189 0 471160 0
[pid=30836/tid=30838] ppid=30834 vsize=1907244 CPUtime=3957.78 cores=0-7
/proc/30836/task/30838/stat : 30838 (claspmt) R 30834 30836 28735 0 -1 4202560 35330 0 0 0 395729 49 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=30836/tid=30839] ppid=30834 vsize=1907244 CPUtime=4399.78 cores=0-7
/proc/30836/task/30839/stat : 30839 (claspmt) R 30834 30836 28735 0 -1 4202560 86080 0 0 0 439887 91 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=30836/tid=30840] ppid=30834 vsize=1907244 CPUtime=4377.64 cores=0-7
/proc/30836/task/30840/stat : 30840 (claspmt) R 30834 30836 28735 0 -1 4202560 41743 0 0 0 437703 61 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=30836/tid=30841] ppid=30834 vsize=1907244 CPUtime=4630.56 cores=0-7
/proc/30836/task/30841/stat : 30841 (claspmt) R 30834 30836 28735 0 -1 4202560 38511 0 0 0 463013 43 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=30836/tid=30842] ppid=30834 vsize=1907244 CPUtime=4444.27 cores=0-7
/proc/30836/task/30842/stat : 30842 (claspmt) R 30834 30836 28735 0 -1 4202560 40140 0 0 0 444375 52 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=30836/tid=30843] ppid=30834 vsize=1907244 CPUtime=4135.57 cores=0-7
/proc/30836/task/30843/stat : 30843 (claspmt) R 30834 30836 28735 0 -1 4202560 52099 0 0 0 413487 70 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=30836/tid=30844] ppid=30834 vsize=1907244 CPUtime=4394.03 cores=0-7
/proc/30836/task/30844/stat : 30844 (claspmt) R 30834 30836 28735 0 -1 4202560 60062 0 0 0 439331 72 0 0 25 0 8 0 550034227 1953017856 388792 33554432000 4194304 4964704 140737028370784 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 34313.5
Current children cumulated vsize (KiB) 1907244

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.2 s]
/proc/loadavg: 9.10 9.07 9.06 10/356 31502
/proc/meminfo: memFree=18267928/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1905620 CPUtime=34314.3 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 401507 0 0 0 3430923 507 0 0 25 0 1 0 550034143 1951354880 386921 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905386337 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/30836/statm: 476405 386921 476 189 0 470754 0
Current children cumulated CPU time (s) 34314.3
Current children cumulated vsize (KiB) 1905620

[startup+5100.3 s]
/proc/loadavg: 9.10 9.07 9.06 10/356 31502
/proc/meminfo: memFree=18267928/32951124 swapFree=67111356/67111528
[pid=30836] ppid=30834 vsize=1323312 CPUtime=34314.4 cores=0-7
/proc/30836/stat : 30836 (claspmt) R 30834 30836 28735 0 -1 4202496 401507 0 0 0 3430931 509 0 0 25 0 1 0 550034143 1355071488 266705 33554432000 4194304 4964704 140737028370784 18446744073709551615 218905774215 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/30836/statm: 330828 266195 476 189 0 325177 0
Current children cumulated CPU time (s) 34314.4
Current children cumulated vsize (KiB) 1323312

Child status: 0
Real time (s): 5100.35
CPU time (s): 34314.5
CPU user time (s): 34309.3
CPU system time (s): 5.13722
CPU usage (%): 672.787
Max. virtual memory (cumulated for all children) (KiB): 1907244

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 34309.3
system time used= 5.13722
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 401508
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 257096
involuntary context switches= 307531

runsolver used 17.1714 second user time and 43.1254 second system time

The end

Launcher Data 

Begin job on node137 at 2011-05-03 02:59:24
IDJOB=3322413
IDBENCH=83178
IDSOLVER=1711
FILE ID=node137/3322413-1304384364
RUNJOBID= node137-1304366465-28749
PBS_JOBID= 13181736
Free space on /tmp= 73596 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-6x40.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322413-1304384364/watcher-3322413-1304384364 -o /tmp/evaluation-result-3322413-1304384364/solver-3322413-1304384364 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322413-1304384364.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3a7a68cedc6ab84c0df9c27d166b0b6b
RANDOM SEED=1457692993

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19742476 kB
Buffers:        968360 kB
Cached:        2695508 kB
SwapCached:         48 kB
Active:        9598416 kB
Inactive:      3104000 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19742476 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            2132 kB
Writeback:           0 kB
AnonPages:     9038404 kB
Mapped:          29320 kB
Slab:           400184 kB
PageTables:      38816 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 76242244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73604 MiB
End job on node137 at 2011-05-03 04:24:26