Trace number 3321347

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 5103.62 5100.99

General information on the benchmark

Nameindustrial/vliw_sat_4.0/
9vliw_m_9stages_iq3_C1_bug5.cnf
MD5SUMc77a56af24208a6401c763e11a38de01
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark6.52301
Satisfiable
(Un)Satisfiability was proved
Number of variables520770
Number of clauses13380350
Sum of the clauses size39210884
Maximum clause length6689
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 28710587
Number of clauses of size 34215482
Number of clauses of size 425225
Number of clauses of size 515593
Number of clauses of size over 5413462

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...336662/instance-3321347-1304336662.cnf
4.59/4.60	c Solving...
5102.64/5100.01	
5102.64/5100.01	*** INTERRUPTED! ***
5102.64/5100.01	*** Shutting down threads...
5103.52/5100.97	s UNKNOWN
5103.52/5100.97	
5103.52/5100.97	c INTERRUPTED   : 1
5103.52/5100.97	c Models        : 0+    
5103.52/5100.97	c Time          : 5100.015s (Solving: 5095.41s 1st Model: 0.00s Unsat: 0.00s)
5103.52/5100.97	c CPU Time      : 5102.650s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321347-1304336662/watcher-3321347-1304336662 -o /tmp/evaluation-result-3321347-1304336662/solver-3321347-1304336662 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321347-1304336662.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.97 7.81 7.69 1/238 11668
/proc/meminfo: memFree=28056620/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=87000 CPUtime=0 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 1132 0 0 0 0 0 0 0 24 0 1 0 623897722 89088000 1063 33554432000 4194304 4964704 140737165104944 18446744073709551615 4700332 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 21750 1069 376 189 0 16624 0

[startup+0.059121 s]
/proc/loadavg: 6.97 7.81 7.69 1/238 11668
/proc/meminfo: memFree=28056620/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=117520 CPUtime=0.05 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 20648 0 0 0 3 2 0 0 24 0 1 0 623897722 120340480 19747 33554432000 4194304 4964704 140737165104944 18446744073709551615 4679071 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 29380 19756 377 189 0 24254 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 117520

[startup+0.101112 s]
/proc/loadavg: 6.97 7.81 7.69 1/238 11668
/proc/meminfo: memFree=28056620/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=125144 CPUtime=0.09 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 30134 0 0 0 6 3 0 0 24 0 1 0 623897722 128147456 25996 33554432000 4194304 4964704 140737165104944 18446744073709551615 4567844 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 31286 25997 388 189 0 26160 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 125144

[startup+0.301075 s]
/proc/loadavg: 6.97 7.81 7.69 1/238 11668
/proc/meminfo: memFree=28056620/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=139940 CPUtime=0.29 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 33983 0 0 0 26 3 0 0 25 0 1 0 623897722 143298560 29687 33554432000 4194304 4964704 140737165104944 18446744073709551615 260020395776 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 34985 29688 388 189 0 29859 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 139940

[startup+0.701002 s]
/proc/loadavg: 6.97 7.81 7.69 1/238 11668
/proc/meminfo: memFree=28056620/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=166420 CPUtime=0.69 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 40849 0 0 0 63 6 0 0 25 0 1 0 623897722 170414080 36274 33554432000 4194304 4964704 140737165104944 18446744073709551615 260020395776 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 41605 36275 388 189 0 36479 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 166420

[startup+1.8748 s]
/proc/loadavg: 6.97 7.81 7.69 1/238 11668
/proc/meminfo: memFree=28056620/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=243048 CPUtime=1.86 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 60685 0 0 0 174 12 0 0 25 0 1 0 623897722 248881152 54919 33554432000 4194304 4964704 140737165104944 18446744073709551615 4581705 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 60762 54919 388 189 0 55636 0
Current children cumulated CPU time (s) 1.86
Current children cumulated vsize (KiB) 243048

[startup+3.10056 s]
/proc/loadavg: 6.97 7.81 7.69 2/239 11669
/proc/meminfo: memFree=27762100/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=323556 CPUtime=3.1 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 81249 0 0 0 293 17 0 0 25 0 1 0 623897722 331321344 74638 33554432000 4194304 4964704 140737165104944 18446744073709551615 260020056773 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/11668/statm: 80889 74638 390 189 0 75763 0
Current children cumulated CPU time (s) 3.1
Current children cumulated vsize (KiB) 323556

[startup+6.30097 s]
/proc/loadavg: 6.49 7.69 7.65 7/246 11676
/proc/meminfo: memFree=27092328/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1492504 CPUtime=8.77 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 341446 0 0 0 726 151 0 0 18 0 8 0 623897722 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 373126 295236 434 189 0 367475 0
[pid=11668/tid=11670] ppid=11666 vsize=1492504 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1492504 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1492504 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1492504 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1492504 CPUtime=1.14 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 34554 0 0 0 95 19 0 0 18 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1492504 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1492504 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1528324096 295236 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.77
Current children cumulated vsize (KiB) 1492504

[startup+12.7008 s]
/proc/loadavg: 6.05 7.58 7.62 2/246 11676
/proc/meminfo: memFree=26878068/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1492504 CPUtime=15.17 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 341609 0 0 0 1366 151 0 0 18 0 8 0 623897722 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 373126 295399 436 189 0 367475 0
[pid=11668/tid=11670] ppid=11666 vsize=1492504 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1492504 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1492504 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1492504 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1492504 CPUtime=7.54 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 34717 0 0 0 735 19 0 0 25 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 4668931 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1492504 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1492504 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1528324096 295399 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 15.17
Current children cumulated vsize (KiB) 1492504

[startup+25.5004 s]
/proc/loadavg: 4.93 7.26 7.51 2/246 11677
/proc/meminfo: memFree=26876396/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=27.97 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 342187 0 0 0 2646 151 0 0 18 0 8 0 623897722 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 295928 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=20.34 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 35295 0 0 0 2015 19 0 0 25 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 4399598 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 295928 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 27.97
Current children cumulated vsize (KiB) 1558040

[startup+51.1007 s]
/proc/loadavg: 3.71 6.79 7.35 2/246 11677
/proc/meminfo: memFree=26875744/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=53.57 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 342439 0 0 0 5205 152 0 0 18 0 8 0 623897722 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 296180 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=45.93 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 35547 0 0 0 4574 19 0 0 25 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 4399606 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 296180 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 53.57
Current children cumulated vsize (KiB) 1558040

[startup+102.301 s]
/proc/loadavg: 2.17 5.89 7.01 2/246 11679
/proc/meminfo: memFree=26874212/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=104.77 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 342804 0 0 0 10325 152 0 0 18 0 8 0 623897722 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 296545 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=97.13 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 35912 0 0 0 9694 19 0 0 25 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 4677956 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 296545 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 104.77
Current children cumulated vsize (KiB) 1558040

[startup+162.301 s]
/proc/loadavg: 1.43 5.00 6.63 2/246 11680
/proc/meminfo: memFree=26873680/32951124 swapFree=67111464/67111528

################
# More data... #
################

[pid=11668] ppid=11666 vsize=1558040 CPUtime=4844.92 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 349023 0 0 0 484335 157 0 0 18 0 8 0 623897722 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 302721 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=4837.29 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 42131 0 0 0 483704 25 0 0 25 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 4399504 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4844.92
Current children cumulated vsize (KiB) 1558040

[startup+4902.3 s]
/proc/loadavg: 1.02 1.01 1.00 2/246 11828
/proc/meminfo: memFree=26846388/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=4904.92 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 349023 0 0 0 490335 157 0 0 18 0 8 0 623897722 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 302721 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=4897.29 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 42131 0 0 0 489704 25 0 0 25 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 4668508 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4904.92
Current children cumulated vsize (KiB) 1558040

[startup+4962.3 s]
/proc/loadavg: 1.00 1.00 1.00 2/246 11829
/proc/meminfo: memFree=26846464/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=4964.92 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 349023 0 0 0 496335 157 0 0 18 0 8 0 623897722 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 302721 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=4957.29 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 42131 0 0 0 495704 25 0 0 25 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 4399292 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4964.92
Current children cumulated vsize (KiB) 1558040

[startup+5022.3 s]
/proc/loadavg: 1.00 1.00 1.00 2/246 11831
/proc/meminfo: memFree=26846432/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=5024.93 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 349023 0 0 0 502336 157 0 0 18 0 8 0 623897722 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 302721 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=5017.3 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 42131 0 0 0 501705 25 0 0 25 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5024.93
Current children cumulated vsize (KiB) 1558040

[startup+5082.3 s]
/proc/loadavg: 1.00 1.00 1.00 2/246 11832
/proc/meminfo: memFree=26846384/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=5084.93 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 349023 0 0 0 508336 157 0 0 18 0 8 0 623897722 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 302721 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=5077.3 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 42131 0 0 0 507705 25 0 0 25 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 4677965 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5084.93
Current children cumulated vsize (KiB) 1558040



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 1.00 1.00 1.00 2/246 11833
/proc/meminfo: memFree=26846240/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1558040 CPUtime=5102.64 cores=0-7
/proc/11668/stat : 11668 (claspmt) S 11666 11668 11573 0 -1 4202496 349023 0 0 0 510107 157 0 0 18 0 8 0 623897722 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11668/statm: 389510 302721 436 189 0 383859 0
[pid=11668/tid=11670] ppid=11666 vsize=1558040 CPUtime=0.19 cores=0-7
/proc/11668/task/11670/stat : 11670 (claspmt) S 11666 11668 11573 0 -1 4202560 27522 0 0 0 4 15 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 6 0 0 0
[pid=11668/tid=11671] ppid=11666 vsize=1558040 CPUtime=0.43 cores=0-7
/proc/11668/task/11671/stat : 11671 (claspmt) S 11666 11668 11573 0 -1 4202560 34492 0 0 0 26 17 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 4 0 0 0
[pid=11668/tid=11672] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11672/stat : 11672 (claspmt) S 11666 11668 11573 0 -1 4202560 34392 0 0 0 48 18 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 0 0 0 0
[pid=11668/tid=11673] ppid=11666 vsize=1558040 CPUtime=0.38 cores=0-7
/proc/11668/task/11673/stat : 11673 (claspmt) S 11666 11668 11573 0 -1 4202560 27659 0 0 0 22 16 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 7 0 0 0
[pid=11668/tid=11674] ppid=11666 vsize=1558040 CPUtime=5095.01 cores=0-7
/proc/11668/task/11674/stat : 11674 (claspmt) R 11666 11668 11573 0 -1 4202560 42131 0 0 0 509476 25 0 0 25 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11668/tid=11675] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11675/stat : 11675 (claspmt) S 11666 11668 11573 0 -1 4202560 31688 0 0 0 49 17 0 0 18 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 3 0 0 0
[pid=11668/tid=11676] ppid=11666 vsize=1558040 CPUtime=0.66 cores=0-7
/proc/11668/task/11676/stat : 11676 (claspmt) S 11666 11668 11573 0 -1 4202560 32024 0 0 0 44 22 0 0 17 0 8 0 623898183 1595432960 302721 33554432000 4194304 4964704 140737165104944 18446744073709551615 260032211865 0 0 4096 18946 18446604446222107584 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5102.64
Current children cumulated vsize (KiB) 1558040

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.4 s]
/proc/loadavg: 1.00 1.00 1.00 2/246 11833
/proc/meminfo: memFree=26846240/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1443604 CPUtime=5103.03 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 349033 0 0 0 510145 158 0 0 18 0 1 0 623897722 1478250496 272760 33554432000 4194304 4964704 140737165104944 18446744073709551615 260020044914 0 0 4096 18946 0 0 0 17 7 0 0 0
/proc/11668/statm: 360901 272760 446 189 0 355250 0
Current children cumulated CPU time (s) 5103.03
Current children cumulated vsize (KiB) 1443604

[startup+5100.8 s]
/proc/loadavg: 1.00 1.00 1.00 2/240 11834
/proc/meminfo: memFree=27040100/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=1149348 CPUtime=5103.42 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 349033 0 0 0 510183 159 0 0 18 0 1 0 623897722 1176932352 219002 33554432000 4194304 4964704 140737165104944 18446744073709551615 260020045192 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/11668/statm: 287337 219002 446 189 0 281686 0
Current children cumulated CPU time (s) 5103.42
Current children cumulated vsize (KiB) 1149348

[startup+5100.9 s]
/proc/loadavg: 1.00 1.00 1.00 2/240 11834
/proc/meminfo: memFree=27040100/32951124 swapFree=67111464/67111528
[pid=11668] ppid=11666 vsize=926312 CPUtime=5103.52 cores=0-7
/proc/11668/stat : 11668 (claspmt) R 11666 11668 11573 0 -1 4202496 349033 0 0 0 510190 162 0 0 18 0 1 0 623897722 948543488 93739 33554432000 4194304 4964704 140737165104944 18446744073709551615 260020051067 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/11668/statm: 231578 93739 446 189 0 225927 0
Current children cumulated CPU time (s) 5103.52
Current children cumulated vsize (KiB) 926312

Child status: 20
Real time (s): 5100.99
CPU time (s): 5103.62
CPU user time (s): 5101.98
CPU system time (s): 1.64475
CPU usage (%): 100.052
Max. virtual memory (cumulated for all children) (KiB): 1558040

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5101.98
system time used= 1.64475
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 349034
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 153697
involuntary context switches= 12307

runsolver used 10.0265 second user time and 27.0789 second system time

The end

Launcher Data 

Begin job on node110 at 2011-05-02 13:44:31
IDJOB=3321347
IDBENCH=20582
IDSOLVER=1711
FILE ID=node110/3321347-1304336662
RUNJOBID= node110-1304335483-11587
PBS_JOBID= 13178013
Free space on /tmp= 73632 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT07/industrial/vliw_sat_4.0/9vliw_m_9stages_iq3_C1_bug5.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321347-1304336662/watcher-3321347-1304336662 -o /tmp/evaluation-result-3321347-1304336662/solver-3321347-1304336662 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321347-1304336662.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c77a56af24208a6401c763e11a38de01
RANDOM SEED=1444774495

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28056900 kB
Buffers:       1049104 kB
Cached:        1828144 kB
SwapCached:         64 kB
Active:        2313956 kB
Inactive:      2144112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28056900 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:          308548 kB
Writeback:           0 kB
AnonPages:     1581220 kB
Mapped:          20904 kB
Slab:           359576 kB
PageTables:      14628 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 34652688 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73632 MiB
End job on node110 at 2011-05-02 15:09:33