Trace number 3381031

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.6 5009.57

General information on the benchmark

Namecrafted/skvortsov/automata-synchronization/
crn_40_1521_s.cnf
MD5SUM4e35cb9894b1ca841d462da2d6957cc5
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables63882
Number of clauses123981
Sum of the clauses size369562
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22381
Number of clauses of size 3121600
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...500809/instance-3381031-1305500809.cnf
0.49/0.54	c Solving...
40000.64/5008.71	
40000.64/5008.71	*** INTERRUPTED! ***
40000.64/5008.71	*** Shutting down threads...
40001.50/5009.54	s UNKNOWN
40001.50/5009.54	
40001.50/5009.54	c INTERRUPTED   : 1
40001.50/5009.54	c Models        : 0+    
40001.50/5009.54	c Time          : 5008.717s (Solving: 5008.18s 1st Model: 0.00s Unsat: 0.00s)
40001.50/5009.54	c CPU Time      : 40000.710s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381031-1305500809/watcher-3381031-1305500809 -o /tmp/evaluation-result-3381031-1305500809/solver-3381031-1305500809 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381031-1305500809.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.48 7.94 7.97 1/348 11056
/proc/meminfo: memFree=19355664/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=29052 CPUtime=0 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 2377 0 0 0 0 0 0 0 25 0 1 0 661680549 29749248 2307 33554432000 4194304 4964800 140733626836752 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11056/statm: 7263 2314 376 189 0 2137 0

[startup+0.0167191 s]
/proc/loadavg: 7.48 7.94 7.97 1/348 11056
/proc/meminfo: memFree=19355664/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=34248 CPUtime=0.01 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 4292 0 0 0 1 0 0 0 25 0 1 0 661680549 35069952 3723 33554432000 4194304 4964800 140733626836752 18446744073709551615 274473967360 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11056/statm: 8562 3723 386 189 0 3436 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 34248

[startup+0.100694 s]
/proc/loadavg: 7.48 7.94 7.97 1/348 11056
/proc/meminfo: memFree=19355664/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=49844 CPUtime=0.09 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 9168 0 0 0 9 0 0 0 25 0 1 0 661680549 51040256 7503 33554432000 4194304 4964800 140733626836752 18446744073709551615 274473628357 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11056/statm: 12461 7503 400 189 0 7335 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 49844

[startup+0.300658 s]
/proc/loadavg: 7.48 7.94 7.97 1/348 11056
/proc/meminfo: memFree=19355664/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=55520 CPUtime=0.29 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 10559 0 0 0 28 1 0 0 25 0 1 0 661680549 56852480 8894 33554432000 4194304 4964800 140733626836752 18446744073709551615 4612829 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/11056/statm: 13880 8894 401 189 0 8754 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 55520

[startup+0.701441 s]
/proc/loadavg: 7.48 7.94 7.97 1/348 11056
/proc/meminfo: memFree=19355664/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=523980 CPUtime=0.85 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 40476 0 0 0 71 14 0 0 24 0 8 0 661680549 536555520 34188 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/11056/statm: 130995 34189 461 189 0 125344 0
Current children cumulated CPU time (s) 0.85
Current children cumulated vsize (KiB) 523980

[startup+1.50038 s]
/proc/loadavg: 7.48 7.94 7.97 9/356 11064
/proc/meminfo: memFree=19215608/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=526876 CPUtime=7.12 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 44685 0 0 0 697 15 0 0 25 0 8 0 661680549 539521024 38243 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11056/statm: 131719 38243 464 189 0 126068 0
[pid=11056/tid=11058] ppid=11054 vsize=526876 CPUtime=0.83 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 3767 0 0 0 81 2 0 0 18 0 8 0 661680604 539521024 38243 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=526876 CPUtime=0.81 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 4193 0 0 0 80 1 0 0 18 0 8 0 661680604 539521024 38243 33554432000 4194304 4964800 140733626836752 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=526876 CPUtime=0.82 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 5683 0 0 0 80 2 0 0 18 0 8 0 661680604 539521024 38243 33554432000 4194304 4964800 140733626836752 18446744073709551615 4332711 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=526876 CPUtime=0.82 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 3774 0 0 0 81 1 0 0 18 0 8 0 661680604 539521024 38243 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=526876 CPUtime=0.85 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 3912 0 0 0 84 1 0 0 19 0 8 0 661680604 539521024 38244 33554432000 4194304 4964800 140733626836752 18446744073709551615 4674719 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=526876 CPUtime=0.82 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 4902 0 0 0 80 2 0 0 18 0 8 0 661680604 539521024 38244 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=526876 CPUtime=0.81 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 5158 0 0 0 79 2 0 0 18 0 8 0 661680604 539521024 38244 33554432000 4194304 4964800 140733626836752 18446744073709551615 4325305 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 7.12
Current children cumulated vsize (KiB) 526876

[startup+3.1011 s]
/proc/loadavg: 7.48 7.94 7.97 9/356 11064
/proc/meminfo: memFree=19192576/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=529152 CPUtime=19.9 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 50397 0 0 0 1972 18 0 0 25 0 8 0 661680549 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370266 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11056/statm: 132288 43955 464 189 0 126637 0
[pid=11056/tid=11058] ppid=11054 vsize=529152 CPUtime=2.43 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 4003 0 0 0 241 2 0 0 21 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4362184 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=529152 CPUtime=2.42 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 5153 0 0 0 240 2 0 0 21 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=529152 CPUtime=2.42 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 6811 0 0 0 240 2 0 0 21 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4644138 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=529152 CPUtime=2.42 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 4170 0 0 0 241 1 0 0 21 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4371527 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=529152 CPUtime=2.45 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 3987 0 0 0 244 1 0 0 22 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370384 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=529152 CPUtime=2.41 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 6006 0 0 0 239 2 0 0 21 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4378951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=529152 CPUtime=2.41 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 6547 0 0 0 239 2 0 0 21 0 8 0 661680604 541851648 43955 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 19.9
Current children cumulated vsize (KiB) 529152

[startup+6.30053 s]
/proc/loadavg: 7.52 7.94 7.97 9/356 11064
/proc/meminfo: memFree=19158736/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=533340 CPUtime=45.45 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 59575 0 0 0 4522 23 0 0 25 0 8 0 661680549 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670534 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11056/statm: 133335 53062 466 189 0 127684 0
[pid=11056/tid=11058] ppid=11054 vsize=533340 CPUtime=5.62 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 4284 0 0 0 560 2 0 0 18 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370381 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=533340 CPUtime=5.61 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 6296 0 0 0 559 2 0 0 25 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4669294 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=533340 CPUtime=5.62 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 7683 0 0 0 560 2 0 0 25 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4644876 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=533340 CPUtime=5.61 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 4964 0 0 0 560 1 0 0 25 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370269 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=533340 CPUtime=5.65 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 5715 0 0 0 563 2 0 0 25 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=533340 CPUtime=5.62 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 7466 0 0 0 559 3 0 0 25 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=533340 CPUtime=5.61 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 8563 0 0 0 558 3 0 0 25 0 8 0 661680604 546140160 53062 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 45.45
Current children cumulated vsize (KiB) 533340

[startup+12.7004 s]
/proc/loadavg: 7.56 7.94 7.97 9/356 11064
/proc/meminfo: memFree=19098488/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=539284 CPUtime=96.55 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 71930 0 0 0 9626 29 0 0 25 0 8 0 661680549 552226816 65417 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370301 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11056/statm: 134821 65417 466 189 0 129170 0
[pid=11056/tid=11058] ppid=11054 vsize=539284 CPUtime=12.02 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 4726 0 0 0 1200 2 0 0 25 0 8 0 661680604 552226816 65417 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370266 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=539284 CPUtime=12.02 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 9498 0 0 0 1198 4 0 0 25 0 8 0 661680604 552226816 65420 33554432000 4194304 4964800 140733626836752 18446744073709551615 274473664267 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=539284 CPUtime=12.02 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 9677 0 0 0 1198 4 0 0 25 0 8 0 661680604 552226816 65420 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=539284 CPUtime=12.01 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 5912 0 0 0 1199 2 0 0 25 0 8 0 661680604 552226816 65420 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=539284 CPUtime=12.04 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 6610 0 0 0 1202 2 0 0 25 0 8 0 661680604 552226816 65420 33554432000 4194304 4964800 140733626836752 18446744073709551615 274474072607 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=539284 CPUtime=12 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 9568 0 0 0 1196 4 0 0 25 0 8 0 661680604 552226816 65420 33554432000 4194304 4964800 140733626836752 18446744073709551615 4321116 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=539284 CPUtime=12 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 10924 0 0 0 1196 4 0 0 25 0 8 0 661680604 552226816 65420 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670330 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 96.55
Current children cumulated vsize (KiB) 539284

[startup+25.5012 s]
/proc/loadavg: 7.65 7.95 7.97 9/356 11064
/proc/meminfo: memFree=19022912/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=554420 CPUtime=198.78 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 90988 0 0 0 19837 41 0 0 25 0 8 0 661680549 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11056/statm: 138605 84444 466 189 0 132954 0
[pid=11056/tid=11058] ppid=11054 vsize=554420 CPUtime=24.82 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 6310 0 0 0 2478 4 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=554420 CPUtime=24.81 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 12790 0 0 0 2476 5 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4379082 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=554420 CPUtime=24.81 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 12273 0 0 0 2476 5 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=554420 CPUtime=24.81 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 8188 0 0 0 2478 3 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=554420 CPUtime=24.85 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 8021 0 0 0 2481 4 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4362381 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=554420 CPUtime=24.8 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 12521 0 0 0 2474 6 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=554420 CPUtime=24.8 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 14806 0 0 0 2473 7 0 0 25 0 8 0 661680604 567726080 84444 33554432000 4194304 4964800 140733626836752 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 198.78
Current children cumulated vsize (KiB) 554420

[startup+51.1007 s]
/proc/loadavg: 7.85 7.97 7.98 9/356 11065
/proc/meminfo: memFree=18927688/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=566588 CPUtime=403.23 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 116311 0 0 0 40265 58 0 0 22 0 8 0 661680549 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370266 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/11056/statm: 141647 109409 466 189 0 135996 0
[pid=11056/tid=11058] ppid=11054 vsize=566588 CPUtime=50.39 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 7511 0 0 0 5035 4 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4668672 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=566588 CPUtime=50.4 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 16235 0 0 0 5033 7 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=566588 CPUtime=50.42 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 16954 0 0 0 5034 8 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=566588 CPUtime=50.39 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 9817 0 0 0 5035 4 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=566588 CPUtime=50.44 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 10681 0 0 0 5038 6 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=566588 CPUtime=50.39 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 16532 0 0 0 5031 8 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=566588 CPUtime=50.4 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 20537 0 0 0 5029 11 0 0 25 0 8 0 661680604 580186112 109409 33554432000 4194304 4964800 140733626836752 18446744073709551615 4663296 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 403.23

################
# More data... #
################

[pid=11056] ppid=11054 vsize=4345168 CPUtime=37713 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1028576 0 0 0 3770543 761 0 0 25 0 8 0 661680549 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670273 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11056/statm: 1086292 995252 466 189 0 1080641 0
[pid=11056/tid=11058] ppid=11054 vsize=4345168 CPUtime=4701.55 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 81832 0 0 0 470090 65 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4669389 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=4345168 CPUtime=4702.84 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 153668 0 0 0 470196 88 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=4345168 CPUtime=4717.06 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 150746 0 0 0 471590 116 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 274473625728 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=4345168 CPUtime=4720.28 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 88778 0 0 0 471973 55 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=4345168 CPUtime=4713.31 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 105752 0 0 0 471250 81 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4334283 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=4345168 CPUtime=4720.63 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 161898 0 0 0 471954 109 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=4345168 CPUtime=4721.3 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 197336 0 0 0 471977 153 0 0 25 0 8 0 661680604 4449452032 995252 33554432000 4194304 4964800 140733626836752 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37713
Current children cumulated vsize (KiB) 4345168

[startup+4782.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/356 11203
/proc/meminfo: memFree=15332992/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=4548800 CPUtime=38192.2 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1037977 0 0 0 3818451 768 0 0 25 0 8 0 661680549 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11056/statm: 1137200 1004282 466 189 0 1131549 0
[pid=11056/tid=11058] ppid=11054 vsize=4548800 CPUtime=4760.84 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 82610 0 0 0 476018 66 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=4548800 CPUtime=4762.81 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 154391 0 0 0 476193 88 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=4548800 CPUtime=4777.04 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 152490 0 0 0 477587 117 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4644138 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=4548800 CPUtime=4780.26 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 89805 0 0 0 477969 57 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=4548800 CPUtime=4773.3 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 106232 0 0 0 477249 81 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=4548800 CPUtime=4780.62 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 163219 0 0 0 477953 109 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=4548800 CPUtime=4781.28 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 199602 0 0 0 477973 155 0 0 25 0 8 0 661680604 4657971200 1004282 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38192.2
Current children cumulated vsize (KiB) 4548800

[startup+4842.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/356 11205
/proc/meminfo: memFree=15316080/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=4549356 CPUtime=38671.4 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1042507 0 0 0 3866372 771 0 0 25 0 8 0 661680549 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4668592 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11056/statm: 1137339 1008381 466 189 0 1131688 0
[pid=11056/tid=11058] ppid=11054 vsize=4549356 CPUtime=4820.1 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 82884 0 0 0 481944 66 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=4549356 CPUtime=4822.81 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 154956 0 0 0 482193 88 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=4549356 CPUtime=4837.04 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 152573 0 0 0 483586 118 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=4549356 CPUtime=4840.25 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 91097 0 0 0 483967 58 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=4549356 CPUtime=4833.3 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 107005 0 0 0 483248 82 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=4549356 CPUtime=4840.61 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 164010 0 0 0 483951 110 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=4549356 CPUtime=4841.28 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 199812 0 0 0 483973 155 0 0 25 0 8 0 661680604 4658540544 1008381 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38671.4
Current children cumulated vsize (KiB) 4549356

[startup+4902.3 s]
/proc/loadavg: 8.05 8.04 8.00 9/356 11206
/proc/meminfo: memFree=15290236/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=4620004 CPUtime=39150.7 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1049586 0 0 0 3914295 775 0 0 25 0 8 0 661680549 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4320976 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11056/statm: 1155001 1015055 466 189 0 1149350 0
[pid=11056/tid=11058] ppid=11054 vsize=4620004 CPUtime=4879.4 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 83280 0 0 0 487873 67 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=4620004 CPUtime=4882.82 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 155822 0 0 0 488194 88 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=4620004 CPUtime=4897.03 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 153457 0 0 0 489585 118 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4659391 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=4620004 CPUtime=4900.22 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 92331 0 0 0 489964 58 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 274473665118 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=4620004 CPUtime=4893.3 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 107825 0 0 0 489248 82 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4668607 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=4620004 CPUtime=4900.61 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 164960 0 0 0 489951 110 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=4620004 CPUtime=4901.29 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 200508 0 0 0 489973 156 0 0 25 0 8 0 661680604 4730884096 1015055 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39150.7
Current children cumulated vsize (KiB) 4620004

[startup+4962.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/356 11208
/proc/meminfo: memFree=15263892/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=4622076 CPUtime=39629.9 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1056747 0 0 0 3962212 782 0 0 25 0 8 0 661680549 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11056/statm: 1155519 1021577 466 189 0 1149868 0
[pid=11056/tid=11058] ppid=11054 vsize=4622076 CPUtime=4938.69 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 83791 0 0 0 493802 67 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=4622076 CPUtime=4942.82 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 156158 0 0 0 494194 88 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=4622076 CPUtime=4957.03 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 153820 0 0 0 495584 119 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=4622076 CPUtime=4960.21 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 92736 0 0 0 495963 58 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=4622076 CPUtime=4953.29 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 108910 0 0 0 495247 82 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=4622076 CPUtime=4960.6 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 165448 0 0 0 495949 111 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=4622076 CPUtime=4961.28 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 203304 0 0 0 495968 160 0 0 25 0 8 0 661680604 4733005824 1021577 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39629.9
Current children cumulated vsize (KiB) 4622076



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.7 s]
/proc/loadavg: 8.06 8.05 8.00 9/356 11209
/proc/meminfo: memFree=15245872/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=4623476 CPUtime=40000.6 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1061415 0 0 0 3999279 785 0 0 25 0 8 0 661680549 4734439424 1025914 33554432000 4194304 4964800 140733626836752 18446744073709551615 4362287 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/11056/statm: 1155869 1025914 466 189 0 1150218 0
[pid=11056/tid=11058] ppid=11054 vsize=4623476 CPUtime=4984.54 cores=0-7
/proc/11056/task/11058/stat : 11058 (claspmt) R 11054 11056 10996 0 -1 4202560 84095 0 0 0 498387 67 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11056/tid=11059] ppid=11054 vsize=4623476 CPUtime=4989.22 cores=0-7
/proc/11056/task/11059/stat : 11059 (claspmt) R 11054 11056 10996 0 -1 4202560 156867 0 0 0 498834 88 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11056/tid=11060] ppid=11054 vsize=4623476 CPUtime=5003.44 cores=0-7
/proc/11056/task/11060/stat : 11060 (claspmt) R 11054 11056 10996 0 -1 4202560 154580 0 0 0 500225 119 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11056/tid=11061] ppid=11054 vsize=4623476 CPUtime=5006.62 cores=0-7
/proc/11056/task/11061/stat : 11061 (claspmt) R 11054 11056 10996 0 -1 4202560 93226 0 0 0 500603 59 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11056/tid=11062] ppid=11054 vsize=4623476 CPUtime=4999.71 cores=0-7
/proc/11056/task/11062/stat : 11062 (claspmt) R 11054 11056 10996 0 -1 4202560 109532 0 0 0 499888 83 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11056/tid=11063] ppid=11054 vsize=4623476 CPUtime=5007.01 cores=0-7
/proc/11056/task/11063/stat : 11063 (claspmt) R 11054 11056 10996 0 -1 4202560 165909 0 0 0 500589 112 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11056/tid=11064] ppid=11054 vsize=4623476 CPUtime=5007.69 cores=0-7
/proc/11056/task/11064/stat : 11064 (claspmt) R 11054 11056 10996 0 -1 4202560 204438 0 0 0 500609 160 0 0 25 0 8 0 661680604 4734439424 1025915 33554432000 4194304 4964800 140733626836752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 4623476

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009 s]
/proc/loadavg: 8.06 8.05 8.00 9/356 11209
/proc/meminfo: memFree=15245872/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=4602852 CPUtime=40001 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1061428 0 0 0 3999315 785 0 0 25 0 1 0 661680549 4713320448 1020925 33554432000 4194304 4964800 140733626836752 18446744073709551615 274473622881 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/11056/statm: 1150713 1020925 476 189 0 1145062 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 4602852

[startup+5009.41 s]
/proc/loadavg: 8.06 8.05 8.00 2/350 11210
/proc/meminfo: memFree=15303704/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=2981888 CPUtime=40001.4 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1061428 0 0 0 3999349 793 0 0 25 0 1 0 661680549 3053453312 672730 33554432000 4194304 4964800 140733626836752 18446744073709551615 274474010983 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/11056/statm: 745472 672220 476 189 0 739821 0
Current children cumulated CPU time (s) 40001.4
Current children cumulated vsize (KiB) 2981888

[startup+5009.5 s]
/proc/loadavg: 8.06 8.05 8.00 2/350 11210
/proc/meminfo: memFree=15303704/32950928 swapFree=67042944/67111528
[pid=11056] ppid=11054 vsize=1383504 CPUtime=40001.5 cores=0-7
/proc/11056/stat : 11056 (claspmt) R 11054 11056 10996 0 -1 4202496 1061428 0 0 0 3999349 801 0 0 25 0 1 0 661680549 1416708096 281583 33554432000 4194304 4964800 140733626836752 18446744073709551615 274474010983 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/11056/statm: 345876 281583 476 189 0 340225 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 1383504

Child status: 0
Real time (s): 5009.57
CPU time (s): 40001.6
CPU user time (s): 39993.5
CPU system time (s): 8.08077
CPU usage (%): 798.504
Max. virtual memory (cumulated for all children) (KiB): 4623476

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.5
system time used= 8.08077
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1061429
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 171281
involuntary context switches= 181897

runsolver used 16.6155 second user time and 41.2587 second system time

The end

Launcher Data 

Begin job on node125 at 2011-05-16 01:06:49
IDJOB=3381031
IDBENCH=83191
IDSOLVER=1865
FILE ID=node125/3381031-1305500809
RUNJOBID= node125-1305500808-11010
PBS_JOBID= 13325461
Free space on /tmp= 73188 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/skvortsov/automata-synchronization/crn_40_1521_s.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381031-1305500809/watcher-3381031-1305500809 -o /tmp/evaluation-result-3381031-1305500809/solver-3381031-1305500809 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381031-1305500809.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4e35cb9894b1ca841d462da2d6957cc5
RANDOM SEED=1360956537

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19355688 kB
Buffers:        186108 kB
Cached:        1905196 kB
SwapCached:      55652 kB
Active:        6133480 kB
Inactive:      7131980 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19355688 kB
SwapTotal:    67111528 kB
SwapFree:     67042944 kB
Dirty:            3552 kB
Writeback:           0 kB
AnonPages:    11118392 kB
Mapped:          34672 kB
Slab:           218052 kB
PageTables:      42932 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 79239488 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73184 MiB
End job on node125 at 2011-05-16 02:30:20