Trace number 3380939

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.2 5009.45

General information on the benchmark

Nameapplication/fuhs/slp-synthesis-AES/
slp-synthesis-aes-bottom25.cnf
MD5SUMc40bd543dad558d19995fef4fd742233
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables67777
Number of clauses227761
Sum of the clauses size582177
Maximum clause length2084
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 2115213
Number of clauses of size 3111398
Number of clauses of size 4324
Number of clauses of size 523
Number of clauses of size over 5802

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...495389/instance-3380939-1305495389.cnf
0.69/0.70	c Solving...
40000.46/5008.82	
40000.46/5008.82	*** INTERRUPTED! ***
40000.46/5008.82	*** Shutting down threads...
40001.14/5009.44	s UNKNOWN
40001.14/5009.44	
40001.14/5009.44	c INTERRUPTED   : 1
40001.14/5009.44	c Models        : 0+    
40001.14/5009.44	c Time          : 5008.819s (Solving: 5008.11s 1st Model: 0.00s Unsat: 0.00s)
40001.14/5009.44	c CPU Time      : 40000.570s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380939-1305495389/watcher-3380939-1305495389 -o /tmp/evaluation-result-3380939-1305495389/solver-3380939-1305495389 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380939-1305495389.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.16 7.83 7.80 2/356 5582
/proc/meminfo: memFree=26446844/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=29548 CPUtime=0 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 1808 0 0 0 0 0 0 0 22 0 1 0 661137427 30257152 1738 33554432000 4194304 4964800 140735957799760 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/5582/statm: 7387 1746 376 189 0 2261 0

[startup+0.10068 s]
/proc/loadavg: 8.16 7.83 7.80 2/356 5582
/proc/meminfo: memFree=26446844/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=50640 CPUtime=0.09 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 10035 0 0 0 8 1 0 0 22 0 1 0 661137427 51855360 7625 33554432000 4194304 4964800 140735957799760 18446744073709551615 210204308165 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/5582/statm: 12660 7626 399 189 0 7534 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 50640

[startup+0.200651 s]
/proc/loadavg: 8.16 7.83 7.80 2/356 5582
/proc/meminfo: memFree=26446844/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=57652 CPUtime=0.19 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 11650 0 0 0 18 1 0 0 22 0 1 0 661137427 59035648 9240 33554432000 4194304 4964800 140735957799760 18446744073709551615 210204308165 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/5582/statm: 14413 9240 404 189 0 9287 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 57652

[startup+0.300637 s]
/proc/loadavg: 8.16 7.83 7.80 2/356 5582
/proc/meminfo: memFree=26446844/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=60424 CPUtime=0.29 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 12353 0 0 0 28 1 0 0 23 0 1 0 661137427 61874176 9943 33554432000 4194304 4964800 140735957799760 18446744073709551615 4600357 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/5582/statm: 15106 9943 404 189 0 9980 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 60424

[startup+0.700586 s]
/proc/loadavg: 8.16 7.83 7.80 2/356 5582
/proc/meminfo: memFree=26446844/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=67028 CPUtime=0.69 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 13987 0 0 0 67 2 0 0 25 0 1 0 661137427 68636672 11577 33554432000 4194304 4964800 140735957799760 18446744073709551615 4685574 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/5582/statm: 16757 11577 406 189 0 11631 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 67028

[startup+1.5004 s]
/proc/loadavg: 8.16 7.83 7.80 9/364 5590
/proc/meminfo: memFree=26342628/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=519464 CPUtime=5.31 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 35122 0 0 0 520 11 0 0 25 0 8 0 661137427 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668721 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/5582/statm: 129866 27845 464 189 0 124215 0
[pid=5582/tid=5584] ppid=5580 vsize=519464 CPUtime=0.55 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 2522 0 0 0 54 1 0 0 16 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4629961 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=519464 CPUtime=0.54 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 3445 0 0 0 53 1 0 0 16 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4673285 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=519464 CPUtime=0.54 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 3217 0 0 0 53 1 0 0 16 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668805 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=519464 CPUtime=0.55 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 2389 0 0 0 55 0 0 0 16 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=519464 CPUtime=0.72 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 2979 0 0 0 72 0 0 0 18 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=519464 CPUtime=0.55 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 3187 0 0 0 54 1 0 0 16 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=519464 CPUtime=0.56 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 3108 0 0 0 55 1 0 0 16 0 8 0 661137498 531931136 27845 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.31
Current children cumulated vsize (KiB) 519464

[startup+3.10113 s]
/proc/loadavg: 8.16 7.83 7.80 10/364 5590
/proc/meminfo: memFree=26330480/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=519464 CPUtime=18.1 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 38959 0 0 0 1794 16 0 0 25 0 8 0 661137427 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4684031 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/5582/statm: 129866 31682 464 189 0 124215 0
[pid=5582/tid=5584] ppid=5580 vsize=519464 CPUtime=2.16 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 2969 0 0 0 214 2 0 0 18 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4321097 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=519464 CPUtime=2.14 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 4210 0 0 0 212 2 0 0 18 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4660236 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=519464 CPUtime=2.15 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 3759 0 0 0 213 2 0 0 18 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669136 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=519464 CPUtime=2.15 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 2981 0 0 0 214 1 0 0 18 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=519464 CPUtime=2.3 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 2994 0 0 0 230 0 0 0 21 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4399292 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=519464 CPUtime=2.15 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 3701 0 0 0 214 1 0 0 18 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=519464 CPUtime=2.15 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 3606 0 0 0 214 1 0 0 18 0 8 0 661137498 531931136 31682 33554432000 4194304 4964800 140735957799760 18446744073709551615 4678045 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 18.1
Current children cumulated vsize (KiB) 519464

[startup+6.3006 s]
/proc/loadavg: 8.23 7.85 7.81 9/364 5590
/proc/meminfo: memFree=26301008/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=524140 CPUtime=43.66 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 46804 0 0 0 4345 21 0 0 25 0 8 0 661137427 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4364587 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/5582/statm: 131035 39527 464 189 0 125384 0
[pid=5582/tid=5584] ppid=5580 vsize=524140 CPUtime=5.35 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 3489 0 0 0 533 2 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4370242 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=524140 CPUtime=5.35 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 5547 0 0 0 532 3 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=524140 CPUtime=5.35 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 5118 0 0 0 532 3 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670520 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=524140 CPUtime=5.35 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 3813 0 0 0 534 1 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4627299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=524140 CPUtime=5.46 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 3729 0 0 0 545 1 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=524140 CPUtime=5.35 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 4700 0 0 0 533 2 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=524140 CPUtime=5.36 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 4991 0 0 0 534 2 0 0 25 0 8 0 661137498 536719360 39527 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 43.66
Current children cumulated vsize (KiB) 524140

[startup+12.7005 s]
/proc/loadavg: 8.21 7.85 7.81 9/364 5591
/proc/meminfo: memFree=26256024/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=527044 CPUtime=94.76 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 56827 0 0 0 9447 29 0 0 25 0 8 0 661137427 539693056 49550 33554432000 4194304 4964800 140735957799760 18446744073709551615 4683657 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/5582/statm: 131761 49550 466 189 0 126110 0
[pid=5582/tid=5584] ppid=5580 vsize=527044 CPUtime=11.75 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 4220 0 0 0 1172 3 0 0 25 0 8 0 661137498 539693056 49550 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=527044 CPUtime=11.75 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 7551 0 0 0 1171 4 0 0 25 0 8 0 661137498 539693056 49550 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=527044 CPUtime=11.75 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 6457 0 0 0 1171 4 0 0 25 0 8 0 661137498 539693056 49550 33554432000 4194304 4964800 140735957799760 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=527044 CPUtime=11.75 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 4851 0 0 0 1173 2 0 0 25 0 8 0 661137498 539693056 49551 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670134 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=527044 CPUtime=11.79 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 5232 0 0 0 1176 3 0 0 25 0 8 0 661137498 539693056 49551 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668627 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=527044 CPUtime=11.73 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 5641 0 0 0 1171 2 0 0 25 0 8 0 661137498 539693056 49551 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=527044 CPUtime=11.76 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 7036 0 0 0 1172 4 0 0 25 0 8 0 661137498 539693056 49551 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 94.76
Current children cumulated vsize (KiB) 527044

[startup+25.5014 s]
/proc/loadavg: 8.16 7.86 7.81 9/364 5591
/proc/meminfo: memFree=26197404/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=535408 CPUtime=197.01 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 71339 0 0 0 19660 41 0 0 25 0 8 0 661137427 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4653507 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/5582/statm: 133852 64062 466 189 0 128201 0
[pid=5582/tid=5584] ppid=5580 vsize=535408 CPUtime=24.55 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 5211 0 0 0 2451 4 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4683952 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=535408 CPUtime=24.55 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 10160 0 0 0 2449 6 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=535408 CPUtime=24.54 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 8214 0 0 0 2449 5 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=535408 CPUtime=24.55 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 6142 0 0 0 2452 3 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4342162 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=535408 CPUtime=24.43 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 6813 0 0 0 2439 4 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669507 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=535408 CPUtime=24.53 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 7257 0 0 0 2450 3 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=535408 CPUtime=24.56 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 10353 0 0 0 2450 6 0 0 25 0 8 0 661137498 548257792 64062 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 197.01
Current children cumulated vsize (KiB) 535408

[startup+51.1011 s]
/proc/loadavg: 8.10 7.87 7.82 9/364 5591
/proc/meminfo: memFree=26103680/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=549732 CPUtime=401.49 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 94301 0 0 0 40086 63 0 0 25 0 8 0 661137427 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668987 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/5582/statm: 137433 87024 466 189 0 131782 0
[pid=5582/tid=5584] ppid=5580 vsize=549732 CPUtime=50.15 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 6796 0 0 0 5009 6 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=549732 CPUtime=50.15 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 13285 0 0 0 5007 8 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=549732 CPUtime=50.15 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 9948 0 0 0 5008 7 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=549732 CPUtime=50.15 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 9182 0 0 0 5008 7 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=549732 CPUtime=49.73 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 9345 0 0 0 4966 7 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=549732 CPUtime=50.13 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 9592 0 0 0 5007 6 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4363605 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=549732 CPUtime=50.15 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 16961 0 0 0 5003 12 0 0 25 0 8 0 661137498 562925568 87024 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 401.49

################
# More data... #
################

[pid=5582] ppid=5580 vsize=2450456 CPUtime=37712.1 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 551245 0 0 0 3770484 729 0 0 25 0 8 0 661137427 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/5582/statm: 612614 540581 466 189 0 606963 0
[pid=5582/tid=5584] ppid=5580 vsize=2450456 CPUtime=4708.43 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 39371 0 0 0 470787 56 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=2450456 CPUtime=4716.4 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 60460 0 0 0 471572 68 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=2450456 CPUtime=4721.17 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 48021 0 0 0 472058 59 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=2450456 CPUtime=4710 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 66033 0 0 0 470900 100 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=2450456 CPUtime=4712.2 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 57658 0 0 0 471138 82 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=2450456 CPUtime=4720.24 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 46560 0 0 0 471967 57 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=2450456 CPUtime=4708.22 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 157909 0 0 0 470620 202 0 0 25 0 8 0 661137498 2509266944 540581 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37712.1
Current children cumulated vsize (KiB) 2450456

[startup+4782.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/364 5730
/proc/meminfo: memFree=24283976/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2450456 CPUtime=38191.3 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 551314 0 0 0 3818397 730 0 0 25 0 8 0 661137427 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/5582/statm: 612614 540650 466 189 0 606963 0
[pid=5582/tid=5584] ppid=5580 vsize=2450456 CPUtime=4768.32 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 39371 0 0 0 476776 56 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=2450456 CPUtime=4776.39 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 60461 0 0 0 477571 68 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=2450456 CPUtime=4781.16 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 48021 0 0 0 478057 59 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=2450456 CPUtime=4769.98 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 66096 0 0 0 476898 100 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=2450456 CPUtime=4772.2 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 57661 0 0 0 477137 83 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=2450456 CPUtime=4780.21 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 46562 0 0 0 477964 57 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=2450456 CPUtime=4768.21 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 157909 0 0 0 476619 202 0 0 25 0 8 0 661137498 2509266944 540650 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38191.3
Current children cumulated vsize (KiB) 2450456

[startup+4842.31 s]
/proc/loadavg: 8.02 8.05 8.00 9/364 5731
/proc/meminfo: memFree=24283564/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2450456 CPUtime=38670.6 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 551381 0 0 0 3866329 731 0 0 25 0 8 0 661137427 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4332401 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/5582/statm: 612614 540717 466 189 0 606963 0
[pid=5582/tid=5584] ppid=5580 vsize=2450456 CPUtime=4828.34 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 39371 0 0 0 482778 56 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=2450456 CPUtime=4836.4 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 60462 0 0 0 483572 68 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4379605 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=2450456 CPUtime=4841.15 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 48021 0 0 0 484056 59 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4653368 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=2450456 CPUtime=4829.98 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 66096 0 0 0 482897 101 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=2450456 CPUtime=4832.21 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 57723 0 0 0 483138 83 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=2450456 CPUtime=4840.21 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 46562 0 0 0 483964 57 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4332135 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=2450456 CPUtime=4828.21 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 157909 0 0 0 482619 202 0 0 25 0 8 0 661137498 2509266944 540717 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669882 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38670.6
Current children cumulated vsize (KiB) 2450456

[startup+4902.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/364 5733
/proc/meminfo: memFree=24283528/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2450456 CPUtime=39149.8 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 551404 0 0 0 3914247 731 0 0 25 0 8 0 661137427 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/5582/statm: 612614 540740 466 189 0 606963 0
[pid=5582/tid=5584] ppid=5580 vsize=2450456 CPUtime=4888.32 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 39374 0 0 0 488776 56 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=2450456 CPUtime=4896.39 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 60467 0 0 0 489571 68 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4362864 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=2450456 CPUtime=4901.13 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 48021 0 0 0 490054 59 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=2450456 CPUtime=4889.97 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 66101 0 0 0 488896 101 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=2450456 CPUtime=4892.2 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 57733 0 0 0 489137 83 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=2450456 CPUtime=4900.19 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 46562 0 0 0 489962 57 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=2450456 CPUtime=4888.2 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 157909 0 0 0 488618 202 0 0 25 0 8 0 661137498 2509266944 540740 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39149.8
Current children cumulated vsize (KiB) 2450456

[startup+4962.3 s]
/proc/loadavg: 8.04 8.04 8.00 9/364 5735
/proc/meminfo: memFree=24281748/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2450456 CPUtime=39629.1 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 551787 0 0 0 3962173 732 0 0 25 0 8 0 661137427 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668564 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/5582/statm: 612614 541123 466 189 0 606963 0
[pid=5582/tid=5584] ppid=5580 vsize=2450456 CPUtime=4948.32 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 39378 0 0 0 494776 56 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=2450456 CPUtime=4956.39 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 60492 0 0 0 495571 68 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=2450456 CPUtime=4961.13 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 48021 0 0 0 496054 59 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=2450456 CPUtime=4949.97 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 66101 0 0 0 494896 101 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=2450456 CPUtime=4952.2 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 57893 0 0 0 495137 83 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=2450456 CPUtime=4960.18 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 46756 0 0 0 495960 58 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=2450456 CPUtime=4948.2 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 157909 0 0 0 494618 202 0 0 25 0 8 0 661137498 2509266944 541123 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39629.1
Current children cumulated vsize (KiB) 2450456



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.8 s]
/proc/loadavg: 8.02 8.03 8.00 9/364 5736
/proc/meminfo: memFree=24280340/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2450456 CPUtime=40000.5 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 552272 0 0 0 3999313 733 0 0 25 0 8 0 661137427 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4669089 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/5582/statm: 612614 541366 466 189 0 606963 0
[pid=5582/tid=5584] ppid=5580 vsize=2450456 CPUtime=4994.79 cores=0-7
/proc/5582/task/5584/stat : 5584 (claspmt) R 5580 5582 5258 0 -1 4202560 39674 0 0 0 499422 57 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=5582/tid=5585] ppid=5580 vsize=2450456 CPUtime=5002.9 cores=0-7
/proc/5582/task/5585/stat : 5585 (claspmt) R 5580 5582 5258 0 -1 4202560 60507 0 0 0 500222 68 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=5582/tid=5586] ppid=5580 vsize=2450456 CPUtime=5007.65 cores=0-7
/proc/5582/task/5586/stat : 5586 (claspmt) R 5580 5582 5258 0 -1 4202560 48037 0 0 0 500706 59 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=5582/tid=5587] ppid=5580 vsize=2450456 CPUtime=4996.48 cores=0-7
/proc/5582/task/5587/stat : 5587 (claspmt) R 5580 5582 5258 0 -1 4202560 66109 0 0 0 499547 101 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=5582/tid=5588] ppid=5580 vsize=2450456 CPUtime=4998.69 cores=0-7
/proc/5582/task/5588/stat : 5588 (claspmt) R 5580 5582 5258 0 -1 4202560 58014 0 0 0 499786 83 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=5582/tid=5589] ppid=5580 vsize=2450456 CPUtime=5006.67 cores=0-7
/proc/5582/task/5589/stat : 5589 (claspmt) R 5580 5582 5258 0 -1 4202560 46781 0 0 0 500609 58 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=5582/tid=5590] ppid=5580 vsize=2450456 CPUtime=4994.71 cores=0-7
/proc/5582/task/5590/stat : 5590 (claspmt) R 5580 5582 5258 0 -1 4202560 157911 0 0 0 499269 202 0 0 25 0 8 0 661137498 2509266944 541366 33554432000 4194304 4964800 140735957799760 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 2450456

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.1 s]
/proc/loadavg: 8.02 8.03 8.00 9/364 5736
/proc/meminfo: memFree=24280340/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2421964 CPUtime=40000.8 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 552283 0 0 0 3999350 734 0 0 25 0 1 0 661137427 2480091136 533459 33554432000 4194304 4964800 140735957799760 18446744073709551615 210204302689 0 0 4096 18946 0 0 0 17 7 0 0 0
/proc/5582/statm: 605491 533459 476 189 0 599840 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2421964

[startup+5009.3 s]
/proc/loadavg: 8.02 8.03 8.00 9/364 5736
/proc/meminfo: memFree=24280340/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=2411956 CPUtime=40001 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 552283 0 0 0 3999370 734 0 0 25 0 1 0 661137427 2469842944 530039 33554432000 4194304 4964800 140735957799760 18446744073709551615 210204303931 0 0 4096 18946 0 0 0 17 7 0 0 0
/proc/5582/statm: 602989 530039 476 189 0 597338 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 2411956

[startup+5009.4 s]
/proc/loadavg: 8.02 8.03 8.00 2/358 5737
/proc/meminfo: memFree=25656044/32951124 swapFree=63649232/67111528
[pid=5582] ppid=5580 vsize=927780 CPUtime=40001.1 cores=0-7
/proc/5582/stat : 5582 (claspmt) R 5580 5582 5258 0 -1 4202496 552283 0 0 0 3999373 741 0 0 25 0 1 0 661137427 950046720 167443 33554432000 4194304 4964800 140735957799760 18446744073709551615 210204673130 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/5582/statm: 231945 167443 476 189 0 226294 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 927780

Child status: 0
Real time (s): 5009.45
CPU time (s): 40001.2
CPU user time (s): 39993.7
CPU system time (s): 7.45387
CPU usage (%): 798.515
Max. virtual memory (cumulated for all children) (KiB): 2450456

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.7
system time used= 7.45387
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 552284
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 83677
involuntary context switches= 190863

runsolver used 15.6036 second user time and 43.6184 second system time

The end

Launcher Data 

Begin job on node130 at 2011-05-15 23:36:29
IDJOB=3380939
IDBENCH=83109
IDSOLVER=1865
FILE ID=node130/3380939-1305495389
RUNJOBID= node130-1305489807-5272
PBS_JOBID= 13325408
Free space on /tmp= 73392 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/fuhs/slp-synthesis-AES/slp-synthesis-aes-bottom25.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380939-1305495389/watcher-3380939-1305495389 -o /tmp/evaluation-result-3380939-1305495389/solver-3380939-1305495389 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380939-1305495389.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c40bd543dad558d19995fef4fd742233
RANDOM SEED=1653650125

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      26447248 kB
Buffers:        300680 kB
Cached:        4103228 kB
SwapCached:      10248 kB
Active:        2277428 kB
Inactive:      3934136 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      26447248 kB
SwapTotal:    67111528 kB
SwapFree:     63649232 kB
Dirty:            5048 kB
Writeback:           0 kB
AnonPages:     1807076 kB
Mapped:          21104 kB
Slab:           189236 kB
PageTables:      34404 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 88885052 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73392 MiB
End job on node130 at 2011-05-16 01:00:00