Trace number 3381109

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.2 5007.69

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
64/aes_64_4_keyfind_1.cnf
MD5SUM07ea67092ce3fd4529e72f69434e3f99
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1808
Number of clauses9968
Sum of the clauses size39152
Maximum clause length5
Minimum clause length1
Number of clauses of size 1144
Number of clauses of size 2320
Number of clauses of size 33904
Number of clauses of size 41344
Number of clauses of size 54256
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...504617/instance-3381109-1305504617.cnf
0.00/0.02	c Solving...
40000.09/5007.61	
40000.09/5007.61	*** INTERRUPTED! ***
40000.09/5007.61	*** Shutting down threads...
40000.09/5007.68	s UNKNOWN
40000.09/5007.68	
40000.09/5007.68	c INTERRUPTED   : 1
40000.09/5007.68	c Models        : 0+    
40000.09/5007.68	c Time          : 5007.616s (Solving: 5007.59s 1st Model: 0.00s Unsat: 0.00s)
40000.09/5007.68	c CPU Time      : 40000.160s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381109-1305504617/watcher-3381109-1305504617 -o /tmp/evaluation-result-3381109-1305504617/solver-3381109-1305504617 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381109-1305504617.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.54 7.97 7.97 2/331 6519
/proc/meminfo: memFree=20806912/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=22092 CPUtime=0 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 870 0 0 0 0 0 0 0 25 0 1 0 642877303 22622208 769 33554432000 4194304 4964800 140736855020848 18446744073709551615 270338044613 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/6519/statm: 5564 778 399 189 0 438 0

[startup+0.125773 s]
/proc/loadavg: 7.54 7.97 7.97 2/331 6519
/proc/meminfo: memFree=20806912/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=383864 CPUtime=1.1 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 3491 0 0 0 108 2 0 0 19 0 8 0 642877303 393076736 3299 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668698 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 95966 3299 464 189 0 90315 0
Current children cumulated CPU time (s) 1.1
Current children cumulated vsize (KiB) 383864

[startup+0.273871 s]
/proc/loadavg: 7.54 7.97 7.97 2/331 6519
/proc/meminfo: memFree=20806912/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=384392 CPUtime=1.9 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 4191 0 0 0 187 3 0 0 19 0 8 0 642877303 393617408 3999 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 96098 3999 464 189 0 90447 0
Current children cumulated CPU time (s) 1.9
Current children cumulated vsize (KiB) 384392

[startup+0.373852 s]
/proc/loadavg: 7.54 7.97 7.97 2/331 6519
/proc/meminfo: memFree=20806912/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=384656 CPUtime=2.7 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 4468 0 0 0 267 3 0 0 19 0 8 0 642877303 393887744 4276 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670538 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 96164 4276 464 189 0 90513 0
Current children cumulated CPU time (s) 2.7
Current children cumulated vsize (KiB) 384656

[startup+0.773773 s]
/proc/loadavg: 7.54 7.97 7.97 2/331 6519
/proc/meminfo: memFree=20806912/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=385200 CPUtime=5.9 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 5147 0 0 0 587 3 0 0 20 0 8 0 642877303 394444800 4955 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670381 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 96300 4955 464 189 0 90649 0
Current children cumulated CPU time (s) 5.9
Current children cumulated vsize (KiB) 385200

[startup+1.50063 s]
/proc/loadavg: 7.66 7.99 7.98 9/339 6527
/proc/meminfo: memFree=20787152/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=385920 CPUtime=11.7 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 6078 0 0 0 1166 4 0 0 21 0 8 0 642877303 395182080 5886 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 96480 5887 464 189 0 90829 0
[pid=6519/tid=6521] ppid=6517 vsize=385920 CPUtime=1.46 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 528 0 0 0 146 0 0 0 19 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668924 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=385920 CPUtime=1.46 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 626 0 0 0 146 0 0 0 20 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 4342204 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=385920 CPUtime=1.45 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 575 0 0 0 145 0 0 0 20 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 270338044961 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=385920 CPUtime=1.46 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 649 0 0 0 146 0 0 0 20 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670351 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=385920 CPUtime=1.46 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 595 0 0 0 146 0 0 0 20 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670203 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=385920 CPUtime=1.45 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 575 0 0 0 145 0 0 0 19 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=385920 CPUtime=1.41 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 1007 0 0 0 141 0 0 0 23 0 8 0 642877305 395182080 5887 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670351 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.7
Current children cumulated vsize (KiB) 385920

[startup+3.10031 s]
/proc/loadavg: 7.66 7.99 7.98 9/339 6527
/proc/meminfo: memFree=20780084/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=387384 CPUtime=24.48 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 8040 0 0 0 2443 5 0 0 25 0 8 0 642877303 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4660442 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 96846 7848 464 189 0 91195 0
[pid=6519/tid=6521] ppid=6517 vsize=387384 CPUtime=3.06 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 748 0 0 0 306 0 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=387384 CPUtime=3.05 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 955 0 0 0 305 0 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=387384 CPUtime=3.06 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 728 0 0 0 305 1 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=387384 CPUtime=3.05 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 942 0 0 0 305 0 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669532 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=387384 CPUtime=3.05 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 795 0 0 0 305 0 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=387384 CPUtime=3.05 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 754 0 0 0 305 0 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4378523 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=387384 CPUtime=2.99 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 1430 0 0 0 299 0 0 0 25 0 8 0 642877305 396681216 7848 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669532 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.48
Current children cumulated vsize (KiB) 387384

[startup+6.30068 s]
/proc/loadavg: 7.68 7.99 7.98 9/339 6528
/proc/meminfo: memFree=20767652/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=389160 CPUtime=50.04 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 12068 0 0 0 4997 7 0 0 25 0 8 0 642877303 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670357 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 97290 11786 466 189 0 91639 0
[pid=6519/tid=6521] ppid=6517 vsize=389160 CPUtime=6.25 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 1168 0 0 0 625 0 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670640 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=389160 CPUtime=6.25 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 1543 0 0 0 625 0 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=389160 CPUtime=6.26 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 1158 0 0 0 625 1 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=389160 CPUtime=6.25 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 1407 0 0 0 625 0 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668721 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=389160 CPUtime=6.25 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 1318 0 0 0 625 0 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4658618 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=389160 CPUtime=6.25 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 992 0 0 0 625 0 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4660187 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=389160 CPUtime=6.16 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 2368 0 0 0 615 1 0 0 25 0 8 0 642877305 398499840 11786 33554432000 4194304 4964800 140736855020848 18446744073709551615 4365968 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.04
Current children cumulated vsize (KiB) 389160

[startup+12.7004 s]
/proc/loadavg: 7.71 7.99 7.98 9/339 6528
/proc/meminfo: memFree=20756548/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=390680 CPUtime=101.16 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 13798 0 0 0 10107 9 0 0 25 0 8 0 642877303 400056320 13480 33554432000 4194304 4964800 140736855020848 18446744073709551615 4342208 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 97670 13480 466 189 0 92019 0
[pid=6519/tid=6521] ppid=6517 vsize=390680 CPUtime=12.66 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 1501 0 0 0 1265 1 0 0 25 0 8 0 642877305 400056320 13480 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=390680 CPUtime=12.66 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 1587 0 0 0 1265 1 0 0 25 0 8 0 642877305 400056320 13481 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668732 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=390680 CPUtime=12.65 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 1326 0 0 0 1264 1 0 0 25 0 8 0 642877305 400056320 13481 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=390680 CPUtime=12.65 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 1518 0 0 0 1265 0 0 0 25 0 8 0 642877305 400056320 13481 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=390680 CPUtime=12.66 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 1690 0 0 0 1265 1 0 0 25 0 8 0 642877305 400056320 13481 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=390680 CPUtime=12.65 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 1115 0 0 0 1265 0 0 0 25 0 8 0 642877305 400056320 13481 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=390680 CPUtime=12.49 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 2757 0 0 0 1248 1 0 0 25 0 8 0 642877305 400056320 13481 33554432000 4194304 4964800 140736855020848 18446744073709551615 4658562 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.16
Current children cumulated vsize (KiB) 390680

[startup+25.5009 s]
/proc/loadavg: 7.75 7.99 7.98 9/339 6528
/proc/meminfo: memFree=20739580/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=394448 CPUtime=203.41 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 18093 0 0 0 20329 12 0 0 25 0 8 0 642877303 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670534 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 98612 17775 466 189 0 92961 0
[pid=6519/tid=6521] ppid=6517 vsize=394448 CPUtime=25.46 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 2017 0 0 0 2545 1 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=394448 CPUtime=25.46 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 2237 0 0 0 2545 1 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=394448 CPUtime=25.45 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 1628 0 0 0 2544 1 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669913 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=394448 CPUtime=25.45 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 1914 0 0 0 2545 0 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669507 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=394448 CPUtime=25.45 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 2357 0 0 0 2543 2 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4465565 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=394448 CPUtime=25.45 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 1507 0 0 0 2544 1 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=394448 CPUtime=25.15 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 3674 0 0 0 2513 2 0 0 25 0 8 0 642877305 403914752 17775 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 203.41
Current children cumulated vsize (KiB) 394448

[startup+51.1008 s]
/proc/loadavg: 7.90 8.00 7.98 9/339 6529
/proc/meminfo: memFree=20719320/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=397164 CPUtime=407.91 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 23152 0 0 0 40774 17 0 0 25 0 8 0 642877303 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668734 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 99291 22834 466 189 0 93640 0
[pid=6519/tid=6521] ppid=6517 vsize=397164 CPUtime=51.04 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 2705 0 0 0 5103 1 0 0 25 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=397164 CPUtime=51.06 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 3373 0 0 0 5104 2 0 0 25 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=397164 CPUtime=51.04 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 2059 0 0 0 5103 1 0 0 18 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=397164 CPUtime=51.05 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 2693 0 0 0 5103 2 0 0 25 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=397164 CPUtime=51.05 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 2888 0 0 0 5103 2 0 0 25 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=397164 CPUtime=51.04 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 2077 0 0 0 5103 1 0 0 25 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=397164 CPUtime=50.46 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 4453 0 0 0 5044 2 0 0 25 0 8 0 642877305 406695936 22834 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.91

################
# More data... #
################


[startup+4722.3 s]
/proc/loadavg: 8.07 8.08 8.02 9/339 6668
/proc/meminfo: memFree=20391048/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=780980 CPUtime=37721.1 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 107536 0 0 0 3771810 297 0 0 25 0 8 0 642877303 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195245 103793 466 189 0 189594 0
[pid=6519/tid=6521] ppid=6517 vsize=780980 CPUtime=4703.18 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 15740 0 0 0 470274 44 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=780980 CPUtime=4721.67 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15570 0 0 0 472131 36 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321065 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=780980 CPUtime=4720.65 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9534 0 0 0 472036 29 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4332288 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=780980 CPUtime=4716.35 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12633 0 0 0 471598 37 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668777 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=780980 CPUtime=4721.33 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13174 0 0 0 472097 36 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=780980 CPUtime=4702.27 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10439 0 0 0 470192 35 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=780980 CPUtime=4713.99 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18394 0 0 0 471355 44 0 0 25 0 8 0 642877305 799723520 103793 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37721.1
Current children cumulated vsize (KiB) 780980

[startup+4782.3 s]
/proc/loadavg: 8.02 8.06 8.01 9/339 6670
/proc/meminfo: memFree=20390548/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=780980 CPUtime=38200.3 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 107645 0 0 0 3819731 300 0 0 25 0 8 0 642877303 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4325338 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195245 103899 466 189 0 189594 0
[pid=6519/tid=6521] ppid=6517 vsize=780980 CPUtime=4763.15 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 15766 0 0 0 476271 44 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4362248 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=780980 CPUtime=4781.65 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15570 0 0 0 478129 36 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=780980 CPUtime=4780.61 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9534 0 0 0 478032 29 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=780980 CPUtime=4776.34 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12637 0 0 0 477596 38 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=780980 CPUtime=4781.32 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13174 0 0 0 478095 37 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4660163 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=780980 CPUtime=4761.61 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10502 0 0 0 476126 35 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=780980 CPUtime=4773.97 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18394 0 0 0 477353 44 0 0 25 0 8 0 642877305 799723520 103899 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38200.3
Current children cumulated vsize (KiB) 780980

[startup+4842.31 s]
/proc/loadavg: 8.01 8.05 8.01 9/339 6672
/proc/meminfo: memFree=20390264/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=781168 CPUtime=38679.7 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 107766 0 0 0 3867664 303 0 0 25 0 8 0 642877303 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195292 104020 466 189 0 189641 0
[pid=6519/tid=6521] ppid=6517 vsize=781168 CPUtime=4823.15 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 15805 0 0 0 482270 45 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=781168 CPUtime=4841.66 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15570 0 0 0 484130 36 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=781168 CPUtime=4840.61 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9540 0 0 0 484032 29 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669376 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=781168 CPUtime=4836.34 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12637 0 0 0 483596 38 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=781168 CPUtime=4841.32 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13176 0 0 0 484095 37 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670282 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=781168 CPUtime=4820.95 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10527 0 0 0 482059 36 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=781168 CPUtime=4833.99 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18415 0 0 0 483354 45 0 0 25 0 8 0 642877305 799916032 104020 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38679.7
Current children cumulated vsize (KiB) 781168

[startup+4902.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/339 6673
/proc/meminfo: memFree=20388136/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=781912 CPUtime=39158.9 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 108435 0 0 0 3915581 307 0 0 25 0 8 0 642877303 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195478 104655 466 189 0 189827 0
[pid=6519/tid=6521] ppid=6517 vsize=781912 CPUtime=4883.11 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 15894 0 0 0 488265 46 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4660500 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=781912 CPUtime=4901.64 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15674 0 0 0 490127 37 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=781912 CPUtime=4900.58 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9602 0 0 0 490028 30 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=781912 CPUtime=4896.33 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12757 0 0 0 489594 39 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=781912 CPUtime=4901.31 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13243 0 0 0 490093 38 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4684063 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=781912 CPUtime=4880.3 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10625 0 0 0 487994 36 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4653368 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=781912 CPUtime=4893.98 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18416 0 0 0 489353 45 0 0 25 0 8 0 642877305 800677888 104655 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39158.9
Current children cumulated vsize (KiB) 781912

[startup+4962.3 s]
/proc/loadavg: 8.03 8.04 8.00 9/339 6675
/proc/meminfo: memFree=20387260/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=782192 CPUtime=39638.2 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 108642 0 0 0 3963506 310 0 0 25 0 8 0 642877303 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195548 104861 466 189 0 189897 0
[pid=6519/tid=6521] ppid=6517 vsize=782192 CPUtime=4942.43 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 15948 0 0 0 494197 46 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=782192 CPUtime=4961.64 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15699 0 0 0 496127 37 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=782192 CPUtime=4960.56 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9619 0 0 0 496026 30 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=782192 CPUtime=4956.33 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12810 0 0 0 495594 39 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=782192 CPUtime=4961.3 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13251 0 0 0 496092 38 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4660157 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=782192 CPUtime=4940.29 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10645 0 0 0 493993 36 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669560 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=782192 CPUtime=4953.97 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18421 0 0 0 495352 45 0 0 25 0 8 0 642877305 800964608 104861 33554432000 4194304 4964800 140736855020848 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39638.2
Current children cumulated vsize (KiB) 782192



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.6 s]
/proc/loadavg: 8.06 8.05 8.00 9/339 6676
/proc/meminfo: memFree=20386504/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=782192 CPUtime=40000.1 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 108932 0 0 0 3999698 311 0 0 25 0 8 0 642877303 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195548 105151 466 189 0 189897 0
[pid=6519/tid=6521] ppid=6517 vsize=782192 CPUtime=4987.24 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 16031 0 0 0 498677 47 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670674 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=782192 CPUtime=5006.95 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15700 0 0 0 500658 37 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=782192 CPUtime=5005.86 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9633 0 0 0 500556 30 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669283 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=782192 CPUtime=5001.64 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12812 0 0 0 500125 39 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=782192 CPUtime=5006.61 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13331 0 0 0 500623 38 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=782192 CPUtime=4985.6 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10675 0 0 0 498523 37 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=782192 CPUtime=4999.28 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18490 0 0 0 499883 45 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 782192

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.6 s]
/proc/loadavg: 8.06 8.05 8.00 9/339 6676
/proc/meminfo: memFree=20386504/32951124 swapFree=57171160/67111528
[pid=6519] ppid=6517 vsize=782192 CPUtime=40000.1 cores=0-7
/proc/6519/stat : 6519 (claspmt) R 6517 6519 6493 0 -1 4202496 108932 0 0 0 3999698 311 0 0 25 0 8 0 642877303 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/6519/statm: 195548 105151 466 189 0 189897 0
[pid=6519/tid=6521] ppid=6517 vsize=782192 CPUtime=4987.24 cores=0-7
/proc/6519/task/6521/stat : 6521 (claspmt) R 6517 6519 6493 0 -1 4202560 16031 0 0 0 498677 47 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670674 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=6519/tid=6522] ppid=6517 vsize=782192 CPUtime=5006.95 cores=0-7
/proc/6519/task/6522/stat : 6522 (claspmt) R 6517 6519 6493 0 -1 4202560 15700 0 0 0 500658 37 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=6519/tid=6523] ppid=6517 vsize=782192 CPUtime=5005.86 cores=0-7
/proc/6519/task/6523/stat : 6523 (claspmt) R 6517 6519 6493 0 -1 4202560 9633 0 0 0 500556 30 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4669283 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=6519/tid=6524] ppid=6517 vsize=782192 CPUtime=5001.64 cores=0-7
/proc/6519/task/6524/stat : 6524 (claspmt) R 6517 6519 6493 0 -1 4202560 12812 0 0 0 500125 39 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=6519/tid=6525] ppid=6517 vsize=782192 CPUtime=5006.61 cores=0-7
/proc/6519/task/6525/stat : 6525 (claspmt) R 6517 6519 6493 0 -1 4202560 13331 0 0 0 500623 38 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=6519/tid=6526] ppid=6517 vsize=782192 CPUtime=4985.6 cores=0-7
/proc/6519/task/6526/stat : 6526 (claspmt) R 6517 6519 6493 0 -1 4202560 10675 0 0 0 498523 37 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=6519/tid=6527] ppid=6517 vsize=782192 CPUtime=4999.28 cores=0-7
/proc/6519/task/6527/stat : 6527 (claspmt) R 6517 6519 6493 0 -1 4202560 18490 0 0 0 499883 45 0 0 25 0 8 0 642877305 800964608 105151 33554432000 4194304 4964800 140736855020848 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 782192

Child status: 0
Real time (s): 5007.69
CPU time (s): 40000.2
CPU user time (s): 39997.1
CPU system time (s): 3.14352
CPU usage (%): 798.776
Max. virtual memory (cumulated for all children) (KiB): 782192

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39997.1
system time used= 3.14352
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 108943
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 386120
involuntary context switches= 193055

runsolver used 16.1575 second user time and 38.4861 second system time

The end

Launcher Data 

Begin job on node143 at 2011-05-16 02:10:17
IDJOB=3381109
IDBENCH=83416
IDSOLVER=1865
FILE ID=node143/3381109-1305504617
RUNJOBID= node143-1305504617-6507
PBS_JOBID= 13325494
Free space on /tmp= 73544 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/64/aes_64_4_keyfind_1.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381109-1305504617/watcher-3381109-1305504617 -o /tmp/evaluation-result-3381109-1305504617/solver-3381109-1305504617 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381109-1305504617.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 07ea67092ce3fd4529e72f69434e3f99
RANDOM SEED=1205736360

node143.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      20807440 kB
Buffers:        181356 kB
Cached:        1271900 kB
SwapCached:      39776 kB
Active:        4292364 kB
Inactive:      7585716 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      20807440 kB
SwapTotal:    67111528 kB
SwapFree:     57171160 kB
Dirty:            1260 kB
Writeback:           0 kB
AnonPages:    10385460 kB
Mapped:          22604 kB
Slab:           135700 kB
PageTables:      62656 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 74731536 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73544 MiB
End job on node143 at 2011-05-16 03:33:47