Trace number 3381107

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.9 5009.87

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_10_keyfind_1.cnf
MD5SUMf35f3aee72237c1bf3d673feb4883756
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables8080
Number of clauses96704
Sum of the clauses size539496
Maximum clause length8
Minimum clause length1
Number of clauses of size 1336
Number of clauses of size 21152
Number of clauses of size 315040
Number of clauses of size 42944
Number of clauses of size 518432
Number of clauses of size over 558800

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...504462/instance-3381107-1305504462.cnf
0.19/0.23	c Solving...
40000.33/5009.42	
40000.33/5009.42	*** INTERRUPTED! ***
40000.33/5009.42	*** Shutting down threads...
40000.83/5009.86	s UNKNOWN
40000.83/5009.86	
40000.83/5009.86	c INTERRUPTED   : 1
40000.83/5009.86	c Models        : 0+    
40000.83/5009.86	c Time          : 5009.422s (Solving: 5009.19s 1st Model: 0.00s Unsat: 0.00s)
40000.83/5009.86	c CPU Time      : 40000.460s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381107-1305504462/watcher-3381107-1305504462 -o /tmp/evaluation-result-3381107-1305504462/solver-3381107-1305504462 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381107-1305504462.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.09 8.10 8.03 2/401 2262
/proc/meminfo: memFree=23070364/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=21888 CPUtime=0 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 674 0 0 0 0 0 0 0 24 0 1 0 662043288 22413312 605 33554432000 4194304 4964800 140736494147120 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/2262/statm: 5472 612 376 189 0 346 0

[startup+0.0787959 s]
/proc/loadavg: 8.09 8.10 8.03 2/401 2262
/proc/meminfo: memFree=23070364/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=33052 CPUtime=0.06 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 4036 0 0 0 6 0 0 0 24 0 1 0 662043288 33845248 3353 33554432000 4194304 4964800 140736494147120 18446744073709551615 4607641 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/2262/statm: 8263 3353 403 189 0 3137 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 33052

[startup+0.100783 s]
/proc/loadavg: 8.09 8.10 8.03 2/401 2262
/proc/meminfo: memFree=23070364/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=33052 CPUtime=0.09 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 4036 0 0 0 9 0 0 0 24 0 1 0 662043288 33845248 3353 33554432000 4194304 4964800 140736494147120 18446744073709551615 4607549 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/2262/statm: 8263 3353 403 189 0 3137 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 33052

[startup+0.300734 s]
/proc/loadavg: 8.09 8.10 8.03 2/401 2262
/proc/meminfo: memFree=23070364/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=466532 CPUtime=0.35 cores=0-7
/proc/2262/stat : 2262 (claspmt) S 2260 2262 2000 0 -1 4202496 14543 0 0 0 30 5 0 0 18 0 8 0 662043288 477728768 13294 33554432000 4194304 4964800 140736494147120 18446744073709551615 239488510873 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/2262/statm: 116633 13296 438 189 0 110982 0
Current children cumulated CPU time (s) 0.35
Current children cumulated vsize (KiB) 466532

[startup+1.1775 s]
/proc/loadavg: 8.09 8.10 8.03 2/401 2262
/proc/meminfo: memFree=23070364/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=468680 CPUtime=7.26 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 21253 0 0 0 715 11 0 0 19 0 8 0 662043288 479928320 20004 33554432000 4194304 4964800 140736494147120 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 117170 20004 464 189 0 111519 0
Current children cumulated CPU time (s) 7.26
Current children cumulated vsize (KiB) 468680

[startup+1.50044 s]
/proc/loadavg: 8.09 8.10 8.03 2/401 2262
/proc/meminfo: memFree=23070364/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=469076 CPUtime=9.83 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 22525 0 0 0 972 11 0 0 19 0 8 0 662043288 480333824 21276 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 117269 21276 464 189 0 111618 0
Current children cumulated CPU time (s) 9.83
Current children cumulated vsize (KiB) 469076

[startup+3.10113 s]
/proc/loadavg: 8.16 8.11 8.03 9/409 2270
/proc/meminfo: memFree=22973760/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=472328 CPUtime=22.6 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 26607 0 0 0 2246 14 0 0 24 0 8 0 662043288 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 118082 25358 464 189 0 112431 0
[pid=2262/tid=2264] ppid=2260 vsize=472328 CPUtime=2.8 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 1731 0 0 0 279 1 0 0 22 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4361745 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=472328 CPUtime=2.8 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 3816 0 0 0 278 2 0 0 23 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=472328 CPUtime=2.79 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 3383 0 0 0 276 3 0 0 22 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=472328 CPUtime=2.81 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 2219 0 0 0 280 1 0 0 23 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670338 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=472328 CPUtime=2.81 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 2869 0 0 0 280 1 0 0 23 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4627928 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=472328 CPUtime=2.8 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 3639 0 0 0 278 2 0 0 20 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=472328 CPUtime=2.81 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 3168 0 0 0 280 1 0 0 22 0 8 0 662043311 483663872 25358 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670351 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 22.6
Current children cumulated vsize (KiB) 472328

[startup+6.30052 s]
/proc/loadavg: 8.15 8.11 8.03 9/409 2270
/proc/meminfo: memFree=22950696/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=474468 CPUtime=48.15 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 31861 0 0 0 4798 17 0 0 25 0 8 0 662043288 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4370242 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 118617 30612 466 189 0 112966 0
[pid=2262/tid=2264] ppid=2260 vsize=474468 CPUtime=6 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 2106 0 0 0 599 1 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4364720 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=474468 CPUtime=6 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 4775 0 0 0 598 2 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=474468 CPUtime=5.98 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 4201 0 0 0 595 3 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670318 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=474468 CPUtime=6.01 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 2485 0 0 0 600 1 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4364717 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=474468 CPUtime=6.01 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 3584 0 0 0 600 1 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4364587 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=474468 CPUtime=6 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 4463 0 0 0 598 2 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=474468 CPUtime=6 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 4081 0 0 0 599 1 0 0 25 0 8 0 662043311 485855232 30612 33554432000 4194304 4964800 140736494147120 18446744073709551615 4660168 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 48.15
Current children cumulated vsize (KiB) 474468

[startup+12.7153 s]
/proc/loadavg: 8.14 8.11 8.03 9/409 2270
/proc/meminfo: memFree=22913204/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=480608 CPUtime=99.37 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 41451 0 0 0 9914 23 0 0 25 0 8 0 662043288 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4669376 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 120152 40202 466 189 0 114501 0
[pid=2262/tid=2264] ppid=2260 vsize=480608 CPUtime=12.41 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 2480 0 0 0 1240 1 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4370377 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=480608 CPUtime=12.42 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 6311 0 0 0 1238 4 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4378907 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=480608 CPUtime=12.4 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 6057 0 0 0 1236 4 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=480608 CPUtime=12.42 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 3255 0 0 0 1240 2 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=480608 CPUtime=12.43 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 4115 0 0 0 1241 2 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=480608 CPUtime=12.41 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 6219 0 0 0 1238 3 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=480608 CPUtime=12.42 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 6515 0 0 0 1239 3 0 0 25 0 8 0 662043311 492142592 40202 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.37
Current children cumulated vsize (KiB) 480608

[startup+25.5008 s]
/proc/loadavg: 8.12 8.10 8.03 9/409 2271
/proc/meminfo: memFree=22871688/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=488160 CPUtime=201.46 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 52369 0 0 0 20114 32 0 0 25 0 8 0 662043288 499875840 51115 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 122040 51116 466 189 0 116389 0
[pid=2262/tid=2264] ppid=2260 vsize=488160 CPUtime=25.19 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 3426 0 0 0 2517 2 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 4654549 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=488160 CPUtime=25.19 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 7979 0 0 0 2514 5 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=488160 CPUtime=25.18 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 9214 0 0 0 2512 6 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670403 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=488160 CPUtime=25.2 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 3782 0 0 0 2518 2 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 4334340 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=488160 CPUtime=25.21 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 4854 0 0 0 2518 3 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 4370352 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=488160 CPUtime=25.19 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 7810 0 0 0 2515 4 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=488160 CPUtime=25.2 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 8181 0 0 0 2517 3 0 0 25 0 8 0 662043311 499875840 51116 33554432000 4194304 4964800 140736494147120 18446744073709551615 239476356129 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 201.46
Current children cumulated vsize (KiB) 488160

[startup+51.1009 s]
/proc/loadavg: 8.07 8.09 8.03 9/409 2271
/proc/meminfo: memFree=22801852/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=499136 CPUtime=405.88 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 69570 0 0 0 40543 45 0 0 25 0 8 0 662043288 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670392 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 124784 67854 466 189 0 119133 0
[pid=2262/tid=2264] ppid=2260 vsize=499136 CPUtime=50.79 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 4594 0 0 0 5076 3 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=499136 CPUtime=50.81 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 10455 0 0 0 5074 7 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=499136 CPUtime=50.78 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 11557 0 0 0 5071 7 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4627488 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=499136 CPUtime=50.81 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 5282 0 0 0 5077 4 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4683957 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=499136 CPUtime=50.81 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 7008 0 0 0 5077 4 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=499136 CPUtime=50.8 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 10163 0 0 0 5073 7 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=499136 CPUtime=50.81 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 11633 0 0 0 5075 6 0 0 25 0 8 0 662043311 511115264 67854 33554432000 4194304 4964800 140736494147120 18446744073709551615 4658455 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 405.88
Current children cumulated vsize (KiB) 499136

[startup+102.301 s]
/proc/loadavg: 8.08 8.09 8.03 9/409 2273
/proc/meminfo: memFree=22714548/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=513416 CPUtime=814.71 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 91948 0 0 0 81408 63 0 0 25 0 8 0 662043288 525737984 90122 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 128354 90122 466 189 0 122703 0
[pid=2262/tid=2264] ppid=2260 vsize=513416 CPUtime=101.95 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 6504 0 0 0 10191 4 0 0 25 0 8 0 662043311 525737984 90122 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=513416 CPUtime=101.98 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 11996 0 0 0 10190 8 0 0 25 0 8 0 662043311 525737984 90122 33554432000 4194304 4964800 140736494147120 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=513416 CPUtime=101.95 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 14320 0 0 0 10186 9 0 0 25 0 8 0 662043311 525737984 90122 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670351 0 0 4096 18946 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 61524 0 0 0 465149 63 0 0 25 0 8 0 662043311 2389655552 526380 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2333648 CPUtime=4643.94 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 135640 0 0 0 464225 169 0 0 25 0 8 0 662043311 2389655552 526380 33554432000 4194304 4964800 140736494147120 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37228.5
Current children cumulated vsize (KiB) 2333648

[startup+4722.3 s]
/proc/loadavg: 8.06 8.07 8.02 9/409 2411
/proc/meminfo: memFree=20941948/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2339292 CPUtime=37707.5 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 542331 0 0 0 3770130 624 0 0 25 0 8 0 662043288 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2262/statm: 584823 531384 466 189 0 579172 0
[pid=2262/tid=2264] ppid=2260 vsize=2339292 CPUtime=4720.78 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 30789 0 0 0 472048 30 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=2339292 CPUtime=4713.7 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 37176 0 0 0 471334 36 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=2339292 CPUtime=4720.74 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 67020 0 0 0 472008 66 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=2339292 CPUtime=4719.52 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 54903 0 0 0 471884 68 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=2339292 CPUtime=4721.13 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 41437 0 0 0 472072 41 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4669338 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=2339292 CPUtime=4712.09 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 61846 0 0 0 471146 63 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2339292 CPUtime=4703.12 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 135672 0 0 0 470142 170 0 0 25 0 8 0 662043311 2395435008 531384 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37707.5
Current children cumulated vsize (KiB) 2339292

[startup+4782.32 s]
/proc/loadavg: 8.05 8.07 8.02 9/409 2412
/proc/meminfo: memFree=20936728/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2344080 CPUtime=38186.8 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 543690 0 0 0 3818055 626 0 0 25 0 8 0 662043288 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2262/statm: 586020 532743 466 189 0 580369 0
[pid=2262/tid=2264] ppid=2260 vsize=2344080 CPUtime=4780.8 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 30789 0 0 0 478050 30 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=2344080 CPUtime=4773.71 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 37177 0 0 0 477335 36 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4654449 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=2344080 CPUtime=4780.74 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 67317 0 0 0 478008 66 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=2344080 CPUtime=4779.53 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 54992 0 0 0 477885 68 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=2344080 CPUtime=4781.14 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 41439 0 0 0 478073 41 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=2344080 CPUtime=4772.11 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 62060 0 0 0 477147 64 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2344080 CPUtime=4762.31 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 135776 0 0 0 476061 170 0 0 25 0 8 0 662043311 2400337920 532743 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38186.8
Current children cumulated vsize (KiB) 2344080

[startup+4842.3 s]
/proc/loadavg: 8.09 8.07 8.02 9/409 2414
/proc/meminfo: memFree=20914288/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2360472 CPUtime=38665.8 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 549534 0 0 0 3865952 632 0 0 25 0 8 0 662043288 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2262/statm: 590118 538371 466 189 0 584467 0
[pid=2262/tid=2264] ppid=2260 vsize=2360472 CPUtime=4840.76 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 31040 0 0 0 484046 30 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4371992 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=2360472 CPUtime=4833.69 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 37206 0 0 0 483333 36 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=2360472 CPUtime=4840.71 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 67423 0 0 0 484005 66 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=2360472 CPUtime=4839.53 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 56177 0 0 0 483883 70 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=2360472 CPUtime=4841.13 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 42354 0 0 0 484071 42 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4668756 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=2360472 CPUtime=4832.1 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 62194 0 0 0 483146 64 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2360472 CPUtime=4822.17 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 136821 0 0 0 482046 171 0 0 25 0 8 0 662043311 2417123328 538371 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38665.8
Current children cumulated vsize (KiB) 2360472

[startup+4902.3 s]
/proc/loadavg: 8.10 8.07 8.01 9/409 2416
/proc/meminfo: memFree=20906840/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2360472 CPUtime=39145 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 551514 0 0 0 3913865 635 0 0 25 0 8 0 662043288 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670313 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2262/statm: 590118 540336 466 189 0 584467 0
[pid=2262/tid=2264] ppid=2260 vsize=2360472 CPUtime=4900.73 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 31054 0 0 0 490043 30 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=2360472 CPUtime=4893.69 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 37206 0 0 0 489333 36 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=2360472 CPUtime=4900.7 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 67515 0 0 0 490004 66 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=2360472 CPUtime=4899.53 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 56721 0 0 0 489882 71 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=2360472 CPUtime=4901.13 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 42356 0 0 0 490071 42 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=2360472 CPUtime=4892.1 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 62358 0 0 0 489146 64 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2360472 CPUtime=4881.48 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 137966 0 0 0 487976 172 0 0 25 0 8 0 662043311 2417123328 540336 33554432000 4194304 4964800 140736494147120 18446744073709551615 4658618 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39145
Current children cumulated vsize (KiB) 2360472

[startup+4962.3 s]
/proc/loadavg: 8.03 8.05 8.01 9/409 2417
/proc/meminfo: memFree=20899264/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2360472 CPUtime=39624.2 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 553414 0 0 0 3961780 639 0 0 25 0 8 0 662043288 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2262/statm: 590118 542235 466 189 0 584467 0
[pid=2262/tid=2264] ppid=2260 vsize=2360472 CPUtime=4960.72 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 31624 0 0 0 496042 30 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=2360472 CPUtime=4953.69 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 37206 0 0 0 495333 36 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=2360472 CPUtime=4960.7 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 67521 0 0 0 496004 66 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=2360472 CPUtime=4959.53 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 56721 0 0 0 495882 71 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=2360472 CPUtime=4961.12 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 42385 0 0 0 496070 42 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=2360472 CPUtime=4952.1 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 62777 0 0 0 495145 65 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2360472 CPUtime=4940.69 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 138842 0 0 0 493894 175 0 0 25 0 8 0 662043311 2417123328 542235 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39624.2
Current children cumulated vsize (KiB) 2360472



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.4 s]
/proc/loadavg: 8.09 8.06 8.01 9/409 2419
/proc/meminfo: memFree=20897640/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2360472 CPUtime=40000.3 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 553875 0 0 0 3999393 640 0 0 25 0 8 0 662043288 2417123328 542696 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670295 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2262/statm: 590118 542696 466 189 0 584467 0
[pid=2262/tid=2264] ppid=2260 vsize=2360472 CPUtime=5007.83 cores=0-7
/proc/2262/task/2264/stat : 2264 (claspmt) R 2260 2262 2000 0 -1 4202560 31631 0 0 0 500753 30 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2262/tid=2265] ppid=2260 vsize=2360472 CPUtime=5000.8 cores=0-7
/proc/2262/task/2265/stat : 2265 (claspmt) R 2260 2262 2000 0 -1 4202560 37206 0 0 0 500044 36 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2262/tid=2266] ppid=2260 vsize=2360472 CPUtime=5007.82 cores=0-7
/proc/2262/task/2266/stat : 2266 (claspmt) R 2260 2262 2000 0 -1 4202560 67528 0 0 0 500715 67 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4320872 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2262/tid=2267] ppid=2260 vsize=2360472 CPUtime=5006.63 cores=0-7
/proc/2262/task/2267/stat : 2267 (claspmt) R 2260 2262 2000 0 -1 4202560 56724 0 0 0 500592 71 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2262/tid=2268] ppid=2260 vsize=2360472 CPUtime=5008.24 cores=0-7
/proc/2262/task/2268/stat : 2268 (claspmt) R 2260 2262 2000 0 -1 4202560 42417 0 0 0 500782 42 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2262/tid=2269] ppid=2260 vsize=2360472 CPUtime=4999.22 cores=0-7
/proc/2262/task/2269/stat : 2269 (claspmt) R 2260 2262 2000 0 -1 4202560 62861 0 0 0 499857 65 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2262/tid=2270] ppid=2260 vsize=2360472 CPUtime=4987.14 cores=0-7
/proc/2262/task/2270/stat : 2270 (claspmt) R 2260 2262 2000 0 -1 4202560 139011 0 0 0 498539 175 0 0 25 0 8 0 662043311 2417123328 542698 33554432000 4194304 4964800 140736494147120 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 2360472

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.6 s]
/proc/loadavg: 8.09 8.06 8.01 9/409 2419
/proc/meminfo: memFree=20897640/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=2354596 CPUtime=40000.6 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 553887 0 0 0 3999423 641 0 0 25 0 1 0 662043288 2411106304 540841 33554432000 4194304 4964800 140736494147120 18446744073709551615 239476351978 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/2262/statm: 588649 540841 476 189 0 582998 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 2354596

[startup+5009.8 s]
/proc/loadavg: 8.09 8.06 8.01 9/409 2419
/proc/meminfo: memFree=20897640/32950928 swapFree=67056568/67111528
[pid=2262] ppid=2260 vsize=1561584 CPUtime=40000.8 cores=0-7
/proc/2262/stat : 2262 (claspmt) R 2260 2262 2000 0 -1 4202496 553887 0 0 0 3999439 644 0 0 25 0 1 0 662043288 1599062016 321828 33554432000 4194304 4964800 140736494147120 18446744073709551615 239476738183 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/2262/statm: 390396 321828 476 189 0 384745 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 1561584

Child status: 0
Real time (s): 5009.87
CPU time (s): 40000.9
CPU user time (s): 39994.4
CPU system time (s): 6.51001
CPU usage (%): 798.443
Max. virtual memory (cumulated for all children) (KiB): 2360472

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39994.4
system time used= 6.51001
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 553888
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 98937
involuntary context switches= 188813

runsolver used 19.3121 second user time and 48.5946 second system time

The end

Launcher Data 

Begin job on node148 at 2011-05-16 02:07:42
IDJOB=3381107
IDBENCH=83413
IDSOLVER=1865
FILE ID=node148/3381107-1305504462
RUNJOBID= node148-1305499433-2014
PBS_JOBID= 13325451
Free space on /tmp= 72856 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_10_keyfind_1.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381107-1305504462/watcher-3381107-1305504462 -o /tmp/evaluation-result-3381107-1305504462/solver-3381107-1305504462 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381107-1305504462.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f35f3aee72237c1bf3d673feb4883756
RANDOM SEED=993523788

node148.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23070528 kB
Buffers:        347148 kB
Cached:        4993204 kB
SwapCached:       8488 kB
Active:        6176208 kB
Inactive:      3341228 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23070528 kB
SwapTotal:    67111528 kB
SwapFree:     67056568 kB
Dirty:            3892 kB
Writeback:           0 kB
AnonPages:     4176224 kB
Mapped:          31940 kB
Slab:           258672 kB
PageTables:      36840 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 105087404 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72852 MiB
End job on node148 at 2011-05-16 03:31:14