Trace number 3309852

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 40000.3 5003.69

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_3_keyfind_1.cnf
MD5SUMff59ad0e88befc51bfa041e45143dc53
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2808
Number of clauses30432
Sum of the clauses size167020
Maximum clause length8
Minimum clause length1
Number of clauses of size 1280
Number of clauses of size 2384
Number of clauses of size 35024
Number of clauses of size 4960
Number of clauses of size 56144
Number of clauses of size over 517640

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3309852-1304011894/watcher-3309852-1304011894 -o /tmp/evaluation-result-3309852-1304011894/solver-3309852-1304011894 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3309852-1304011894.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.09 8.06 7.99 2/170 26988
/proc/meminfo: memFree=15194276/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=9876 CPUtime=0 cores=0-7
/proc/26988/stat : 26988 (plingeling) R 26986 26988 26704 0 -1 4202496 563 1251 0 0 0 0 0 0 24 0 1 0 512789016 10113024 491 33554432000 4194304 4417332 140736609010304 18446744073709551615 274473661387 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26988/statm: 2469 491 147 55 0 355 0

[startup+0.083811 s]
/proc/loadavg: 8.09 8.06 7.99 2/170 26988
/proc/meminfo: memFree=15194276/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=27764 CPUtime=0.07 cores=0-7
/proc/26988/stat : 26988 (plingeling) R 26986 26988 26704 0 -1 4202496 5036 1251 0 0 7 0 0 0 24 0 1 0 512789016 28430336 4964 33554432000 4194304 4417332 140736609010304 18446744073709551615 274473967360 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26988/statm: 6941 4964 148 55 0 4827 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 27764

[startup+0.893669 s]
/proc/loadavg: 8.09 8.06 7.99 2/170 26988
/proc/meminfo: memFree=15194276/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=184692 CPUtime=6.3 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 9192 1251 0 0 629 1 0 0 25 0 9 0 512789016 189124608 7461 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 46173 7461 180 55 0 44059 0
Current children cumulated CPU time (s) 6.3
Current children cumulated vsize (KiB) 184692

[startup+0.993653 s]
/proc/loadavg: 8.09 8.06 7.99 2/170 26988
/proc/meminfo: memFree=15194276/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=186660 CPUtime=7.1 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 9923 1251 0 0 709 1 0 0 25 0 9 0 512789016 191139840 7949 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 46665 7949 180 55 0 44551 0
Current children cumulated CPU time (s) 7.1
Current children cumulated vsize (KiB) 186660

[startup+1.09364 s]
/proc/loadavg: 8.09 8.06 7.99 2/170 26988
/proc/meminfo: memFree=15194276/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=188092 CPUtime=7.9 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 10923 1251 0 0 789 1 0 0 25 0 9 0 512789016 192606208 8307 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 47023 8307 180 55 0 44909 0
Current children cumulated CPU time (s) 7.9
Current children cumulated vsize (KiB) 188092

[startup+1.50056 s]
/proc/loadavg: 8.09 8.06 7.99 2/170 26988
/proc/meminfo: memFree=15194276/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=192908 CPUtime=11.16 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 15161 1251 0 0 1114 2 0 0 25 0 9 0 512789016 197537792 9543 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 48227 9543 180 55 0 46113 0
Current children cumulated CPU time (s) 11.16
Current children cumulated vsize (KiB) 192908

[startup+3.10028 s]
/proc/loadavg: 8.08 8.06 7.99 9/179 27003
/proc/meminfo: memFree=15147588/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=201668 CPUtime=23.95 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 31093 1251 0 0 2389 6 0 0 25 0 9 0 512789016 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 50417 11657 180 55 0 48303 0
[pid=26988/tid=26995] ppid=26986 vsize=201668 CPUtime=2.99 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 3355 1251 0 0 299 0 0 0 24 0 9 0 512789025 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 4249110 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=201668 CPUtime=2.99 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 2830 1251 0 0 299 0 0 0 25 0 9 0 512789025 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 4236816 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=201668 CPUtime=3 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 5486 1251 0 0 299 1 0 0 25 0 9 0 512789025 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364568 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=201668 CPUtime=2.98 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 2801 1251 0 0 298 0 0 0 25 0 9 0 512789025 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 4251028 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=201668 CPUtime=3 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 2857 1251 0 0 300 0 0 0 25 0 9 0 512789026 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=201668 CPUtime=2.99 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 2300 1251 0 0 299 0 0 0 25 0 9 0 512789026 206508032 11657 33554432000 4194304 4417332 140736609010304 18446744073709551615 4251045 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=201744 CPUtime=2.91 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 2740 1251 0 0 291 0 0 0 25 0 9 0 512789034 206585856 11676 33554432000 4194304 4417332 140736609010304 18446744073709551615 274473661396 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=201744 CPUtime=2.91 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 3691 1251 0 0 291 0 0 0 25 0 9 0 512789034 206585856 11676 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.95
Current children cumulated vsize (KiB) 201668

[startup+6.30072 s]
/proc/loadavg: 8.08 8.06 7.99 9/179 27003
/proc/meminfo: memFree=15123212/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=228104 CPUtime=49.53 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 63169 1251 0 0 4940 13 0 0 25 0 9 0 512789016 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 57026 18264 180 55 0 54912 0
[pid=26988/tid=26995] ppid=26986 vsize=228104 CPUtime=6.19 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 8314 1251 0 0 618 1 0 0 25 0 9 0 512789025 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4357348 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=228104 CPUtime=6.19 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 7176 1251 0 0 618 1 0 0 25 0 9 0 512789025 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4353665 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=228104 CPUtime=6.19 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 10399 1251 0 0 617 2 0 0 25 0 9 0 512789025 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364724 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=228104 CPUtime=6.17 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 4643 1251 0 0 616 1 0 0 25 0 9 0 512789025 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364639 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=228104 CPUtime=6.19 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 5627 1251 0 0 619 0 0 0 25 0 9 0 512789026 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=228104 CPUtime=6.19 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 5227 1251 0 0 618 1 0 0 25 0 9 0 512789026 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=228104 CPUtime=6.11 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 8853 1251 0 0 610 1 0 0 25 0 9 0 512789034 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4236982 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=228104 CPUtime=6.11 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 7878 1251 0 0 610 1 0 0 25 0 9 0 512789034 233578496 18264 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.53
Current children cumulated vsize (KiB) 228104

[startup+12.7006 s]
/proc/loadavg: 8.07 8.05 7.99 9/179 27003
/proc/meminfo: memFree=15108012/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=242576 CPUtime=100.73 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 113398 1251 0 0 10048 25 0 0 25 0 9 0 512789016 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 60644 21914 180 55 0 58530 0
[pid=26988/tid=26995] ppid=26986 vsize=242576 CPUtime=12.6 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 16467 1251 0 0 1257 3 0 0 25 0 9 0 512789025 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364736 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=242576 CPUtime=12.6 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 13282 1251 0 0 1257 3 0 0 25 0 9 0 512789025 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=242576 CPUtime=12.61 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 17658 1251 0 0 1257 4 0 0 25 0 9 0 512789025 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364608 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=242576 CPUtime=12.55 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 7972 1251 0 0 1253 2 0 0 25 0 9 0 512789025 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364531 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=242576 CPUtime=12.6 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 12156 1251 0 0 1258 2 0 0 25 0 9 0 512789026 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364502 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=242576 CPUtime=12.59 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 8220 1251 0 0 1258 1 0 0 25 0 9 0 512789026 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=242576 CPUtime=12.52 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 17907 1251 0 0 1249 3 0 0 25 0 9 0 512789034 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=242576 CPUtime=12.52 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 14684 1251 0 0 1249 3 0 0 25 0 9 0 512789034 248397824 21914 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364713 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.73
Current children cumulated vsize (KiB) 242576

[startup+25.5003 s]
/proc/loadavg: 8.06 8.05 7.99 9/179 27003
/proc/meminfo: memFree=15084928/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=269784 CPUtime=203.03 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 187970 1251 0 0 20263 40 0 0 25 0 9 0 512789016 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 67446 28685 180 55 0 65332 0
[pid=26988/tid=26995] ppid=26986 vsize=269784 CPUtime=25.39 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 30931 1251 0 0 2533 6 0 0 25 0 9 0 512789025 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=269784 CPUtime=25.4 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 24173 1251 0 0 2534 6 0 0 25 0 9 0 512789025 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364541 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=269784 CPUtime=25.4 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 28637 1251 0 0 2534 6 0 0 25 0 9 0 512789025 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4354300 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=269784 CPUtime=25.28 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 12593 1251 0 0 2525 3 0 0 25 0 9 0 512789025 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364919 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=269784 CPUtime=25.4 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 17816 1251 0 0 2537 3 0 0 25 0 9 0 512789026 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4354385 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=269784 CPUtime=25.39 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 15416 1251 0 0 2536 3 0 0 25 0 9 0 512789026 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364800 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=269784 CPUtime=25.31 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 26730 1251 0 0 2527 4 0 0 25 0 9 0 512789034 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4354322 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=269784 CPUtime=25.31 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 26622 1251 0 0 2526 5 0 0 25 0 9 0 512789034 276258816 28685 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.03
Current children cumulated vsize (KiB) 269784

[startup+51.1008 s]
/proc/loadavg: 8.10 8.06 8.00 9/179 27004
/proc/meminfo: memFree=15055792/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=316080 CPUtime=407.7 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 294034 1251 0 0 40707 63 0 0 25 0 9 0 512789016 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 79020 40259 180 55 0 76906 0
[pid=26988/tid=26995] ppid=26986 vsize=316080 CPUtime=50.99 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 46502 1251 0 0 5089 10 0 0 25 0 9 0 512789025 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366004 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=316080 CPUtime=51 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 35928 1251 0 0 5092 8 0 0 25 0 9 0 512789025 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=316080 CPUtime=50.99 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 46172 1251 0 0 5090 9 0 0 25 0 9 0 512789025 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=316080 CPUtime=50.74 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 19537 1251 0 0 5070 4 0 0 25 0 9 0 512789025 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=316080 CPUtime=51 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 26688 1251 0 0 5095 5 0 0 25 0 9 0 512789026 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=316080 CPUtime=50.99 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 24347 1251 0 0 5094 5 0 0 25 0 9 0 512789026 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=316080 CPUtime=50.91 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 41010 1251 0 0 5083 8 0 0 25 0 9 0 512789034 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=316080 CPUtime=50.91 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 48798 1251 0 0 5081 10 0 0 25 0 9 0 512789034 323665920 40259 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366061 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 407.7
Current children cumulated vsize (KiB) 316080

[startup+102.301 s]
/proc/loadavg: 8.12 8.07 8.00 9/179 27005

################
# More data... #
################

/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 469627 32 0 0 25 0 9 0 512789025 1894567936 431095 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1850164 CPUtime=4722.29 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 305386 1251 0 0 472173 56 0 0 25 0 9 0 512789026 1894567936 431095 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364652 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1850164 CPUtime=4722.16 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 470238 1251 0 0 472114 102 0 0 25 0 9 0 512789026 1894567936 431095 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366716 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1850164 CPUtime=4721.62 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 497591 1251 0 0 472076 86 0 0 25 0 9 0 512789034 1894567936 431095 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1850164 CPUtime=4721.57 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 575713 1251 0 0 472051 106 0 0 25 0 9 0 512789034 1894567936 431095 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37751.2
Current children cumulated vsize (KiB) 1850164

[startup+4782.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/179 27142
/proc/meminfo: memFree=13421948/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=1911616 CPUtime=38230.9 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 4133573 1251 0 0 3822351 744 0 0 25 0 9 0 512789016 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 477904 443112 180 55 0 475790 0
[pid=26988/tid=26995] ppid=26986 vsize=1911616 CPUtime=4782.28 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 1340288 1251 0 0 478015 213 0 0 25 0 9 0 512789025 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=1911616 CPUtime=4782.29 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 270687 1251 0 0 478178 51 0 0 25 0 9 0 512789025 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=1911616 CPUtime=4782.28 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 442556 1251 0 0 478147 81 0 0 25 0 9 0 512789025 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=1911616 CPUtime=4756.26 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 475594 32 0 0 25 0 9 0 512789025 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4357612 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1911616 CPUtime=4782.3 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 308746 1251 0 0 478173 57 0 0 25 0 9 0 512789026 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1911616 CPUtime=4782.16 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 493670 1251 0 0 478107 109 0 0 25 0 9 0 512789026 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1911616 CPUtime=4781.63 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 504180 1251 0 0 478075 88 0 0 25 0 9 0 512789034 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1911616 CPUtime=4781.58 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 580896 1251 0 0 478050 108 0 0 25 0 9 0 512789034 1957494784 443112 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38231
Current children cumulated vsize (KiB) 1911616

[startup+4842.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/179 27143
/proc/meminfo: memFree=13493320/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=1812188 CPUtime=38710.6 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 4192662 1251 0 0 3870304 755 0 0 25 0 9 0 512789016 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 453047 425327 180 55 0 450933 0
[pid=26988/tid=26995] ppid=26986 vsize=1812188 CPUtime=4842.28 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 1356229 1251 0 0 484014 214 0 0 25 0 9 0 512789025 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364722 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=1812188 CPUtime=4842.29 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 276059 1251 0 0 484177 52 0 0 25 0 9 0 512789025 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=1812188 CPUtime=4842.28 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 458940 1251 0 0 484142 86 0 0 25 0 9 0 512789025 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=1812188 CPUtime=4815.94 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 481562 32 0 0 25 0 9 0 512789025 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364565 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1812188 CPUtime=4842.29 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 311439 1251 0 0 484172 57 0 0 25 0 9 0 512789026 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1812188 CPUtime=4842.15 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 494301 1251 0 0 484105 110 0 0 25 0 9 0 512789026 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1812188 CPUtime=4841.62 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 517358 1251 0 0 484072 90 0 0 25 0 9 0 512789034 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1812188 CPUtime=4841.57 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 585786 1251 0 0 484049 108 0 0 25 0 9 0 512789034 1855680512 425327 33554432000 4194304 4417332 140736609010304 18446744073709551615 4356738 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38710.6
Current children cumulated vsize (KiB) 1812188

[startup+4902.31 s]
/proc/loadavg: 8.05 8.05 8.00 9/179 27145
/proc/meminfo: memFree=13639604/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=1690684 CPUtime=39190.3 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 4227437 1251 0 0 3918267 763 0 0 25 0 9 0 512789016 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 422671 388830 180 55 0 420557 0
[pid=26988/tid=26995] ppid=26986 vsize=1690684 CPUtime=4902.29 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 1370723 1251 0 0 490011 218 0 0 25 0 9 0 512789025 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364541 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=1690684 CPUtime=4902.29 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 280755 1251 0 0 490177 52 0 0 25 0 9 0 512789025 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=1690684 CPUtime=4902.28 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 458940 1251 0 0 490142 86 0 0 25 0 9 0 512789025 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=1690684 CPUtime=4875.61 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 487529 32 0 0 25 0 9 0 512789025 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4365977 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1690684 CPUtime=4902.3 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 317385 1251 0 0 490171 59 0 0 25 0 9 0 512789026 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1690684 CPUtime=4902.16 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 498040 1251 0 0 490105 111 0 0 25 0 9 0 512789026 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1690684 CPUtime=4901.61 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 519082 1251 0 0 490070 91 0 0 25 0 9 0 512789034 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1690684 CPUtime=4901.58 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 589962 1251 0 0 490049 109 0 0 25 0 9 0 512789034 1731260416 388830 33554432000 4194304 4417332 140736609010304 18446744073709551615 4365963 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39190.3
Current children cumulated vsize (KiB) 1690684

[startup+4962.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/179 27147
/proc/meminfo: memFree=13615496/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=1723980 CPUtime=39669.9 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 4264822 1251 0 0 3966226 768 0 0 25 0 9 0 512789016 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 430995 395459 180 55 0 428881 0
[pid=26988/tid=26995] ppid=26986 vsize=1723980 CPUtime=4962.28 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 1389101 1251 0 0 496008 220 0 0 25 0 9 0 512789025 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=1723980 CPUtime=4962.29 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 283926 1251 0 0 496176 53 0 0 25 0 9 0 512789025 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=1723980 CPUtime=4962.28 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 458940 1251 0 0 496142 86 0 0 25 0 9 0 512789025 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4366303 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=1723980 CPUtime=4935.28 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 493496 32 0 0 25 0 9 0 512789025 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1723980 CPUtime=4962.3 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 320607 1251 0 0 496171 59 0 0 25 0 9 0 512789026 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1723980 CPUtime=4962.15 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 501357 1251 0 0 496104 111 0 0 25 0 9 0 512789026 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1723980 CPUtime=4961.61 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 522348 1251 0 0 496069 92 0 0 25 0 9 0 512789034 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1723980 CPUtime=4961.57 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 595993 1251 0 0 496047 110 0 0 25 0 9 0 512789034 1765355520 395459 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364722 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39669.9
Current children cumulated vsize (KiB) 1723980



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.6 s]
/proc/loadavg: 8.01 8.03 8.00 9/179 27148
/proc/meminfo: memFree=13556304/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=1767260 CPUtime=40000.1 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 4308481 1251 0 0 3999242 773 0 0 25 0 9 0 512789016 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 441815 409423 180 55 0 439701 0
[pid=26988/tid=26995] ppid=26986 vsize=1767260 CPUtime=5003.59 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 1402228 1251 0 0 500137 222 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=1767260 CPUtime=5003.6 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 285977 1251 0 0 500307 53 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=1767260 CPUtime=5003.59 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 471516 1251 0 0 500271 88 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=1767260 CPUtime=4976.36 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 497604 32 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1767260 CPUtime=5003.61 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 325642 1251 0 0 500301 60 0 0 25 0 9 0 512789026 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1767260 CPUtime=5003.46 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 508323 1251 0 0 500234 112 0 0 25 0 9 0 512789026 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1767260 CPUtime=5002.91 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 524972 1251 0 0 500199 92 0 0 25 0 9 0 512789034 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1767260 CPUtime=5002.89 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 597273 1251 0 0 500178 111 0 0 25 0 9 0 512789034 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 1767260

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5003.6 s]
/proc/loadavg: 8.01 8.03 8.00 9/179 27148
/proc/meminfo: memFree=13556304/32950928 swapFree=67111528/67111528
[pid=26988] ppid=26986 vsize=1767260 CPUtime=40000.1 cores=0-7
/proc/26988/stat : 26988 (plingeling) S 26986 26988 26704 0 -1 4202496 4308481 1251 0 0 3999242 773 0 0 25 0 9 0 512789016 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/26988/statm: 441815 409423 180 55 0 439701 0
[pid=26988/tid=26995] ppid=26986 vsize=1767260 CPUtime=5003.59 cores=0-7
/proc/26988/task/26995/stat : 26995 (plingeling) R 26986 26988 26704 0 -1 4202560 1402228 1251 0 0 500137 222 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26988/tid=26996] ppid=26986 vsize=1767260 CPUtime=5003.6 cores=0-7
/proc/26988/task/26996/stat : 26996 (plingeling) R 26986 26988 26704 0 -1 4202560 285977 1251 0 0 500307 53 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26988/tid=26997] ppid=26986 vsize=1767260 CPUtime=5003.59 cores=0-7
/proc/26988/task/26997/stat : 26997 (plingeling) R 26986 26988 26704 0 -1 4202560 471516 1251 0 0 500271 88 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26988/tid=26998] ppid=26986 vsize=1767260 CPUtime=4976.36 cores=0-7
/proc/26988/task/26998/stat : 26998 (plingeling) R 26986 26988 26704 0 -1 4202560 187498 1251 0 0 497604 32 0 0 25 0 9 0 512789025 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26988/tid=26999] ppid=26986 vsize=1767260 CPUtime=5003.61 cores=0-7
/proc/26988/task/26999/stat : 26999 (plingeling) R 26986 26988 26704 0 -1 4202560 325642 1251 0 0 500301 60 0 0 25 0 9 0 512789026 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26988/tid=27000] ppid=26986 vsize=1767260 CPUtime=5003.46 cores=0-7
/proc/26988/task/27000/stat : 27000 (plingeling) R 26986 26988 26704 0 -1 4202560 508323 1251 0 0 500234 112 0 0 25 0 9 0 512789026 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26988/tid=27001] ppid=26986 vsize=1767260 CPUtime=5002.91 cores=0-7
/proc/26988/task/27001/stat : 27001 (plingeling) R 26986 26988 26704 0 -1 4202560 524972 1251 0 0 500199 92 0 0 25 0 9 0 512789034 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26988/tid=27002] ppid=26986 vsize=1767260 CPUtime=5002.89 cores=0-7
/proc/26988/task/27002/stat : 27002 (plingeling) R 26986 26988 26704 0 -1 4202560 597273 1251 0 0 500178 111 0 0 25 0 9 0 512789034 1809674240 409423 33554432000 4194304 4417332 140736609010304 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 1767260

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5003.69
CPU time (s): 40000.3
CPU user time (s): 39992.5
CPU system time (s): 7.82181
CPU usage (%): 799.415
Max. virtual memory (cumulated for all children) (KiB): 2102844

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.5
system time used= 7.82181
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4309732
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 414
involuntary context switches= 191284

runsolver used 8.05977 second user time and 19.2111 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-28 19:31:34
IDJOB=3309852
IDBENCH=83408
IDSOLVER=1695
FILE ID=node125/3309852-1304011894
RUNJOBID= node125-1304006221-26718
PBS_JOBID= 13168823
Free space on /tmp= 72764 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_3_keyfind_1.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3309852-1304011894/watcher-3309852-1304011894 -o /tmp/evaluation-result-3309852-1304011894/solver-3309852-1304011894 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3309852-1304011894.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ff59ad0e88befc51bfa041e45143dc53
RANDOM SEED=967777305

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15194424 kB
Buffers:       2213936 kB
Cached:       14942576 kB
SwapCached:          0 kB
Active:        8759816 kB
Inactive:      8463256 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15194424 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            1176 kB
Writeback:           0 kB
AnonPages:       66452 kB
Mapped:          14708 kB
Slab:           467984 kB
PageTables:       3880 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186744 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72764 MiB
End job on node125 at 2011-04-28 20:54:59