Trace number 3370909

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.28 1204.23

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q918.75-n50-p30-s52376212.cnf
MD5SUMf34a5dd1cdce70a322dd395bcb5f1303
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2548
Number of clauses582679
Sum of the clauses size1167806
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2582629
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3370909-1305432948/watcher-3370909-1305432948 -o /tmp/evaluation-result-3370909-1305432948/solver-3370909-1305432948 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3370909-1305432948.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.84 7.53 7.83 2/364 11942
/proc/meminfo: memFree=25557248/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=10588 CPUtime=0 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) R 11940 11942 10794 0 -1 4202496 738 1252 0 0 0 0 0 0 18 0 1 0 635710472 10842112 669 33554432000 4194304 4417452 140733803015216 18446744073709551615 264726404864 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/11942/statm: 2647 669 147 55 0 533 0

[startup+0.0427219 s]
/proc/loadavg: 5.84 7.53 7.83 2/364 11942
/proc/meminfo: memFree=25557248/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=14244 CPUtime=0.03 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) R 11940 11942 10794 0 -1 4202496 1701 1252 0 0 3 0 0 0 18 0 1 0 635710472 14585856 1584 33554432000 4194304 4417452 140733803015216 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/11942/statm: 3561 1584 148 55 0 1447 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 14244

[startup+0.100711 s]
/proc/loadavg: 5.84 7.53 7.83 2/364 11942
/proc/meminfo: memFree=25557248/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=26532 CPUtime=0.09 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) R 11940 11942 10794 0 -1 4202496 4773 1252 0 0 9 0 0 0 18 0 1 0 635710472 27168768 4656 33554432000 4194304 4417452 140733803015216 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/11942/statm: 6633 4656 148 55 0 4519 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26532

[startup+0.300671 s]
/proc/loadavg: 5.84 7.53 7.83 2/364 11942
/proc/meminfo: memFree=25557248/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=75684 CPUtime=0.28 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) R 11940 11942 10794 0 -1 4202496 17061 1252 0 0 27 1 0 0 19 0 1 0 635710472 77500416 16944 33554432000 4194304 4417452 140733803015216 18446744073709551615 264726404864 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/11942/statm: 18921 16944 148 55 0 16807 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 75684

[startup+0.700556 s]
/proc/loadavg: 5.84 7.53 7.83 2/364 11942
/proc/meminfo: memFree=25557248/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=123128 CPUtime=1.63 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 19014 1252 0 0 161 2 0 0 19 0 5 0 635710472 126083072 18585 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 30782 18585 175 55 0 28668 0
Current children cumulated CPU time (s) 1.63
Current children cumulated vsize (KiB) 123128

[startup+1.5004 s]
/proc/loadavg: 5.84 7.53 7.83 9/373 11956
/proc/meminfo: memFree=25458480/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=133436 CPUtime=4.81 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 37389 1252 0 0 477 4 0 0 19 0 5 0 635710472 136638464 21146 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 33359 21146 180 55 0 31245 0
[pid=11942/tid=11953] ppid=11940 vsize=133436 CPUtime=1.11 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 4280 1252 0 0 111 0 0 0 25 0 5 0 635710510 136638464 21146 33554432000 4194304 4417452 140733803015216 18446744073709551615 4258241 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=133436 CPUtime=1.1 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 4398 1252 0 0 110 0 0 0 25 0 5 0 635710510 136638464 21146 33554432000 4194304 4417452 140733803015216 18446744073709551615 4250022 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=133436 CPUtime=1.1 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 5128 1252 0 0 110 0 0 0 25 0 5 0 635710510 136638464 21146 33554432000 4194304 4417452 140733803015216 18446744073709551615 4236448 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=133436 CPUtime=1.1 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 6510 1252 0 0 109 1 0 0 24 0 5 0 635710510 136638464 21146 33554432000 4194304 4417452 140733803015216 18446744073709551615 4250502 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4.81
Current children cumulated vsize (KiB) 133436

[startup+3.1011 s]
/proc/loadavg: 5.84 7.53 7.83 9/373 11956
/proc/meminfo: memFree=25442724/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=147676 CPUtime=11.21 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 49585 1252 0 0 1115 6 0 0 19 0 5 0 635710472 151220224 24706 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 36919 24706 180 55 0 34805 0
[pid=11942/tid=11953] ppid=11940 vsize=147676 CPUtime=2.71 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 5254 1252 0 0 271 0 0 0 25 0 5 0 635710510 151220224 24706 33554432000 4194304 4417452 140733803015216 18446744073709551615 4250072 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=147676 CPUtime=2.7 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 5372 1252 0 0 270 0 0 0 25 0 5 0 635710510 151220224 24706 33554432000 4194304 4417452 140733803015216 18446744073709551615 4249181 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=147676 CPUtime=2.7 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 5778 1252 0 0 270 0 0 0 25 0 5 0 635710510 151220224 24706 33554432000 4194304 4417452 140733803015216 18446744073709551615 4249909 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=147676 CPUtime=2.68 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 16108 1252 0 0 266 2 0 0 25 0 5 0 635710510 151220224 24706 33554432000 4194304 4417452 140733803015216 18446744073709551615 4320914 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.21
Current children cumulated vsize (KiB) 147676

[startup+6.3005 s]
/proc/loadavg: 6.01 7.54 7.83 9/373 11956
/proc/meminfo: memFree=25400316/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=181972 CPUtime=23.97 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 65711 1252 0 0 2388 9 0 0 19 0 5 0 635710472 186339328 33280 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 45493 33280 180 55 0 43379 0
[pid=11942/tid=11953] ppid=11940 vsize=181972 CPUtime=5.91 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 6553 1252 0 0 590 1 0 0 25 0 5 0 635710510 186339328 33280 33554432000 4194304 4417452 140733803015216 18446744073709551615 4250074 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=181972 CPUtime=5.9 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 6671 1252 0 0 590 0 0 0 25 0 5 0 635710510 186339328 33280 33554432000 4194304 4417452 140733803015216 18446744073709551615 4236414 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=181972 CPUtime=5.9 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 5778 1252 0 0 590 0 0 0 25 0 5 0 635710510 186339328 33280 33554432000 4194304 4417452 140733803015216 18446744073709551615 4250022 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=181972 CPUtime=5.84 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 29636 1252 0 0 580 4 0 0 25 0 5 0 635710510 186339328 33280 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.97
Current children cumulated vsize (KiB) 181972

[startup+12.7003 s]
/proc/loadavg: 6.17 7.55 7.83 9/373 11956
/proc/meminfo: memFree=25345604/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=197612 CPUtime=49.49 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 88623 1252 0 0 4937 12 0 0 19 0 5 0 635710472 202354688 36167 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 49403 36167 180 55 0 47289 0
[pid=11942/tid=11953] ppid=11940 vsize=197612 CPUtime=12.31 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 6553 1252 0 0 1230 1 0 0 25 0 5 0 635710510 202354688 36167 33554432000 4194304 4417452 140733803015216 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=197612 CPUtime=12.3 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 6671 1252 0 0 1230 0 0 0 25 0 5 0 635710510 202354688 36167 33554432000 4194304 4417452 140733803015216 18446744073709551615 4336288 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=197612 CPUtime=12.3 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 7409 1252 0 0 1230 0 0 0 25 0 5 0 635710510 202354688 36167 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=197612 CPUtime=12.16 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 50917 1252 0 0 1209 7 0 0 25 0 5 0 635710510 202354688 36167 33554432000 4194304 4417452 140733803015216 18446744073709551615 4353598 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.49
Current children cumulated vsize (KiB) 197612

[startup+25.5009 s]
/proc/loadavg: 6.57 7.57 7.83 10/373 11956
/proc/meminfo: memFree=25093132/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=331316 CPUtime=100.53 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 142627 1252 0 0 10034 19 0 0 19 0 5 0 635710472 339267584 54265 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 82829 54265 180 55 0 80715 0
[pid=11942/tid=11953] ppid=11940 vsize=331316 CPUtime=25.11 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 8201 1252 0 0 2510 1 0 0 25 0 5 0 635710510 339267584 54265 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=331316 CPUtime=25.11 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 12141 1252 0 0 2510 1 0 0 25 0 5 0 635710510 339267584 54265 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=331316 CPUtime=25.11 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 10033 1252 0 0 2510 1 0 0 25 0 5 0 635710510 339267584 54265 33554432000 4194304 4417452 140733803015216 18446744073709551615 4339868 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=331316 CPUtime=24.81 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 95179 1252 0 0 2467 14 0 0 25 0 5 0 635710510 339267584 54265 33554432000 4194304 4417452 140733803015216 18446744073709551615 4250502 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.53
Current children cumulated vsize (KiB) 331316

[startup+51.101 s]
/proc/loadavg: 7.20 7.64 7.84 10/373 11957
/proc/meminfo: memFree=25035028/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=396360 CPUtime=202.62 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 163122 1252 0 0 20241 21 0 0 19 0 5 0 635710472 405872640 70526 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 99090 70526 180 55 0 96976 0
[pid=11942/tid=11953] ppid=11940 vsize=396360 CPUtime=50.7 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 10424 1252 0 0 5069 1 0 0 25 0 5 0 635710510 405872640 70526 33554432000 4194304 4417452 140733803015216 18446744073709551615 4261379 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=396360 CPUtime=50.7 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 16470 1252 0 0 5069 1 0 0 25 0 5 0 635710510 405872640 70526 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364958 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=396360 CPUtime=50.7 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 15111 1252 0 0 5069 1 0 0 25 0 5 0 635710510 405872640 70526 33554432000 4194304 4417452 140733803015216 18446744073709551615 4249935 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=396360 CPUtime=50.11 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 104044 1252 0 0 4996 15 0 0 25 0 5 0 635710510 405872640 70526 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 202.62
Current children cumulated vsize (KiB) 396360

[startup+102.313 s]
/proc/loadavg: 7.65 7.69 7.85 10/373 11958
/proc/meminfo: memFree=24854208/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=418016 CPUtime=406.86 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 217220 1252 0 0 40657 29 0 0 19 0 5 0 635710472 428048384 75611 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 104504 75611 180 55 0 102390 0
[pid=11942/tid=11953] ppid=11940 vsize=418016 CPUtime=101.9 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 17988 1252 0 0 10188 2 0 0 25 0 5 0 635710510 428048384 75611 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=418016 CPUtime=101.92 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 18009 1252 0 0 10191 1 0 0 25 0 5 0 635710510 428048384 75611 33554432000 4194304 4417452 140733803015216 18446744073709551615 4308025 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=418016 CPUtime=101.92 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 15111 1252 0 0 10191 1 0 0 25 0 5 0 635710510 428048384 75611 33554432000 4194304 4417452 140733803015216 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=418016 CPUtime=100.71 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 149039 1252 0 0 10049 22 0 0 25 0 5 0 635710510 428048384 75611 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.86
Current children cumulated vsize (KiB) 418016

[startup+162.301 s]
/proc/loadavg: 8.01 7.78 7.86 9/373 11960
/proc/meminfo: memFree=24383592/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=652256 CPUtime=646.11 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 283792 1252 0 0 64573 38 0 0 19 0 5 0 635710472 667910144 134297 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 163064 134297 180 55 0 160950 0
[pid=11942/tid=11953] ppid=11940 vsize=652256 CPUtime=161.88 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 26449 1252 0 0 16185 3 0 0 25 0 5 0 635710510 667910144 134297 33554432000 4194304 4417452 140733803015216 18446744073709551615 4259618 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=652256 CPUtime=161.91 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 18241 1252 0 0 16190 1 0 0 25 0 5 0 635710510 667910144 134297 33554432000 4194304 4417452 140733803015216 18446744073709551615 4232680 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=652256 CPUtime=161.9 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 20508 1252 0 0 16189 1 0 0 25 0 5 0 635710510 667910144 134297 33554432000 4194304 4417452 140733803015216 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=652256 CPUtime=160 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 201521 1252 0 0 15971 29 0 0 25 0 5 0 635710510 667910144 134297 33554432000 4194304 4417452 140733803015216 18446744073709551615 4366074 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 646.11
Current children cumulated vsize (KiB) 652256


################
# More data... #
################

/proc/meminfo: memFree=23136536/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=1434576 CPUtime=3038.19 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1135897 1252 0 0 303646 173 0 0 19 0 5 0 635710472 1469005824 330080 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 358644 330080 180 55 0 356530 0
[pid=11942/tid=11953] ppid=11940 vsize=1434576 CPUtime=756.23 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 173112 1252 0 0 75600 23 0 0 25 0 5 0 635710510 1469005824 330080 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=1434576 CPUtime=761.94 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 140339 1252 0 0 76176 18 0 0 25 0 5 0 635710510 1469005824 330080 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=1434576 CPUtime=761.94 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 263524 1252 0 0 76158 36 0 0 25 0 5 0 635710510 1469005824 330080 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=1434576 CPUtime=757.68 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 541849 1252 0 0 75676 92 0 0 25 0 5 0 635710510 1469005824 330080 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3038.19
Current children cumulated vsize (KiB) 1434576

[startup+822.3 s]
/proc/loadavg: 8.18 8.13 8.00 9/373 11981
/proc/meminfo: memFree=22693692/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=1813824 CPUtime=3277.43 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1311488 1252 0 0 327547 196 0 0 19 0 5 0 635710472 1857355776 424892 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 453456 424892 180 55 0 451342 0
[pid=11942/tid=11953] ppid=11940 vsize=1813824 CPUtime=815.51 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 211973 1252 0 0 81523 28 0 0 25 0 5 0 635710510 1857355776 424892 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=1813824 CPUtime=821.93 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 187135 1252 0 0 82169 24 0 0 25 0 5 0 635710510 1857355776 424892 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=1813824 CPUtime=821.93 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 329060 1252 0 0 82148 45 0 0 25 0 5 0 635710510 1857355776 424892 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=1813824 CPUtime=817.66 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 566247 1252 0 0 81670 96 0 0 25 0 5 0 635710510 1857355776 424892 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3277.43
Current children cumulated vsize (KiB) 1813824

[startup+882.301 s]
/proc/loadavg: 8.24 8.15 8.02 9/373 11982
/proc/meminfo: memFree=22606600/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=1877692 CPUtime=3516.72 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1515798 1252 0 0 351444 228 0 0 19 0 5 0 635710472 1922756608 440859 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 469423 440859 180 55 0 467309 0
[pid=11942/tid=11953] ppid=11940 vsize=1877692 CPUtime=874.79 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 259104 1252 0 0 87442 37 0 0 25 0 5 0 635710510 1922756608 440859 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=1877692 CPUtime=881.93 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 252671 1252 0 0 88160 33 0 0 25 0 5 0 635710510 1922756608 440859 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=1877692 CPUtime=881.93 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 370020 1252 0 0 88143 50 0 0 25 0 5 0 635710510 1922756608 440859 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=1877692 CPUtime=877.66 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 616930 1252 0 0 87661 105 0 0 25 0 5 0 635710510 1922756608 440859 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364972 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3516.72
Current children cumulated vsize (KiB) 1877692

[startup+942.301 s]
/proc/loadavg: 8.09 8.12 8.01 9/373 11984
/proc/meminfo: memFree=22518404/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=2064176 CPUtime=3756.01 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1627955 1252 0 0 375355 246 0 0 19 0 5 0 635710472 2113716224 487480 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 516044 487480 180 55 0 513930 0
[pid=11942/tid=11953] ppid=11940 vsize=2064176 CPUtime=934.09 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 260736 1252 0 0 93372 37 0 0 25 0 5 0 635710510 2113716224 487480 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=2064176 CPUtime=941.94 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 277660 1252 0 0 94158 36 0 0 25 0 5 0 635710510 2113716224 487480 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364558 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=2064176 CPUtime=941.92 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 419172 1252 0 0 94136 56 0 0 25 0 5 0 635710510 2113716224 487480 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=2064176 CPUtime=937.64 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 653314 1252 0 0 93651 113 0 0 25 0 5 0 635710510 2113716224 487480 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3756.01
Current children cumulated vsize (KiB) 2064176

[startup+1002.3 s]
/proc/loadavg: 8.10 8.11 8.01 9/373 11985
/proc/meminfo: memFree=22319960/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=2278052 CPUtime=3995.29 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1706000 1252 0 0 399274 255 0 0 19 0 5 0 635710472 2332725248 540949 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 569513 540949 180 55 0 567399 0
[pid=11942/tid=11953] ppid=11940 vsize=2278052 CPUtime=993.38 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 264832 1252 0 0 99300 38 0 0 25 0 5 0 635710510 2332725248 540949 33554432000 4194304 4417452 140733803015216 18446744073709551615 4311285 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=2278052 CPUtime=1001.93 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 301053 1252 0 0 100155 38 0 0 25 0 5 0 635710510 2332725248 540949 33554432000 4194304 4417452 140733803015216 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=2278052 CPUtime=1001.93 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 466513 1252 0 0 100132 61 0 0 25 0 5 0 635710510 2332725248 540949 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=2278052 CPUtime=997.63 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 656529 1252 0 0 99650 113 0 0 25 0 5 0 635710510 2332725248 540949 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3995.29
Current children cumulated vsize (KiB) 2278052

[startup+1062.3 s]
/proc/loadavg: 8.03 8.09 8.00 9/373 11987
/proc/meminfo: memFree=22220480/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=2377232 CPUtime=4234.58 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1794549 1252 0 0 423190 268 0 0 19 0 5 0 635710472 2434285568 565744 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 594308 565744 180 55 0 592194 0
[pid=11942/tid=11953] ppid=11940 vsize=2377232 CPUtime=1052.67 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 343659 1252 0 0 105217 50 0 0 25 0 5 0 635710510 2434285568 565744 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=2377232 CPUtime=1061.94 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 310775 1252 0 0 106154 40 0 0 25 0 5 0 635710510 2434285568 565744 33554432000 4194304 4417452 140733803015216 18446744073709551615 4311122 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=2377232 CPUtime=1061.93 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 466513 1252 0 0 106132 61 0 0 25 0 5 0 635710510 2434285568 565744 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=2377232 CPUtime=1057.63 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 656529 1252 0 0 105650 113 0 0 25 0 5 0 635710510 2434285568 565744 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4234.58
Current children cumulated vsize (KiB) 2377232

[startup+1122.3 s]
/proc/loadavg: 8.14 8.10 8.01 9/373 11989
/proc/meminfo: memFree=21893448/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=2729912 CPUtime=4473.86 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 1950111 1252 0 0 447091 295 0 0 19 0 5 0 635710472 2795429888 653914 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 682478 653914 180 55 0 680364 0
[pid=11942/tid=11953] ppid=11940 vsize=2729912 CPUtime=1111.96 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 368235 1252 0 0 111140 56 0 0 25 0 5 0 635710510 2795429888 653914 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=2729912 CPUtime=1121.94 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 363692 1252 0 0 112145 49 0 0 25 0 5 0 635710510 2795429888 653914 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=2729912 CPUtime=1121.94 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 524086 1252 0 0 112124 70 0 0 25 0 5 0 635710510 2795429888 653914 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=2729912 CPUtime=1117.62 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 677025 1252 0 0 111645 117 0 0 25 0 5 0 635710510 2795429888 653914 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4473.86
Current children cumulated vsize (KiB) 2729912

[startup+1182.3 s]
/proc/loadavg: 8.05 8.08 8.00 9/373 11990
/proc/meminfo: memFree=21619996/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=2800228 CPUtime=4713.16 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 2045956 1252 0 0 471005 311 0 0 19 0 5 0 635710472 2867433472 671493 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 700057 671493 180 55 0 697943 0
[pid=11942/tid=11953] ppid=11940 vsize=2800228 CPUtime=1171.26 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 393788 1252 0 0 117067 59 0 0 25 0 5 0 635710510 2867433472 671493 33554432000 4194304 4417452 140733803015216 18446744073709551615 4366458 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=2800228 CPUtime=1181.94 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 366978 1252 0 0 118144 50 0 0 25 0 5 0 635710510 2867433472 671493 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=2800228 CPUtime=1181.94 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 524086 1252 0 0 118124 70 0 0 25 0 5 0 635710510 2867433472 671493 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=2800228 CPUtime=1177.61 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 744031 1252 0 0 117633 128 0 0 25 0 5 0 635710510 2867433472 671493 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4713.16
Current children cumulated vsize (KiB) 2800228



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.1 s]
/proc/loadavg: 8.03 8.07 8.00 5/367 11992
/proc/meminfo: memFree=22862872/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=2816928 CPUtime=4800.12 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) S 11940 11942 10794 0 -1 4202496 2050131 1252 0 0 479701 311 0 0 19 0 5 0 635710472 2884534272 675668 33554432000 4194304 4417452 140733803015216 18446744073709551615 264738207717 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 704232 675668 180 55 0 702118 0
[pid=11942/tid=11953] ppid=11940 vsize=2816928 CPUtime=1192.79 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4202560 393788 1252 0 0 119220 59 0 0 25 0 5 0 635710510 2884534272 675668 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11942/tid=11954] ppid=11940 vsize=2816928 CPUtime=1203.76 cores=0,2,4,6
/proc/11942/task/11954/stat : 11954 (plingeling) R 11940 11942 10794 0 -1 4202560 369312 1252 0 0 120325 51 0 0 25 0 5 0 635710510 2884534272 675668 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11942/tid=11955] ppid=11940 vsize=2816928 CPUtime=1203.75 cores=0,2,4,6
/proc/11942/task/11955/stat : 11955 (plingeling) R 11940 11942 10794 0 -1 4202560 524086 1252 0 0 120305 70 0 0 25 0 5 0 635710510 2884534272 675668 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11942/tid=11956] ppid=11940 vsize=2816928 CPUtime=1199.42 cores=0,2,4,6
/proc/11942/task/11956/stat : 11956 (plingeling) R 11940 11942 10794 0 -1 4202560 745872 1252 0 0 119814 128 0 0 25 0 5 0 635710510 2884534272 675668 33554432000 4194304 4417452 140733803015216 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 2816928

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.22 s]
/proc/loadavg: 8.03 8.07 8.00 5/367 11992
/proc/meminfo: memFree=22862872/32951124 swapFree=63083164/67111528
[pid=11942] ppid=11940 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/11942/stat : 11942 (plingeling) Z 11940 11942 10794 0 -1 4203548 2050131 1252 0 0 479704 321 0 0 18 0 2 0 635710472 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/11942/statm: 0 0 0 0 0 0 0
[pid=11942/tid=11953] ppid=11940 vsize=0 CPUtime=1192.9 cores=0,2,4,6
/proc/11942/task/11953/stat : 11953 (plingeling) R 11940 11942 10794 0 -1 4203588 393788 1252 0 0 119221 69 0 0 25 0 2 0 635710510 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1204.23
CPU time (s): 4800.28
CPU user time (s): 4797.05
CPU system time (s): 3.22951
CPU usage (%): 398.618
Max. virtual memory (cumulated for all children) (KiB): 2902140

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.05
system time used= 3.22951
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2051383
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 83
involuntary context switches= 25945

runsolver used 4.20936 second user time and 10.0315 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-15 06:15:49
IDJOB=3370909
IDBENCH=88204
IDSOLVER=1848
FILE ID=node141/3370909-1305432948
RUNJOBID= node141-1305432948-11917
PBS_JOBID= 13324603
Free space on /tmp= 73352 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q918.75-n50-p30-s52376212.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3370909-1305432948/watcher-3370909-1305432948 -o /tmp/evaluation-result-3370909-1305432948/solver-3370909-1305432948 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3370909-1305432948.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f34a5dd1cdce70a322dd395bcb5f1303
RANDOM SEED=1819185598

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25557288 kB
Buffers:        248504 kB
Cached:        4918020 kB
SwapCached:      15708 kB
Active:        2071876 kB
Inactive:      5081332 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25557288 kB
SwapTotal:    67111528 kB
SwapFree:     63083164 kB
Dirty:            9372 kB
Writeback:           0 kB
AnonPages:     1978724 kB
Mapped:          12808 kB
Slab:           137088 kB
PageTables:      38352 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 86269368 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73352 MiB
End job on node141 at 2011-05-15 06:35:55