Trace number 3394874

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40001 5010.85

General information on the benchmark

NameSAT05/crafted/sorge05/QG7a/
gensys-ukn004.shuffled-as.sat05-3844.cnf
MD5SUM7d96ba103689bec761c9e7df7b158323
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark1407.09
Satisfiable
(Un)Satisfiability was proved
Number of variables2731
Number of clauses17983
Sum of the clauses size126997
Maximum clause length13
Minimum clause length1
Number of clauses of size 142
Number of clauses of size 2442
Number of clauses of size 32370
Number of clauses of size 4477
Number of clauses of size 5642
Number of clauses of size over 514010

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394874-1306108431/watcher-3394874-1306108431 -o /tmp/evaluation-result-3394874-1306108431/solver-3394874-1306108431 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394874-1306108431.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.05 8.06 8.00 2/491 27826
/proc/meminfo: memFree=28065572/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=13160 CPUtime=0.01 cores=0-7
/proc/27826/stat : 27826 (plingeling) R 27824 27826 27522 0 -1 4202496 1367 1251 0 0 1 0 0 0 18 0 1 0 722444031 13475840 1297 33554432000 4194304 4417452 140735321457872 18446744073709551615 270613110528 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/27826/statm: 3290 1297 148 55 0 1176 0

[startup+0.0476281 s]
/proc/loadavg: 8.05 8.06 8.00 2/491 27826
/proc/meminfo: memFree=28065572/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=17864 CPUtime=0.04 cores=0-7
/proc/27826/stat : 27826 (plingeling) R 27824 27826 27522 0 -1 4202496 2608 1251 0 0 4 0 0 0 18 0 1 0 722444031 18292736 2490 33554432000 4194304 4417452 140735321457872 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/27826/statm: 4466 2490 149 55 0 2352 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 17864

[startup+0.100572 s]
/proc/loadavg: 8.05 8.06 8.00 2/491 27826
/proc/meminfo: memFree=28065572/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=102364 CPUtime=0.21 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 4588 1251 0 0 21 0 0 0 18 0 9 0 722444031 104820736 3140 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 25591 3168 174 55 0 23477 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 102364

[startup+0.300535 s]
/proc/loadavg: 8.05 8.06 8.00 2/491 27826
/proc/meminfo: memFree=28065572/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=174456 CPUtime=1.8 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 6535 1251 0 0 180 0 0 0 18 0 9 0 722444031 178642944 4844 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 43614 4844 180 55 0 41500 0
Current children cumulated CPU time (s) 1.8
Current children cumulated vsize (KiB) 174456

[startup+0.700465 s]
/proc/loadavg: 8.05 8.06 8.00 2/491 27826
/proc/meminfo: memFree=28065572/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=182632 CPUtime=5 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 10475 1251 0 0 499 1 0 0 18 0 9 0 722444031 187015168 6931 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 45658 6931 180 55 0 43544 0
Current children cumulated CPU time (s) 5
Current children cumulated vsize (KiB) 182632

[startup+1.50033 s]
/proc/loadavg: 7.57 7.96 7.97 9/500 27840
/proc/meminfo: memFree=28030784/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=191148 CPUtime=11.39 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 21815 1251 0 0 1135 4 0 0 18 0 9 0 722444031 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 47787 9039 180 55 0 45673 0
[pid=27826/tid=27833] ppid=27824 vsize=191148 CPUtime=1.4 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 1794 1251 0 0 140 0 0 0 25 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=191148 CPUtime=1.39 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 2064 1251 0 0 139 0 0 0 25 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=191148 CPUtime=1.41 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 2044 1251 0 0 141 0 0 0 20 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=191148 CPUtime=1.41 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 3000 1251 0 0 141 0 0 0 20 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366655 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=191148 CPUtime=1.41 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 2128 1251 0 0 141 0 0 0 23 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=191148 CPUtime=1.4 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 2408 1251 0 0 140 0 0 0 23 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4354401 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=191148 CPUtime=1.4 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 1985 1251 0 0 140 0 0 0 23 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=191148 CPUtime=1.41 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 2101 1251 0 0 141 0 0 0 19 0 9 0 722444039 195735552 9039 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.39
Current children cumulated vsize (KiB) 191148

[startup+3.10104 s]
/proc/loadavg: 7.57 7.96 7.97 9/500 27840
/proc/meminfo: memFree=28015036/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=216552 CPUtime=24.17 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 43882 1251 0 0 2408 9 0 0 18 0 9 0 722444031 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 54138 15427 180 55 0 52024 0
[pid=27826/tid=27833] ppid=27824 vsize=216552 CPUtime=3.01 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 4298 1251 0 0 300 1 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4232648 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=216552 CPUtime=2.97 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 4692 1251 0 0 296 1 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=216552 CPUtime=3 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 4877 1251 0 0 300 0 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=216552 CPUtime=3.01 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 5750 1251 0 0 301 0 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=216552 CPUtime=3.01 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 5149 1251 0 0 300 1 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=216552 CPUtime=3.01 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 5603 1251 0 0 300 1 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=216552 CPUtime=3 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 4417 1251 0 0 300 0 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=216552 CPUtime=3.01 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 4805 1251 0 0 301 0 0 0 25 0 9 0 722444039 221749248 15427 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.17
Current children cumulated vsize (KiB) 216552

[startup+6.30048 s]
/proc/loadavg: 7.60 7.96 7.97 9/500 27840
/proc/meminfo: memFree=28001520/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=233868 CPUtime=49.72 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 69878 1251 0 0 4957 15 0 0 18 0 9 0 722444031 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 58467 19739 180 55 0 56353 0
[pid=27826/tid=27833] ppid=27824 vsize=233868 CPUtime=6.2 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 7573 1251 0 0 619 1 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364861 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=233868 CPUtime=6.12 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 7292 1251 0 0 610 2 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=233868 CPUtime=6.21 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 8677 1251 0 0 620 1 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=233868 CPUtime=6.21 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 9141 1251 0 0 620 1 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=233868 CPUtime=6.21 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 8551 1251 0 0 619 2 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=233868 CPUtime=6.2 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 8643 1251 0 0 619 1 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4310188 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=233868 CPUtime=6.21 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 8453 1251 0 0 619 2 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=233868 CPUtime=6.21 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 7257 1251 0 0 621 0 0 0 25 0 9 0 722444039 239480832 19739 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 49.72
Current children cumulated vsize (KiB) 233868

[startup+12.7013 s]
/proc/loadavg: 7.63 7.96 7.97 9/500 27841
/proc/meminfo: memFree=27957540/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=266468 CPUtime=100.81 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 114869 1251 0 0 10057 24 0 0 18 0 9 0 722444031 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 66617 27896 180 55 0 64503 0
[pid=27826/tid=27833] ppid=27824 vsize=266468 CPUtime=12.61 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 13093 1251 0 0 1259 2 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366079 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=266468 CPUtime=12.41 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 11249 1251 0 0 1238 3 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=266468 CPUtime=12.61 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 13313 1251 0 0 1259 2 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=266468 CPUtime=12.61 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 15932 1251 0 0 1258 3 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=266468 CPUtime=12.61 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 13975 1251 0 0 1258 3 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=266468 CPUtime=12.61 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 15752 1251 0 0 1258 3 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=266468 CPUtime=12.6 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 13521 1251 0 0 1257 3 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=266468 CPUtime=12.61 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 13743 1251 0 0 1259 2 0 0 25 0 9 0 722444039 272863232 27896 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 100.81
Current children cumulated vsize (KiB) 266468

[startup+25.5011 s]
/proc/loadavg: 7.71 7.96 7.97 9/500 27841
/proc/meminfo: memFree=27859308/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=359292 CPUtime=203 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 186684 1251 0 0 20259 41 0 0 18 0 9 0 722444031 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 89823 51074 180 55 0 87709 0
[pid=27826/tid=27833] ppid=27824 vsize=359292 CPUtime=25.41 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 22690 1251 0 0 2537 4 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=359292 CPUtime=24.99 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 20913 1251 0 0 2494 5 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=359292 CPUtime=25.41 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 20775 1251 0 0 2538 3 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=359292 CPUtime=25.41 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 22656 1251 0 0 2537 4 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4353727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=359292 CPUtime=25.41 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 23041 1251 0 0 2535 6 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=359292 CPUtime=25.4 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 25033 1251 0 0 2536 4 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=359292 CPUtime=25.4 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 23152 1251 0 0 2535 5 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=359292 CPUtime=25.41 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 24133 1251 0 0 2537 4 0 0 25 0 9 0 722444039 367915008 51074 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 203
Current children cumulated vsize (KiB) 359292

[startup+51.1006 s]
/proc/loadavg: 7.81 7.96 7.97 9/500 27842
/proc/meminfo: memFree=27800192/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=421616 CPUtime=407.37 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 272604 1251 0 0 40677 60 0 0 18 0 9 0 722444031 431734784 66657 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 105404 66657 180 55 0 103290 0
[pid=27826/tid=27833] ppid=27824 vsize=421616 CPUtime=51.01 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 32891 1251 0 0 5095 6 0 0 25 0 9 0 722444039 431734784 66657 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364984 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=421616 CPUtime=50.18 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 29842 1251 0 0 5011 7 0 0 25 0 9 0 722444039 431734784 66657 33554432000 4194304 4417452 140735321457872 18446744073709551615 4354332 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=421616 CPUtime=51.01 cores=0-7

################
# More data... #
################


[startup+4722.32 s]
/proc/loadavg: 8.00 8.08 8.06 9/500 27990
/proc/meminfo: memFree=25639564/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=2574244 CPUtime=37698.4 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 3539725 1251 0 0 3769131 709 0 0 18 0 9 0 722444031 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 643561 604830 180 55 0 641447 0
[pid=27826/tid=27833] ppid=27824 vsize=2574244 CPUtime=4722.31 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 443836 1251 0 0 472141 90 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=2574244 CPUtime=4644.97 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 463739 1251 0 0 464411 86 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364861 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=2574244 CPUtime=4722.35 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 497987 1251 0 0 472138 97 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=2574244 CPUtime=4722.35 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 261558 1251 0 0 472176 59 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=2574244 CPUtime=4722.16 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 443651 1251 0 0 472122 94 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=2574244 CPUtime=4721.69 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 480479 1251 0 0 472069 100 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=2574244 CPUtime=4722.33 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 476892 1251 0 0 472146 87 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=2574244 CPUtime=4720.1 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 467292 1251 0 0 471917 93 0 0 25 0 9 0 722444039 2636025856 604830 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37698.4
Current children cumulated vsize (KiB) 2574244

[startup+4782.3 s]
/proc/loadavg: 8.05 8.08 8.06 9/500 27992
/proc/meminfo: memFree=25639548/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=2573952 CPUtime=38177.3 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 3539725 1251 0 0 3817017 709 0 0 18 0 9 0 722444031 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 643488 604757 180 55 0 641374 0
[pid=27826/tid=27833] ppid=27824 vsize=2573952 CPUtime=4782.3 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 443836 1251 0 0 478140 90 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=2573952 CPUtime=4703.98 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 463739 1251 0 0 470312 86 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=2573952 CPUtime=4782.33 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 497987 1251 0 0 478136 97 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=2573952 CPUtime=4782.33 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 261558 1251 0 0 478174 59 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=2573952 CPUtime=4782.15 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 443651 1251 0 0 478121 94 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=2573952 CPUtime=4781.67 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 480479 1251 0 0 478067 100 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=2573952 CPUtime=4782.31 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 476892 1251 0 0 478144 87 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366113 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=2573952 CPUtime=4780.06 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 467292 1251 0 0 477913 93 0 0 25 0 9 0 722444039 2635726848 604757 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38177.3
Current children cumulated vsize (KiB) 2573952

[startup+4842.3 s]
/proc/loadavg: 8.07 8.07 8.06 9/500 27993
/proc/meminfo: memFree=25573920/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=2639488 CPUtime=38656.3 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 3556109 1251 0 0 3864914 712 0 0 18 0 9 0 722444031 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 659872 621141 180 55 0 657758 0
[pid=27826/tid=27833] ppid=27824 vsize=2639488 CPUtime=4842.3 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 443836 1251 0 0 484140 90 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=2639488 CPUtime=4762.99 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 463739 1251 0 0 476213 86 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=2639488 CPUtime=4842.33 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 497987 1251 0 0 484136 97 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=2639488 CPUtime=4842.33 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 261558 1251 0 0 484174 59 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=2639488 CPUtime=4842.14 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 443651 1251 0 0 484120 94 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=2639488 CPUtime=4841.67 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 480479 1251 0 0 484067 100 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=2639488 CPUtime=4842.31 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 476892 1251 0 0 484144 87 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=2639488 CPUtime=4840.05 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 483676 1251 0 0 483910 95 0 0 25 0 9 0 722444039 2702835712 621141 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38656.3
Current children cumulated vsize (KiB) 2639488

[startup+4902.3 s]
/proc/loadavg: 8.07 8.07 8.06 9/500 27995
/proc/meminfo: memFree=25602416/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=2611968 CPUtime=39135.3 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 3578308 1251 0 0 3912813 716 0 0 18 0 9 0 722444031 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 652992 614261 180 55 0 650878 0
[pid=27826/tid=27833] ppid=27824 vsize=2611968 CPUtime=4902.3 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 443836 1251 0 0 490140 90 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=2611968 CPUtime=4822.02 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 463739 1251 0 0 482116 86 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=2611968 CPUtime=4902.33 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 497987 1251 0 0 490136 97 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=2611968 CPUtime=4902.33 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 261558 1251 0 0 490174 59 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=2611968 CPUtime=4902.14 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 443651 1251 0 0 490120 94 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=2611968 CPUtime=4901.67 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 502678 1251 0 0 490063 104 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=2611968 CPUtime=4902.31 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 476892 1251 0 0 490144 87 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=2611968 CPUtime=4900.03 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 483676 1251 0 0 489908 95 0 0 25 0 9 0 722444039 2674655232 614261 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39135.3
Current children cumulated vsize (KiB) 2611968

[startup+4962.3 s]
/proc/loadavg: 8.06 8.07 8.05 9/500 27997
/proc/meminfo: memFree=25536180/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=2677504 CPUtime=39614.3 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 3594692 1251 0 0 3960711 718 0 0 18 0 9 0 722444031 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 669376 630645 180 55 0 667262 0
[pid=27826/tid=27833] ppid=27824 vsize=2677504 CPUtime=4962.3 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 443836 1251 0 0 496140 90 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366079 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=2677504 CPUtime=4881.03 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 463739 1251 0 0 488017 86 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=2677504 CPUtime=4962.33 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 514371 1251 0 0 496135 98 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=2677504 CPUtime=4962.34 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 261558 1251 0 0 496175 59 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=2677504 CPUtime=4962.14 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 443651 1251 0 0 496120 94 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4365092 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=2677504 CPUtime=4961.67 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 502678 1251 0 0 496063 104 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=2677504 CPUtime=4962.32 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 476892 1251 0 0 496145 87 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4365058 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=2677504 CPUtime=4960.01 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 483676 1251 0 0 495906 95 0 0 25 0 9 0 722444039 2741764096 630645 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39614.3
Current children cumulated vsize (KiB) 2677504



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.7 s]
/proc/loadavg: 8.07 8.07 8.06 9/500 27998
/proc/meminfo: memFree=25541492/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=2672256 CPUtime=40000.8 cores=0-7
/proc/27826/stat : 27826 (plingeling) S 27824 27826 27522 0 -1 4202496 3623565 1251 0 0 3999357 725 0 0 18 0 9 0 722444031 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 270624913381 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 668064 629333 180 55 0 665950 0
[pid=27826/tid=27833] ppid=27824 vsize=2672256 CPUtime=5010.72 cores=0-7
/proc/27826/task/27833/stat : 27833 (plingeling) R 27824 27826 27522 0 -1 4202560 443836 1251 0 0 500982 90 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=2672256 CPUtime=4928.64 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4202560 463739 1251 0 0 492778 86 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=27826/tid=27835] ppid=27824 vsize=2672256 CPUtime=5010.75 cores=0-7
/proc/27826/task/27835/stat : 27835 (plingeling) R 27824 27826 27522 0 -1 4202560 514371 1251 0 0 500977 98 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4308382 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=27826/tid=27836] ppid=27824 vsize=2672256 CPUtime=5010.76 cores=0-7
/proc/27826/task/27836/stat : 27836 (plingeling) R 27824 27826 27522 0 -1 4202560 261558 1251 0 0 501017 59 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=27826/tid=27837] ppid=27824 vsize=2672256 CPUtime=5010.56 cores=0-7
/proc/27826/task/27837/stat : 27837 (plingeling) R 27824 27826 27522 0 -1 4202560 472524 1251 0 0 500955 101 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=27826/tid=27838] ppid=27824 vsize=2672256 CPUtime=5010.09 cores=0-7
/proc/27826/task/27838/stat : 27838 (plingeling) R 27824 27826 27522 0 -1 4202560 502678 1251 0 0 500905 104 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=27826/tid=27839] ppid=27824 vsize=2672256 CPUtime=5010.74 cores=0-7
/proc/27826/task/27839/stat : 27839 (plingeling) R 27824 27826 27522 0 -1 4202560 476892 1251 0 0 500987 87 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=27826/tid=27840] ppid=27824 vsize=2672256 CPUtime=5008.42 cores=0-7
/proc/27826/task/27840/stat : 27840 (plingeling) R 27824 27826 27522 0 -1 4202560 483676 1251 0 0 500747 95 0 0 25 0 9 0 722444039 2736390144 629333 33554432000 4194304 4417452 140735321457872 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2672256

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010.82 s]
/proc/loadavg: 8.07 8.07 8.06 9/500 27998
/proc/meminfo: memFree=25541492/32951124 swapFree=53876544/67111528
[pid=27826] ppid=27824 vsize=0 CPUtime=40001 cores=0-7
/proc/27826/stat : 27826 (plingeling) Z 27824 27826 27522 0 -1 4203548 3623565 1251 0 0 3999362 735 0 0 18 0 2 0 722444031 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 7 0 0 0
/proc/27826/statm: 0 0 0 0 0 0 0
[pid=27826/tid=27834] ppid=27824 vsize=0 CPUtime=4928.74 cores=0-7
/proc/27826/task/27834/stat : 27834 (plingeling) R 27824 27826 27522 0 -1 4203588 463739 1251 0 0 492778 96 0 0 25 0 2 0 722444039 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5010.85
CPU time (s): 40001
CPU user time (s): 39993.6
CPU system time (s): 7.37288
CPU usage (%): 798.288
Max. virtual memory (cumulated for all children) (KiB): 2880232

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.6
system time used= 7.37288
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3624816
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 205
involuntary context switches= 194821

runsolver used 22.4176 second user time and 59.6959 second system time

The end

Launcher Data 

Begin job on node108 at 2011-05-23 01:53:51
IDJOB=3394874
IDBENCH=85805
IDSOLVER=1851
FILE ID=node108/3394874-1306108431
RUNJOBID= node108-1306105485-27536
PBS_JOBID= 13388022
Free space on /tmp= 73076 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT05/crafted/sorge05/QG7a/gensys-ukn004.shuffled-as.sat05-3844.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394874-1306108431/watcher-3394874-1306108431 -o /tmp/evaluation-result-3394874-1306108431/solver-3394874-1306108431 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394874-1306108431.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 7d96ba103689bec761c9e7df7b158323
RANDOM SEED=1537410341

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28065728 kB
Buffers:        127312 kB
Cached:        3531328 kB
SwapCached:      16344 kB
Active:        1055040 kB
Inactive:      3597248 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28065728 kB
SwapTotal:    67111528 kB
SwapFree:     53876544 kB
Dirty:             804 kB
Writeback:           0 kB
AnonPages:      988144 kB
Mapped:          12304 kB
Slab:            96192 kB
PageTables:      65572 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 134535400 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73076 MiB
End job on node108 at 2011-05-23 03:17:24