Trace number 3372470

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40001.6 5024.07

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
transport-transport-three-cities-sequential-14nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.040-NOTKNOWN.cnf
MD5SUM6c2d25eda70423643946d76935ceb950
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1512832
Number of clauses8233086
Sum of the clauses size19530286
Maximum clause length1178
Minimum clause length1
Number of clauses of size 1846
Number of clauses of size 28165680
Number of clauses of size 31280
Number of clauses of size 41600
Number of clauses of size 51600
Number of clauses of size over 562080

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372470-1305438385/watcher-3372470-1305438385 -o /tmp/evaluation-result-3372470-1305438385/solver-3372470-1305438385 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372470-1305438385.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.48 7.46 6.22 1/344 20616
/proc/meminfo: memFree=17430912/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=27328 CPUtime=0 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 4889 1249 0 0 0 0 0 0 23 0 1 0 734069676 27983872 4819 33554432000 4194304 4417452 140737291799392 18446744073709551615 256415391488 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 6832 4819 144 55 0 4718 0

[startup+0.109046 s]
/proc/loadavg: 6.48 7.46 6.22 1/344 20616
/proc/meminfo: memFree=17430912/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=44956 CPUtime=0.09 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 9333 1249 0 0 9 0 0 0 23 0 1 0 734069676 46034944 9262 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232654 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 11239 9262 148 55 0 9125 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 44956

[startup+0.201045 s]
/proc/loadavg: 6.48 7.46 6.22 1/344 20616
/proc/meminfo: memFree=17430912/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=55380 CPUtime=0.19 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 11908 1249 0 0 18 1 0 0 23 0 1 0 734069676 56709120 11837 33554432000 4194304 4417452 140737291799392 18446744073709551615 4231327 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 13845 11837 149 55 0 11731 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 55380

[startup+0.301021 s]
/proc/loadavg: 6.48 7.46 6.22 1/344 20616
/proc/meminfo: memFree=17430912/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=79872 CPUtime=0.28 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 18063 1249 0 0 27 1 0 0 24 0 1 0 734069676 81788928 17992 33554432000 4194304 4417452 140737291799392 18446744073709551615 4231327 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 19968 17992 149 55 0 17854 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 79872

[startup+0.700959 s]
/proc/loadavg: 6.48 7.46 6.22 1/344 20616
/proc/meminfo: memFree=17430912/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=117148 CPUtime=0.68 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 27382 1249 0 0 66 2 0 0 25 0 1 0 734069676 119959552 27311 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264940 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 29287 27311 149 55 0 27173 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 117148

[startup+1.50079 s]
/proc/loadavg: 6.48 7.46 6.22 2/345 20622
/proc/meminfo: memFree=17233008/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=213792 CPUtime=1.48 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 51576 1249 0 0 143 5 0 0 25 0 1 0 734069676 218923008 51472 33554432000 4194304 4417452 140737291799392 18446744073709551615 256415391488 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 53448 51472 149 55 0 51334 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 213792

[startup+3.1005 s]
/proc/loadavg: 6.48 7.46 6.22 2/345 20622
/proc/meminfo: memFree=17020948/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=486348 CPUtime=3.09 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 119812 1249 0 0 297 12 0 0 25 0 1 0 734069676 498020352 119611 33554432000 4194304 4417452 140737291799392 18446744073709551615 4231547 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 121587 119611 149 55 0 119473 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 486348

[startup+6.30088 s]
/proc/loadavg: 6.04 7.36 6.19 2/345 20622
/proc/meminfo: memFree=16497792/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=940884 CPUtime=6.28 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 233734 1249 0 0 604 24 0 0 25 0 1 0 734069676 963465216 233245 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264940 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 235221 233245 149 55 0 233107 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 940884

[startup+12.7007 s]
/proc/loadavg: 5.64 7.25 6.16 2/345 20622
/proc/meminfo: memFree=15603164/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=1833812 CPUtime=12.69 cores=0-7
/proc/20616/stat : 20616 (plingeling) R 20614 20616 20436 0 -1 4202496 456966 1249 0 0 1220 49 0 0 25 0 1 0 734069676 1877823488 456477 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264940 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/20616/statm: 458453 456477 149 55 0 456339 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 1833812

[startup+25.5012 s]
/proc/loadavg: 5.68 7.17 6.15 9/353 20631
/proc/meminfo: memFree=14744124/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=2873108 CPUtime=82.51 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 1183011 1249 0 0 8070 181 0 0 18 0 9 0 734069676 2942062592 682111 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 718277 682137 177 55 0 716163 0
[pid=20616/tid=20624] ppid=20614 vsize=2873108 CPUtime=8.04 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 59299 1249 0 0 795 9 0 0 25 0 9 0 734071401 2942062592 682148 33554432000 4194304 4417452 140737291799392 18446744073709551615 4326609 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=2873108 CPUtime=8.12 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 79601 1249 0 0 793 19 0 0 25 0 9 0 734071401 2942062592 682162 33554432000 4194304 4417452 140737291799392 18446744073709551615 256415085661 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=2873108 CPUtime=8.17 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 97944 1249 0 0 799 18 0 0 25 0 9 0 734071401 2942062592 682177 33554432000 4194304 4417452 140737291799392 18446744073709551615 4259140 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=2873108 CPUtime=8.23 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 34075 1249 0 0 815 8 0 0 25 0 9 0 734071401 2942062592 682190 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=2873108 CPUtime=8.18 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 54416 1249 0 0 810 8 0 0 25 0 9 0 734071401 2942062592 682205 33554432000 4194304 4417452 140737291799392 18446744073709551615 4277545 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=2873108 CPUtime=8.21 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 87200 1249 0 0 802 19 0 0 25 0 9 0 734071401 2942062592 682220 33554432000 4194304 4417452 140737291799392 18446744073709551615 4256312 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=2873108 CPUtime=8.19 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 96413 1249 0 0 799 20 0 0 25 0 9 0 734071402 2942062592 682234 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232464 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=2873108 CPUtime=8.06 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 69678 1249 0 0 794 12 0 0 25 0 9 0 734071402 2942062592 682248 33554432000 4194304 4417452 140737291799392 18446744073709551615 4261313 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 82.51
Current children cumulated vsize (KiB) 2873108

[startup+51.1013 s]
/proc/loadavg: 6.47 7.24 6.20 9/353 20631
/proc/meminfo: memFree=14971048/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=2682420 CPUtime=286.79 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 1891399 1249 0 0 28366 313 0 0 18 0 9 0 734069676 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 670605 620747 180 55 0 668491 0
[pid=20616/tid=20624] ppid=20614 vsize=2682420 CPUtime=33.35 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 212846 1249 0 0 3296 39 0 0 25 0 9 0 734071401 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4252343 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=2682420 CPUtime=33.7 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 177272 1249 0 0 3335 35 0 0 25 0 9 0 734071401 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4253176 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=2682420 CPUtime=33.75 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 162677 1249 0 0 3348 27 0 0 25 0 9 0 734071401 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4253524 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=2682420 CPUtime=33.74 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 34075 1249 0 0 3366 8 0 0 25 0 9 0 734071401 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4236830 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=2682420 CPUtime=33.74 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 161344 1249 0 0 3341 33 0 0 25 0 9 0 734071401 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4319195 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=2682420 CPUtime=33.79 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 153173 1249 0 0 3350 29 0 0 25 0 9 0 734071401 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4239060 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=2682420 CPUtime=33.76 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 177132 1249 0 0 3339 37 0 0 25 0 9 0 734071402 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4254425 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=2682420 CPUtime=33.64 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 208443 1249 0 0 3327 37 0 0 25 0 9 0 734071402 2746798080 620747 33554432000 4194304 4417452 140737291799392 18446744073709551615 4253198 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 286.79
Current children cumulated vsize (KiB) 2682420

[startup+102.301 s]
/proc/loadavg: 7.34 7.35 6.29 9/353 20633
/proc/meminfo: memFree=14764132/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=2815328 CPUtime=695.88 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 2086745 1249 0 0 69248 340 0 0 18 0 9 0 734069676 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 703832 666268 180 55 0 701718 0
[pid=20616/tid=20624] ppid=20614 vsize=2815328 CPUtime=83.97 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 221338 1249 0 0 8356 41 0 0 25 0 9 0 734071401 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=2815328 CPUtime=84.9 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 190450 1249 0 0 8453 37 0 0 25 0 9 0 734071401 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232804 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=2815328 CPUtime=84.97 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 191943 1249 0 0 8467 30 0 0 25 0 9 0 734071401 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4365538 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=2815328 CPUtime=84.95 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 106201 1249 0 0 8480 15 0 0 25 0 9 0 734071401 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264934 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=2815328 CPUtime=84.95 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 171883 1249 0 0 8460 35 0 0 25 0 9 0 734071401 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232577 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=2815328 CPUtime=85.01 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 163712 1249 0 0 8470 31 0 0 25 0 9 0 734071401 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=2815328 CPUtime=84.98 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 213110 1249 0 0 8455 43 0 0 25 0 9 0 734071402 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=2815328 CPUtime=84.85 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 223671 1249 0 0 8445 40 0 0 25 0 9 0 734071402 2882895872 666268 33554432000 4194304 4417452 140737291799392 18446744073709551615 4355830 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 695.88
Current children cumulated vsize (KiB) 2815328

[startup+162.301 s]
/proc/loadavg: 7.79 7.48 6.40 9/353 20634
/proc/meminfo: memFree=14417540/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=3160820 CPUtime=1175.09 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 2252129 1249 0 0 117147 362 0 0 18 0 9 0 734069676 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 790205 752579 180 55 0 788091 0
[pid=20616/tid=20624] ppid=20614 vsize=3160820 CPUtime=143.26 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 242746 1249 0 0 14282 44 0 0 25 0 9 0 734071401 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=3160820 CPUtime=144.89 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 193447 1249 0 0 14451 38 0 0 25 0 9 0 734071401 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4237015 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=3160820 CPUtime=144.96 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 193562 1249 0 0 14466 30 0 0 25 0 9 0 734071401 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=3160820 CPUtime=144.93 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 140798 1249 0 0 14473 20 0 0 25 0 9 0 734071401 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=3160820 CPUtime=144.93 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 171883 1249 0 0 14458 35 0 0 25 0 9 0 734071401 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=3160820 CPUtime=145 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 176041 1249 0 0 14468 32 0 0 25 0 9 0 734071401 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=3160820 CPUtime=144.97 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 291898 1249 0 0 14444 53 0 0 25 0 9 0 734071402 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=3160820 CPUtime=144.84 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 237317 1249 0 0 14443 41 0 0 25 0 9 0 734071402 3236679680 752579 33554432000 4194304 4417452 140737291799392 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1175.09
Current children cumulated vsize (KiB) 3160820


################
# More data... #
################

/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 2112633 1249 0 0 476047 441 0 0 25 0 9 0 734071402 15386284032 3718896 33554432000 4194304 4417452 140737291799392 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38079.7
Current children cumulated vsize (KiB) 15025668

[startup+4842.31 s]
/proc/loadavg: 8.01 8.04 8.00 9/353 20780
/proc/meminfo: memFree=1332432/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=16219640 CPUtime=38559.1 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 22375100 1249 0 0 3850914 4998 0 0 18 0 9 0 734069676 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 4054910 4017389 180 55 0 4052796 0
[pid=20616/tid=20624] ppid=20614 vsize=16219640 CPUtime=4807.21 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 2284053 1249 0 0 480229 492 0 0 25 0 9 0 734071401 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=16219640 CPUtime=4824.21 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 2046688 1249 0 0 481942 479 0 0 25 0 9 0 734071401 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=16219640 CPUtime=4786.01 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 2620166 1249 0 0 478026 575 0 0 25 0 9 0 734071401 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4270130 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=16219640 CPUtime=4824.82 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 5966151 1249 0 0 481036 1446 0 0 25 0 9 0 734071401 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264909 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=16219640 CPUtime=4824.92 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 1872253 1249 0 0 482103 389 0 0 25 0 9 0 734071401 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=16219640 CPUtime=4824.71 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 2178427 1249 0 0 481966 505 0 0 25 0 9 0 734071401 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=16219640 CPUtime=4825.05 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 2690292 1249 0 0 481902 603 0 0 25 0 9 0 734071402 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=16219640 CPUtime=4824.89 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 2112633 1249 0 0 482048 441 0 0 25 0 9 0 734071402 16608911360 4017389 33554432000 4194304 4417452 140737291799392 18446744073709551615 4252389 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38559.1
Current children cumulated vsize (KiB) 16219640

[startup+4902.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/353 20782
/proc/meminfo: memFree=96636/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=17843516 CPUtime=39038 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 23162913 1249 0 0 3898553 5243 0 0 18 0 9 0 734069676 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 4460879 4423358 180 55 0 4458765 0
[pid=20616/tid=20624] ppid=20614 vsize=17843516 CPUtime=4867.2 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 2299509 1249 0 0 486226 494 0 0 25 0 9 0 734071401 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=17843516 CPUtime=4884.2 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 2046688 1249 0 0 487941 479 0 0 25 0 9 0 734071401 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4236634 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=17843516 CPUtime=4845.31 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 2885292 1249 0 0 483881 650 0 0 25 0 9 0 734071401 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=17843516 CPUtime=4884.81 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 5966151 1249 0 0 487035 1446 0 0 25 0 9 0 734071401 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4358093 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=17843516 CPUtime=4884.91 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 1872253 1249 0 0 488102 389 0 0 25 0 9 0 734071401 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=17843516 CPUtime=4884.45 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 2178427 1249 0 0 487940 505 0 0 25 0 9 0 734071401 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=17843516 CPUtime=4885.04 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 2963286 1249 0 0 487802 702 0 0 25 0 9 0 734071402 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232580 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=17843516 CPUtime=4884.74 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 2346870 1249 0 0 487964 510 0 0 25 0 9 0 734071402 18271760384 4423358 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39038
Current children cumulated vsize (KiB) 17843516

[startup+4962.3 s]
/proc/loadavg: 8.02 8.03 8.00 9/353 20783
/proc/meminfo: memFree=96148/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=18335596 CPUtime=39516.7 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 23285980 1249 0 0 3946393 5280 0 0 18 0 9 0 734069676 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 4583899 4546378 180 55 0 4581785 0
[pid=20616/tid=20624] ppid=20614 vsize=18335596 CPUtime=4927.19 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 2338114 1249 0 0 492210 509 0 0 25 0 9 0 734071401 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=18335596 CPUtime=4943.5 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 2046688 1249 0 0 493871 479 0 0 25 0 9 0 734071401 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232801 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=18335596 CPUtime=4905.09 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 2885292 1249 0 0 489859 650 0 0 25 0 9 0 734071401 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=18335596 CPUtime=4944.72 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 6050613 1249 0 0 493004 1468 0 0 25 0 9 0 734071401 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4250873 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=18335596 CPUtime=4944.91 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 1872253 1249 0 0 494102 389 0 0 25 0 9 0 734071401 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4302981 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=18335596 CPUtime=4944.42 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 2178427 1249 0 0 493937 505 0 0 25 0 9 0 734071401 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=18335596 CPUtime=4945.04 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 2963286 1249 0 0 493802 702 0 0 25 0 9 0 734071402 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4259665 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=18335596 CPUtime=4944.54 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 2346870 1249 0 0 493944 510 0 0 25 0 9 0 734071402 18775650304 4546378 33554432000 4194304 4417452 140737291799392 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39516.7
Current children cumulated vsize (KiB) 18335596

[startup+5022.3 s]
/proc/loadavg: 8.33 8.09 8.02 9/353 20785
/proc/meminfo: memFree=96708/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=19464180 CPUtime=39992.7 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 23656985 1249 0 0 3993811 5456 0 0 18 0 9 0 734069676 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 4866045 4828524 180 55 0 4863931 0
[pid=20616/tid=20624] ppid=20614 vsize=19464180 CPUtime=4986.57 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 2468871 1249 0 0 498077 580 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364805 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=19464180 CPUtime=5002.84 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 2046688 1249 0 0 499804 480 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=19464180 CPUtime=4964.37 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 2885292 1249 0 0 495787 650 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=19464180 CPUtime=5003.63 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 6290861 1249 0 0 498792 1571 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364542 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=19464180 CPUtime=5004.68 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 1872253 1249 0 0 500078 390 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4309869 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=19464180 CPUtime=5003.9 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 2178427 1249 0 0 499885 505 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4355860 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=19464180 CPUtime=5005.05 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 2963286 1249 0 0 499803 702 0 0 25 0 9 0 734071402 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232537 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=19464180 CPUtime=5004.34 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 2346870 1249 0 0 499924 510 0 0 25 0 9 0 734071402 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39992.7
Current children cumulated vsize (KiB) 19464180



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5023.3 s]
/proc/loadavg: 8.33 8.09 8.02 9/353 20785
/proc/meminfo: memFree=96964/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=19464180 CPUtime=40000.7 cores=0-7
/proc/20616/stat : 20616 (plingeling) S 20614 20616 20436 0 -1 4202496 23656985 1249 0 0 3994618 5456 0 0 18 0 9 0 734069676 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 4866045 4828524 180 55 0 4863931 0
[pid=20616/tid=20624] ppid=20614 vsize=19464180 CPUtime=4987.58 cores=0-7
/proc/20616/task/20624/stat : 20624 (plingeling) R 20614 20616 20436 0 -1 4202560 2468871 1249 0 0 498178 580 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=20616/tid=20625] ppid=20614 vsize=19464180 CPUtime=5003.85 cores=0-7
/proc/20616/task/20625/stat : 20625 (plingeling) R 20614 20616 20436 0 -1 4202560 2046688 1249 0 0 499905 480 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20616/tid=20626] ppid=20614 vsize=19464180 CPUtime=4965.38 cores=0-7
/proc/20616/task/20626/stat : 20626 (plingeling) R 20614 20616 20436 0 -1 4202560 2885292 1249 0 0 495888 650 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20616/tid=20627] ppid=20614 vsize=19464180 CPUtime=5004.64 cores=0-7
/proc/20616/task/20627/stat : 20627 (plingeling) R 20614 20616 20436 0 -1 4202560 6290861 1249 0 0 498893 1571 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20616/tid=20628] ppid=20614 vsize=19464180 CPUtime=5005.69 cores=0-7
/proc/20616/task/20628/stat : 20628 (plingeling) R 20614 20616 20436 0 -1 4202560 1872253 1249 0 0 500179 390 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4308211 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=20616/tid=20629] ppid=20614 vsize=19464180 CPUtime=5004.91 cores=0-7
/proc/20616/task/20629/stat : 20629 (plingeling) R 20614 20616 20436 0 -1 4202560 2178427 1249 0 0 499986 505 0 0 25 0 9 0 734071401 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20616/tid=20630] ppid=20614 vsize=19464180 CPUtime=5006.06 cores=0-7
/proc/20616/task/20630/stat : 20630 (plingeling) R 20614 20616 20436 0 -1 4202560 2963286 1249 0 0 499904 702 0 0 25 0 9 0 734071402 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4237600 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=19464180 CPUtime=5005.32 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4202560 2346870 1249 0 0 500022 510 0 0 25 0 9 0 734071402 19931320320 4828524 33554432000 4194304 4417452 140737291799392 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 19464180

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5023.5 s]
/proc/loadavg: 8.33 8.09 8.02 9/353 20785
/proc/meminfo: memFree=96964/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=0 CPUtime=40001 cores=0-7
/proc/20616/stat : 20616 (plingeling) Z 20614 20616 20436 0 -1 4203548 23656985 1249 0 0 3994623 5474 0 0 18 0 2 0 734069676 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 0 0 0 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=0 CPUtime=5005.51 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4203588 2346870 1249 0 0 500023 528 0 0 25 0 2 0 734071402 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 0

[startup+5023.9 s]
/proc/loadavg: 7.74 7.97 7.98 2/347 20786
/proc/meminfo: memFree=10197556/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=0 CPUtime=40001.4 cores=0-7
/proc/20616/stat : 20616 (plingeling) Z 20614 20616 20436 0 -1 4203548 23656985 1249 0 0 3994623 5514 0 0 18 0 2 0 734069676 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 0 0 0 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=0 CPUtime=5005.91 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4203588 2346870 1249 0 0 500023 568 0 0 25 0 2 0 734071402 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40001.4
Current children cumulated vsize (KiB) 0

[startup+5024 s]
/proc/loadavg: 7.74 7.97 7.98 2/347 20786
/proc/meminfo: memFree=10197556/32950928 swapFree=67111200/67111528
[pid=20616] ppid=20614 vsize=0 CPUtime=40001.5 cores=0-7
/proc/20616/stat : 20616 (plingeling) Z 20614 20616 20436 0 -1 4203548 23656985 1249 0 0 3994623 5524 0 0 18 0 2 0 734069676 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20616/statm: 0 0 0 0 0 0 0
[pid=20616/tid=20631] ppid=20614 vsize=0 CPUtime=5006.01 cores=0-7
/proc/20616/task/20631/stat : 20631 (plingeling) R 20614 20616 20436 0 -1 4203588 2346870 1249 0 0 500023 578 0 0 25 0 2 0 734071402 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5024.07
CPU time (s): 40001.6
CPU user time (s): 39946.2
CPU system time (s): 55.3196
CPU usage (%): 796.199
Max. virtual memory (cumulated for all children) (KiB): 19464180

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39946.2
system time used= 55.3196
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 23658234
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 961
involuntary context switches= 193339

runsolver used 15.5596 second user time and 43.1604 second system time

The end

Launcher Data 

Begin job on node115 at 2011-05-15 07:46:30
IDJOB=3372470
IDBENCH=82404
IDSOLVER=1851
FILE ID=node115/3372470-1305438385
RUNJOBID= node115-1305437446-20452
PBS_JOBID= 13324775
Free space on /tmp= 73136 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/transport-transport-three-cities-sequential-14nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.040-NOTKNOWN.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372470-1305438385/watcher-3372470-1305438385 -o /tmp/evaluation-result-3372470-1305438385/solver-3372470-1305438385 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372470-1305438385.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 6c2d25eda70423643946d76935ceb950
RANDOM SEED=1257038284

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      17431068 kB
Buffers:        301132 kB
Cached:        4873880 kB
SwapCached:          0 kB
Active:       12160924 kB
Inactive:      3038768 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      17431068 kB
SwapTotal:    67111528 kB
SwapFree:     67111200 kB
Dirty:          166012 kB
Writeback:           0 kB
AnonPages:    10024644 kB
Mapped:          27684 kB
Slab:           212932 kB
PageTables:      40564 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 69653036 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73136 MiB
End job on node115 at 2011-05-15 09:10:15