Trace number 3249668

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.29 1201.96

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-14x38.cnf
MD5SUM56980124bab6f1dbe07ec523b7d59019
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2128
Number of clauses259616
Sum of the clauses size1032080
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 23192
Number of clauses of size 30
Number of clauses of size 4256424
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249668-1304087279/watcher-3249668-1304087279 -o /tmp/evaluation-result-3249668-1304087279/solver-3249668-1304087279 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3249668-1304087279.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.45 5.20 6.79 5/181 20676
/proc/meminfo: memFree=30854048/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=10188 CPUtime=0 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) R 20674 20676 19459 0 -1 4202496 623 1251 0 0 0 0 0 0 25 0 1 0 598958509 10432512 552 33554432000 4194304 4417332 140735444654400 18446744073709551615 260020395776 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20676/statm: 2547 552 146 55 0 433 0

[startup+0.0277489 s]
/proc/loadavg: 3.45 5.20 6.79 5/181 20676
/proc/meminfo: memFree=30854048/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=16364 CPUtime=0.02 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) R 20674 20676 19459 0 -1 4202496 2280 1251 0 0 2 0 0 0 25 0 1 0 598958509 16756736 2113 33554432000 4194304 4417332 140735444654400 18446744073709551615 4259889 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20676/statm: 4091 2113 147 55 0 1977 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 16364

[startup+0.100736 s]
/proc/loadavg: 3.45 5.20 6.79 5/181 20676
/proc/meminfo: memFree=30854048/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=34796 CPUtime=0.08 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) R 20674 20676 19459 0 -1 4202496 6888 1251 0 0 8 0 0 0 25 0 1 0 598958509 35631104 6721 33554432000 4194304 4417332 140735444654400 18446744073709551615 4259896 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20676/statm: 8699 6721 147 55 0 6585 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 34796

[startup+0.300703 s]
/proc/loadavg: 3.45 5.20 6.79 5/181 20676
/proc/meminfo: memFree=30854048/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=75756 CPUtime=0.29 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) R 20674 20676 19459 0 -1 4202496 17128 1251 0 0 27 2 0 0 25 0 1 0 598958509 77574144 16961 33554432000 4194304 4417332 140735444654400 18446744073709551615 260020395776 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/20676/statm: 18939 16961 147 55 0 16825 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 75756

[startup+0.700592 s]
/proc/loadavg: 3.45 5.20 6.79 5/181 20676
/proc/meminfo: memFree=30854048/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=113820 CPUtime=1.85 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 20431 1251 0 0 182 3 0 0 25 0 5 0 598958509 116551680 16269 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 28455 16269 175 55 0 26341 0
Current children cumulated CPU time (s) 1.85
Current children cumulated vsize (KiB) 113820

[startup+1.50045 s]
/proc/loadavg: 3.45 5.20 6.79 9/186 20686
/proc/meminfo: memFree=30754072/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=139316 CPUtime=5.05 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 28833 1251 0 0 500 5 0 0 25 0 5 0 598958509 142659584 22615 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 34829 22615 179 55 0 32715 0
[pid=20676/tid=20683] ppid=20674 vsize=139316 CPUtime=1.18 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 2315 1251 0 0 118 0 0 0 25 0 5 0 598958540 142659584 22615 33554432000 4194304 4417332 140735444654400 18446744073709551615 4260519 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=139316 CPUtime=1.17 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 2329 1251 0 0 117 0 0 0 25 0 5 0 598958540 142659584 22615 33554432000 4194304 4417332 140735444654400 18446744073709551615 4320455 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=139316 CPUtime=1.17 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 3317 1251 0 0 117 0 0 0 25 0 5 0 598958540 142659584 22615 33554432000 4194304 4417332 140735444654400 18446744073709551615 4239418 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=139316 CPUtime=1.17 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 3732 1251 0 0 117 0 0 0 25 0 5 0 598958540 142659584 22615 33554432000 4194304 4417332 140735444654400 18446744073709551615 4354419 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.05
Current children cumulated vsize (KiB) 139316

[startup+3.10115 s]
/proc/loadavg: 3.45 5.20 6.79 9/186 20686
/proc/meminfo: memFree=30720964/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=187844 CPUtime=11.44 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 42687 1251 0 0 1137 7 0 0 25 0 5 0 598958509 192352256 34747 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 46961 34747 179 55 0 44847 0
[pid=20676/tid=20683] ppid=20674 vsize=187844 CPUtime=2.78 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 6071 1251 0 0 277 1 0 0 25 0 5 0 598958540 192352256 34747 33554432000 4194304 4417332 140735444654400 18446744073709551615 4230433 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=187844 CPUtime=2.77 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 6085 1251 0 0 277 0 0 0 25 0 5 0 598958540 192352256 34747 33554432000 4194304 4417332 140735444654400 18446744073709551615 4321008 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=187844 CPUtime=2.77 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 5821 1251 0 0 276 1 0 0 25 0 5 0 598958540 192352256 34747 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=187844 CPUtime=2.78 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 7570 1251 0 0 277 1 0 0 25 0 5 0 598958540 192352256 34747 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364713 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.44
Current children cumulated vsize (KiB) 187844

[startup+6.30056 s]
/proc/loadavg: 3.82 5.25 6.80 9/186 20687
/proc/meminfo: memFree=30626072/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=197116 CPUtime=24.22 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 53599 1251 0 0 2413 9 0 0 25 0 5 0 598958509 201846784 37065 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 49279 37065 179 55 0 47165 0
[pid=20676/tid=20683] ppid=20674 vsize=197116 CPUtime=5.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 6163 1251 0 0 597 1 0 0 25 0 5 0 598958540 201846784 37065 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249583 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=197116 CPUtime=5.97 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 6089 1251 0 0 597 0 0 0 25 0 5 0 598958540 201846784 37065 33554432000 4194304 4417332 140735444654400 18446744073709551615 4250082 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=197116 CPUtime=5.95 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 5821 1251 0 0 594 1 0 0 25 0 5 0 598958540 201846784 37065 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=197116 CPUtime=5.98 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 18386 1251 0 0 595 3 0 0 25 0 5 0 598958540 201846784 37065 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.22
Current children cumulated vsize (KiB) 197116

[startup+12.7004 s]
/proc/loadavg: 4.15 5.30 6.81 9/186 20687
/proc/meminfo: memFree=30578592/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=260436 CPUtime=49.79 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 77698 1251 0 0 4966 13 0 0 25 0 5 0 598958509 266686464 36518 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 65109 36518 179 55 0 62995 0
[pid=20676/tid=20683] ppid=20674 vsize=260436 CPUtime=12.38 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 7697 1251 0 0 1237 1 0 0 25 0 5 0 598958540 266686464 36518 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=260436 CPUtime=12.38 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 10948 1251 0 0 1237 1 0 0 25 0 5 0 598958540 266686464 36518 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249186 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=260436 CPUtime=12.31 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 11667 1251 0 0 1229 2 0 0 25 0 5 0 598958540 266686464 36518 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=260436 CPUtime=12.38 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 30246 1251 0 0 1233 5 0 0 25 0 5 0 598958540 266686464 36518 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 49.79
Current children cumulated vsize (KiB) 260436

[startup+25.501 s]
/proc/loadavg: 5.07 5.44 6.83 9/186 20687
/proc/meminfo: memFree=30404132/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=305660 CPUtime=100.92 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 130027 1251 0 0 10071 21 0 0 25 0 5 0 598958509 312995840 47880 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 76415 47880 180 55 0 74301 0
[pid=20676/tid=20683] ppid=20674 vsize=305660 CPUtime=25.17 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 13810 1251 0 0 2516 1 0 0 25 0 5 0 598958540 312995840 47880 33554432000 4194304 4417332 140735444654400 18446744073709551615 4357123 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=305660 CPUtime=25.17 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 19729 1251 0 0 2515 2 0 0 25 0 5 0 598958540 312995840 47880 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=305660 CPUtime=25.04 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 23481 1251 0 0 2500 4 0 0 25 0 5 0 598958540 312995840 47880 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=305660 CPUtime=25.18 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 55867 1251 0 0 2509 9 0 0 25 0 5 0 598958540 312995840 47880 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 100.92
Current children cumulated vsize (KiB) 305660

[startup+51.1003 s]
/proc/loadavg: 6.07 5.65 6.86 9/186 20688
/proc/meminfo: memFree=30028764/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=454060 CPUtime=203.17 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 252404 1251 0 0 20276 41 0 0 25 0 5 0 598958509 464957440 84951 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 113515 84951 180 55 0 111401 0
[pid=20676/tid=20683] ppid=20674 vsize=454060 CPUtime=50.78 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 40494 1251 0 0 5073 5 0 0 25 0 5 0 598958540 464957440 84951 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=454060 CPUtime=50.77 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 56350 1251 0 0 5068 9 0 0 25 0 5 0 598958540 464957440 84951 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=454060 CPUtime=50.5 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 55667 1251 0 0 5041 9 0 0 25 0 5 0 598958540 464957440 84951 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=454060 CPUtime=50.78 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 82753 1251 0 0 5064 14 0 0 25 0 5 0 598958540 464957440 84951 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.17
Current children cumulated vsize (KiB) 454060

[startup+102.307 s]
/proc/loadavg: 7.27 6.06 6.93 9/186 20689
/proc/meminfo: memFree=29631920/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=722116 CPUtime=407.71 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 443598 1251 0 0 40703 68 0 0 25 0 5 0 598958509 739446784 151965 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 180529 151965 180 55 0 178415 0
[pid=20676/tid=20683] ppid=20674 vsize=722116 CPUtime=101.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 90751 1251 0 0 10187 11 0 0 25 0 5 0 598958540 739446784 151965 33554432000 4194304 4417332 140735444654400 18446744073709551615 4199406 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=722116 CPUtime=101.98 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 108825 1251 0 0 10182 16 0 0 25 0 5 0 598958540 739446784 151965 33554432000 4194304 4417332 140735444654400 18446744073709551615 4366576 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=722116 CPUtime=101.43 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 107249 1251 0 0 10125 18 0 0 25 0 5 0 598958540 739446784 151965 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=722116 CPUtime=101.98 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 119633 1251 0 0 10180 18 0 0 25 0 5 0 598958540 739446784 151965 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.71
Current children cumulated vsize (KiB) 722116

[startup+162.301 s]
/proc/loadavg: 7.80 6.41 7.00 9/186 20691
/proc/meminfo: memFree=29452912/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=687324 CPUtime=647.38 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 593487 1251 0 0 64647 91 0 0 25 0 5 0 598958509 703819776 143267 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 171831 143267 180 55 0 169717 0
[pid=20676/tid=20683] ppid=20674 vsize=687324 CPUtime=161.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 128746 1251 0 0 16180 18 0 0 25 0 5 0 598958540 703819776 143267 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364942 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=687324 CPUtime=161.98 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 154025 1251 0 0 16176 22 0 0 25 0 5 0 598958540 703819776 143267 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=687324 CPUtime=161.1 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 149366 1251 0 0 16085 25 0 0 25 0 5 0 598958540 703819776 143267 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364639 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=687324 CPUtime=161.98 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 144210 1251 0 0 16176 22 0 0 25 0 5 0 598958540 703819776 143267 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.38
Current children cumulated vsize (KiB) 687324


################
# More data... #
################

[pid=20676/tid=20684] ppid=20674 vsize=1760748 CPUtime=761.99 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 531488 1251 0 0 76126 73 0 0 25 0 5 0 598958540 1803005952 411623 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1760748 CPUtime=757.85 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 908930 1251 0 0 75636 149 0 0 25 0 5 0 598958540 1803005952 411623 33554432000 4194304 4417332 140735444654400 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1760748 CPUtime=762.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 350357 1251 0 0 76146 55 0 0 25 0 5 0 598958540 1803005952 411623 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364957 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3044.17
Current children cumulated vsize (KiB) 1760748

[startup+822.301 s]
/proc/loadavg: 8.11 7.97 7.57 9/186 20711
/proc/meminfo: memFree=27886108/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1565068 CPUtime=3283.83 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 2403043 1251 0 0 328016 367 0 0 25 0 5 0 598958509 1602629632 362703 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 391267 362703 180 55 0 389153 0
[pid=20676/tid=20683] ppid=20674 vsize=1565068 CPUtime=821.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 559324 1251 0 0 82118 80 0 0 25 0 5 0 598958540 1602629632 362703 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1565068 CPUtime=821.99 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 552845 1251 0 0 82124 75 0 0 25 0 5 0 598958540 1602629632 362703 33554432000 4194304 4417332 140735444654400 18446744073709551615 4356793 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1565068 CPUtime=817.52 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 923377 1251 0 0 81599 153 0 0 25 0 5 0 598958540 1602629632 362703 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1565068 CPUtime=822.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 350357 1251 0 0 82146 55 0 0 25 0 5 0 598958540 1602629632 362703 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3283.83
Current children cumulated vsize (KiB) 1565068

[startup+882.301 s]
/proc/loadavg: 8.07 7.99 7.60 9/186 20713
/proc/meminfo: memFree=27643876/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1797700 CPUtime=3523.5 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 2590456 1251 0 0 351959 391 0 0 25 0 5 0 598958509 1840844800 420861 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 449425 420861 180 55 0 447311 0
[pid=20676/tid=20683] ppid=20674 vsize=1797700 CPUtime=881.97 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 626266 1251 0 0 88108 89 0 0 25 0 5 0 598958540 1840844800 420861 33554432000 4194304 4417332 140735444654400 18446744073709551615 4251258 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1797700 CPUtime=881.99 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 612421 1251 0 0 88116 83 0 0 25 0 5 0 598958540 1840844800 420861 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1797700 CPUtime=877.19 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 984272 1251 0 0 87559 160 0 0 25 0 5 0 598958540 1840844800 420861 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1797700 CPUtime=882.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 350357 1251 0 0 88146 55 0 0 25 0 5 0 598958540 1840844800 420861 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3523.5
Current children cumulated vsize (KiB) 1797700

[startup+942.301 s]
/proc/loadavg: 8.07 8.00 7.63 9/186 20715
/proc/meminfo: memFree=27617648/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1642104 CPUtime=3763.19 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 2748758 1251 0 0 375905 414 0 0 25 0 5 0 598958509 1681514496 381962 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 410526 381962 180 55 0 408412 0
[pid=20676/tid=20683] ppid=20674 vsize=1642104 CPUtime=941.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 652050 1251 0 0 94104 94 0 0 25 0 5 0 598958540 1681514496 381962 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1642104 CPUtime=942 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 634237 1251 0 0 94114 86 0 0 25 0 5 0 598958540 1681514496 381962 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1642104 CPUtime=936.87 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1081837 1251 0 0 93514 173 0 0 25 0 5 0 598958540 1681514496 381962 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1642104 CPUtime=942.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 363494 1251 0 0 94144 57 0 0 25 0 5 0 598958540 1681514496 381962 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3763.19
Current children cumulated vsize (KiB) 1642104

[startup+1002.3 s]
/proc/loadavg: 8.06 8.01 7.65 9/186 20716
/proc/meminfo: memFree=27986848/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1469232 CPUtime=4002.86 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 2850737 1251 0 0 399858 428 0 0 25 0 5 0 598958509 1504493568 338744 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 367308 338744 180 55 0 365194 0
[pid=20676/tid=20683] ppid=20674 vsize=1469232 CPUtime=1001.97 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 684775 1251 0 0 100098 99 0 0 25 0 5 0 598958540 1504493568 338744 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1469232 CPUtime=1001.99 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 655462 1251 0 0 100111 88 0 0 25 0 5 0 598958540 1504493568 338744 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1469232 CPUtime=996.54 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1091749 1251 0 0 99480 174 0 0 25 0 5 0 598958540 1504493568 338744 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364652 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1469232 CPUtime=1002.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 401611 1251 0 0 100138 63 0 0 25 0 5 0 598958540 1504493568 338744 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4002.86
Current children cumulated vsize (KiB) 1469232

[startup+1062.3 s]
/proc/loadavg: 8.10 8.04 7.68 9/186 20718
/proc/meminfo: memFree=27752548/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1730760 CPUtime=4242.54 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 2996696 1251 0 0 423809 445 0 0 25 0 5 0 598958509 1772298240 404126 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 432690 404126 180 55 0 430576 0
[pid=20676/tid=20683] ppid=20674 vsize=1730760 CPUtime=1061.97 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 728465 1251 0 0 106093 104 0 0 25 0 5 0 598958540 1772298240 404126 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364736 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1730760 CPUtime=1062 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 678973 1251 0 0 106109 91 0 0 25 0 5 0 598958540 1772298240 404126 33554432000 4194304 4417332 140735444654400 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1730760 CPUtime=1056.21 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1166180 1251 0 0 105439 182 0 0 25 0 5 0 598958540 1772298240 404126 33554432000 4194304 4417332 140735444654400 18446744073709551615 4250873 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1730760 CPUtime=1062.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 405938 1251 0 0 106138 63 0 0 25 0 5 0 598958540 1772298240 404126 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4242.54
Current children cumulated vsize (KiB) 1730760

[startup+1122.3 s]
/proc/loadavg: 8.07 8.04 7.70 9/186 20719
/proc/meminfo: memFree=27661708/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1635976 CPUtime=4482.22 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 3107569 1251 0 0 447760 462 0 0 25 0 5 0 598958509 1675239424 380430 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 408994 380430 180 55 0 406880 0
[pid=20676/tid=20683] ppid=20674 vsize=1635976 CPUtime=1121.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 758711 1251 0 0 112089 109 0 0 25 0 5 0 598958540 1675239424 380430 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364547 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1635976 CPUtime=1122 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 719055 1251 0 0 112103 97 0 0 25 0 5 0 598958540 1675239424 380430 33554432000 4194304 4417332 140735444654400 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1635976 CPUtime=1115.89 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1194118 1251 0 0 111403 186 0 0 25 0 5 0 598958540 1675239424 380430 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1635976 CPUtime=1122.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 418545 1251 0 0 112136 65 0 0 25 0 5 0 598958540 1675239424 380430 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4482.22
Current children cumulated vsize (KiB) 1635976

[startup+1182.3 s]
/proc/loadavg: 8.06 8.05 7.73 9/186 20721
/proc/meminfo: memFree=27583120/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1760156 CPUtime=4721.89 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 3220063 1251 0 0 471713 476 0 0 25 0 5 0 598958509 1802399744 411507 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 440039 411507 180 55 0 437925 0
[pid=20676/tid=20683] ppid=20674 vsize=1760156 CPUtime=1181.98 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 793588 1251 0 0 118084 114 0 0 25 0 5 0 598958540 1802399744 411507 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1760156 CPUtime=1182 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 748551 1251 0 0 118099 101 0 0 25 0 5 0 598958540 1802399744 411507 33554432000 4194304 4417332 140735444654400 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1760156 CPUtime=1175.57 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1242239 1251 0 0 117365 192 0 0 25 0 5 0 598958540 1802399744 411507 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1760156 CPUtime=1182.01 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 418545 1251 0 0 118136 65 0 0 25 0 5 0 598958540 1802399744 411507 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364683 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4721.89
Current children cumulated vsize (KiB) 1760156



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.9 s]
/proc/loadavg: 8.04 8.04 7.73 5/180 20722
/proc/meminfo: memFree=29691008/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1307500 CPUtime=4800.21 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 3243952 1251 0 0 479539 482 0 0 25 0 5 0 598958509 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 326875 298343 180 55 0 324761 0
[pid=20676/tid=20683] ppid=20674 vsize=1307500 CPUtime=1201.58 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 804567 1251 0 0 120042 116 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1307500 CPUtime=1201.61 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 753273 1251 0 0 120059 102 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1307500 CPUtime=1195.06 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1250427 1251 0 0 119312 194 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1307500 CPUtime=1201.62 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 418545 1251 0 0 120097 65 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 1307500

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.9 s]
/proc/loadavg: 8.04 8.04 7.73 5/180 20722
/proc/meminfo: memFree=29691008/32951124 swapFree=67111464/67111528
[pid=20676] ppid=20674 vsize=1307500 CPUtime=4800.21 cores=1,3,5,7
/proc/20676/stat : 20676 (plingeling) S 20674 20676 19459 0 -1 4202496 3243952 1251 0 0 479539 482 0 0 25 0 5 0 598958509 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 260032198629 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/20676/statm: 326875 298343 180 55 0 324761 0
[pid=20676/tid=20683] ppid=20674 vsize=1307500 CPUtime=1201.58 cores=1,3,5,7
/proc/20676/task/20683/stat : 20683 (plingeling) R 20674 20676 19459 0 -1 4202560 804567 1251 0 0 120042 116 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20676/tid=20684] ppid=20674 vsize=1307500 CPUtime=1201.61 cores=1,3,5,7
/proc/20676/task/20684/stat : 20684 (plingeling) R 20674 20676 19459 0 -1 4202560 753273 1251 0 0 120059 102 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=20676/tid=20685] ppid=20674 vsize=1307500 CPUtime=1195.06 cores=1,3,5,7
/proc/20676/task/20685/stat : 20685 (plingeling) R 20674 20676 19459 0 -1 4202560 1250427 1251 0 0 119312 194 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20676/tid=20686] ppid=20674 vsize=1307500 CPUtime=1201.62 cores=1,3,5,7
/proc/20676/task/20686/stat : 20686 (plingeling) R 20674 20676 19459 0 -1 4202560 418545 1251 0 0 120097 65 0 0 25 0 5 0 598958540 1338880000 298343 33554432000 4194304 4417332 140735444654400 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 1307500

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.96
CPU time (s): 4800.29
CPU user time (s): 4795.42
CPU system time (s): 4.86926
CPU usage (%): 399.372
Max. virtual memory (cumulated for all children) (KiB): 1976328

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.42
system time used= 4.86926
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3245203
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 114
involuntary context switches= 36739

runsolver used 1.92371 second user time and 4.64029 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 16:27:59
IDJOB=3249668
IDBENCH=83180
IDSOLVER=1605
FILE ID=node110/3249668-1304087279
RUNJOBID= node110-1304087278-20612
PBS_JOBID= 13170503
Free space on /tmp= 72832 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-14x38.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249668-1304087279/watcher-3249668-1304087279 -o /tmp/evaluation-result-3249668-1304087279/solver-3249668-1304087279 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3249668-1304087279.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 56980124bab6f1dbe07ec523b7d59019
RANDOM SEED=457937998

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30854336 kB
Buffers:        893472 kB
Cached:         698244 kB
SwapCached:         64 kB
Active:         158608 kB
Inactive:      1532584 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30854336 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:            7288 kB
Writeback:          56 kB
AnonPages:       98932 kB
Mapped:          15136 kB
Slab:           342252 kB
PageTables:       4488 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   284628 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72832 MiB
End job on node110 at 2011-04-29 16:48:03