Trace number 3371044

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.42 1203.73

General information on the benchmark

NameSAT04/handmade/marijn/equilarge-l/
equilarge_l2.used-as.sat04-894.cnf
MD5SUMb3ddb2ee2bb2b83e1c98b2a9018351fc
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark590.41
Satisfiable
(Un)Satisfiability was proved
Number of variables4487
Number of clauses18555
Sum of the clauses size54310
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21355
Number of clauses of size 317200
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3371044-1305431081/watcher-3371044-1305431081 -o /tmp/evaluation-result-3371044-1305431081/solver-3371044-1305431081 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3371044-1305431081.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.17 8.11 7.61 6/368 16358
/proc/meminfo: memFree=30201448/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=11012 CPUtime=0 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) R 16355 16358 8955 0 -1 4202496 840 1256 0 0 0 0 0 0 25 0 1 0 635524320 11276288 761 33554432000 4194304 4417452 140735490032320 18446744073709551615 212039655168 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/16358/statm: 2753 761 149 55 0 639 0

[startup+0.016014 s]
/proc/loadavg: 8.17 8.11 7.61 6/368 16358
/proc/meminfo: memFree=30201448/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=12300 CPUtime=0.01 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) R 16355 16358 8955 0 -1 4202496 1178 1256 0 0 1 0 0 0 25 0 1 0 635524320 12595200 1099 33554432000 4194304 4417452 140735490032320 18446744073709551615 4274358 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/16358/statm: 3075 1099 149 55 0 961 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 12300

[startup+0.149948 s]
/proc/loadavg: 8.17 8.11 7.61 6/368 16358
/proc/meminfo: memFree=30201448/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=57420 CPUtime=0.5 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 3318 1256 0 0 50 0 0 0 25 0 5 0 635524320 58798080 2140 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 14355 2140 178 55 0 12241 0
Current children cumulated CPU time (s) 0.5
Current children cumulated vsize (KiB) 57420

[startup+0.300921 s]
/proc/loadavg: 8.17 8.11 7.61 6/368 16358
/proc/meminfo: memFree=30201448/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=58940 CPUtime=1.1 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 3747 1256 0 0 110 0 0 0 25 0 5 0 635524320 60354560 2533 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 14735 2533 179 55 0 12621 0
Current children cumulated CPU time (s) 1.1
Current children cumulated vsize (KiB) 58940

[startup+0.70085 s]
/proc/loadavg: 8.17 8.11 7.61 6/368 16358
/proc/meminfo: memFree=30201448/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=59356 CPUtime=2.7 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 3875 1256 0 0 270 0 0 0 25 0 5 0 635524320 60780544 2661 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 14839 2661 180 55 0 12725 0
Current children cumulated CPU time (s) 2.7
Current children cumulated vsize (KiB) 59356

[startup+1.50071 s]
/proc/loadavg: 8.15 8.11 7.62 9/373 16371
/proc/meminfo: memFree=30188436/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=62464 CPUtime=5.89 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 4809 1256 0 0 589 0 0 0 25 0 5 0 635524320 63963136 3339 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 15616 3339 180 55 0 13502 0
[pid=16358/tid=16367] ppid=16355 vsize=62464 CPUtime=1.47 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 1127 1256 0 0 147 0 0 0 25 0 5 0 635524323 63963136 3339 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=62464 CPUtime=1.45 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 631 1256 0 0 145 0 0 0 25 0 5 0 635524323 63963136 3339 33554432000 4194304 4417452 140735490032320 18446744073709551615 4357260 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=62464 CPUtime=1.47 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 773 1256 0 0 147 0 0 0 25 0 5 0 635524323 63963136 3339 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=62464 CPUtime=1.47 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 316 1256 0 0 147 0 0 0 25 0 5 0 635524323 63963136 3339 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.89
Current children cumulated vsize (KiB) 62464

[startup+3.10042 s]
/proc/loadavg: 8.15 8.11 7.62 9/373 16371
/proc/meminfo: memFree=30180500/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=66120 CPUtime=12.28 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 7071 1256 0 0 1228 0 0 0 25 0 5 0 635524320 67706880 4346 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 16530 4346 180 55 0 14416 0
[pid=16358/tid=16367] ppid=16355 vsize=66120 CPUtime=3.07 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 2240 1256 0 0 307 0 0 0 25 0 5 0 635524323 67706880 4346 33554432000 4194304 4417452 140735490032320 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=66120 CPUtime=3.04 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 870 1256 0 0 304 0 0 0 25 0 5 0 635524323 67706880 4346 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=66120 CPUtime=3.07 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 1177 1256 0 0 307 0 0 0 25 0 5 0 635524323 67706880 4346 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364567 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=66120 CPUtime=3.07 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 822 1256 0 0 307 0 0 0 25 0 5 0 635524323 67706880 4346 33554432000 4194304 4417452 140735490032320 18446744073709551615 4236593 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 12.28
Current children cumulated vsize (KiB) 66120

[startup+6.30085 s]
/proc/loadavg: 8.14 8.11 7.62 9/373 16371
/proc/meminfo: memFree=30170340/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=72760 CPUtime=25.04 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 13906 1256 0 0 2503 1 0 0 25 0 5 0 635524320 74506240 5989 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 18190 5989 180 55 0 16076 0
[pid=16358/tid=16367] ppid=16355 vsize=72760 CPUtime=6.27 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 4244 1256 0 0 627 0 0 0 25 0 5 0 635524323 74506240 5989 33554432000 4194304 4417452 140735490032320 18446744073709551615 4250825 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=72760 CPUtime=6.2 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 2844 1256 0 0 620 0 0 0 25 0 5 0 635524323 74506240 5989 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=72760 CPUtime=6.27 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 2109 1256 0 0 627 0 0 0 25 0 5 0 635524323 74506240 5989 33554432000 4194304 4417452 140735490032320 18446744073709551615 4237262 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=72760 CPUtime=6.27 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 2747 1256 0 0 627 0 0 0 25 0 5 0 635524323 74506240 5989 33554432000 4194304 4417452 140735490032320 18446744073709551615 4250980 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 25.04
Current children cumulated vsize (KiB) 72760

[startup+12.7007 s]
/proc/loadavg: 8.13 8.11 7.62 9/373 16415
/proc/meminfo: memFree=30166252/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=76260 CPUtime=50.55 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 27562 1256 0 0 5052 3 0 0 25 0 5 0 635524320 78090240 6870 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 19065 6870 180 55 0 16951 0
[pid=16358/tid=16367] ppid=16355 vsize=76260 CPUtime=12.66 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 8401 1256 0 0 1266 0 0 0 25 0 5 0 635524323 78090240 6870 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=76260 CPUtime=12.52 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 6567 1256 0 0 1251 1 0 0 25 0 5 0 635524323 78090240 6870 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=76260 CPUtime=12.66 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 5236 1256 0 0 1266 0 0 0 25 0 5 0 635524323 78090240 6870 33554432000 4194304 4417452 140735490032320 18446744073709551615 18446744073699065863 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=76260 CPUtime=12.66 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 5396 1256 0 0 1266 0 0 0 25 0 5 0 635524323 78090240 6870 33554432000 4194304 4417452 140735490032320 18446744073709551615 4366141 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 50.55
Current children cumulated vsize (KiB) 76260

[startup+25.5004 s]
/proc/loadavg: 8.10 8.10 7.63 9/373 16415
/proc/meminfo: memFree=30139904/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=88104 CPUtime=101.6 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 57417 1256 0 0 10152 8 0 0 25 0 5 0 635524320 90218496 9832 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 22026 9832 180 55 0 19912 0
[pid=16358/tid=16367] ppid=16355 vsize=88104 CPUtime=25.46 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 16215 1256 0 0 2545 1 0 0 25 0 5 0 635524323 90218496 9832 33554432000 4194304 4417452 140735490032320 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=88104 CPUtime=25.17 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 15934 1256 0 0 2514 3 0 0 25 0 5 0 635524323 90218496 9832 33554432000 4194304 4417452 140735490032320 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=88104 CPUtime=25.45 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 12933 1256 0 0 2544 1 0 0 25 0 5 0 635524323 90218496 9832 33554432000 4194304 4417452 140735490032320 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=88104 CPUtime=25.46 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 10373 1256 0 0 2545 1 0 0 25 0 5 0 635524323 90218496 9832 33554432000 4194304 4417452 140735490032320 18446744073709551615 4236820 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.6
Current children cumulated vsize (KiB) 88104

[startup+51.1008 s]
/proc/loadavg: 8.12 8.11 7.64 9/373 16415
/proc/meminfo: memFree=30083592/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=111916 CPUtime=203.7 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 116301 1256 0 0 20353 17 0 0 25 0 5 0 635524320 114601984 15784 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 27979 15784 180 55 0 25865 0
[pid=16358/tid=16367] ppid=16355 vsize=111916 CPUtime=51.06 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 32120 1256 0 0 5102 4 0 0 25 0 5 0 635524323 114601984 15784 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=111916 CPUtime=50.47 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 34031 1256 0 0 5042 5 0 0 25 0 5 0 635524323 114601984 15784 33554432000 4194304 4417452 140735490032320 18446744073709551615 4354070 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=111916 CPUtime=51.05 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 27405 1256 0 0 5102 3 0 0 25 0 5 0 635524323 114601984 15784 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=111916 CPUtime=51.06 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 20783 1256 0 0 5104 2 0 0 25 0 5 0 635524323 114601984 15784 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364619 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.7
Current children cumulated vsize (KiB) 111916

[startup+102.314 s]
/proc/loadavg: 8.21 8.14 7.67 9/373 16417
/proc/meminfo: memFree=30043332/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=128924 CPUtime=407.93 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 224400 1256 0 0 40759 34 0 0 25 0 5 0 635524320 132018176 20036 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 32231 20036 180 55 0 30117 0
[pid=16358/tid=16367] ppid=16355 vsize=128924 CPUtime=102.28 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 58180 1256 0 0 10219 9 0 0 25 0 5 0 635524323 132018176 20036 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=128924 CPUtime=101.07 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 66614 1256 0 0 10098 9 0 0 25 0 5 0 635524323 132018176 20036 33554432000 4194304 4417452 140735490032320 18446744073709551615 4250697 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=128924 CPUtime=102.25 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 57810 1256 0 0 10217 8 0 0 25 0 5 0 635524323 132018176 20036 33554432000 4194304 4417452 140735490032320 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=128924 CPUtime=102.27 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 39834 1256 0 0 10221 6 0 0 25 0 5 0 635524323 132018176 20036 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 407.93
Current children cumulated vsize (KiB) 128924

[startup+162.301 s]
/proc/loadavg: 8.11 8.12 7.69 9/373 16418
/proc/meminfo: memFree=29997392/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=142056 CPUtime=647.17 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 343817 1256 0 0 64664 53 0 0 25 0 5 0 635524320 145465344 23320 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 35514 23320 180 55 0 33400 0
[pid=16358/tid=16367] ppid=16355 vsize=142056 CPUtime=162.27 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 93574 1256 0 0 16212 15 0 0 25 0 5 0 635524323 145465344 23320 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=142056 CPUtime=160.36 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 95960 1256 0 0 16023 13 0 0 25 0 5 0 635524323 145465344 23320 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=142056 CPUtime=162.24 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 94641 1256 0 0 16209 15 0 0 25 0 5 0 635524323 145465344 23320 33554432000 4194304 4417452 140735490032320 18446744073709551615 4253198 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=142056 CPUtime=162.27 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 57680 1256 0 0 16218 9 0 0 25 0 5 0 635524323 145465344 23320 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 647.17
Current children cumulated vsize (KiB) 142056


################
# More data... #
################

[pid=16358/tid=16368] ppid=16355 vsize=223828 CPUtime=753.28 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 357397 1256 0 0 75280 48 0 0 25 0 5 0 635524323 229199872 43762 33554432000 4194304 4417452 140735490032320 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=223828 CPUtime=762.15 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 256649 1256 0 0 76177 38 0 0 25 0 5 0 635524323 229199872 43762 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=223828 CPUtime=762.3 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 215847 1256 0 0 76199 31 0 0 25 0 5 0 635524323 229199872 43762 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3040.06
Current children cumulated vsize (KiB) 223828

[startup+822.301 s]
/proc/loadavg: 8.06 8.09 7.88 9/373 16438
/proc/meminfo: memFree=29833032/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=239504 CPUtime=3279.31 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1214785 1256 0 0 327752 179 0 0 25 0 5 0 635524320 245252096 47681 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 59876 47681 180 55 0 57762 0
[pid=16358/tid=16367] ppid=16355 vsize=239504 CPUtime=822.27 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 330893 1256 0 0 82175 52 0 0 25 0 5 0 635524323 245252096 47681 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=239504 CPUtime=812.57 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 378711 1256 0 0 81206 51 0 0 25 0 5 0 635524323 245252096 47681 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=239504 CPUtime=822.14 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 271351 1256 0 0 82173 41 0 0 25 0 5 0 635524323 245252096 47681 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=239504 CPUtime=822.28 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 231868 1256 0 0 82195 33 0 0 25 0 5 0 635524323 245252096 47681 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3279.31
Current children cumulated vsize (KiB) 239504

[startup+882.3 s]
/proc/loadavg: 8.13 8.10 7.90 9/373 16440
/proc/meminfo: memFree=29778916/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=274380 CPUtime=3518.58 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1276848 1256 0 0 351670 188 0 0 25 0 5 0 635524320 280965120 56400 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 68595 56400 180 55 0 66481 0
[pid=16358/tid=16367] ppid=16355 vsize=274380 CPUtime=882.26 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 346263 1256 0 0 88172 54 0 0 25 0 5 0 635524323 280965120 56400 33554432000 4194304 4417452 140735490032320 18446744073709551615 4365042 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=274380 CPUtime=871.85 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 397212 1256 0 0 87132 53 0 0 25 0 5 0 635524323 280965120 56400 33554432000 4194304 4417452 140735490032320 18446744073709551615 4354549 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=274380 CPUtime=882.12 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 284891 1256 0 0 88169 43 0 0 25 0 5 0 635524323 280965120 56400 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=274380 CPUtime=882.28 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 246520 1256 0 0 88193 35 0 0 25 0 5 0 635524323 280965120 56400 33554432000 4194304 4417452 140735490032320 18446744073709551615 4353699 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3518.58
Current children cumulated vsize (KiB) 274380

[startup+942.301 s]
/proc/loadavg: 8.05 8.08 7.91 9/373 16441
/proc/meminfo: memFree=29808116/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=255048 CPUtime=3757.88 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1336544 1256 0 0 375592 196 0 0 25 0 5 0 635524320 261169152 51567 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 63762 51567 180 55 0 61648 0
[pid=16358/tid=16367] ppid=16355 vsize=255048 CPUtime=942.26 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 359555 1256 0 0 94170 56 0 0 25 0 5 0 635524323 261169152 51567 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=255048 CPUtime=931.15 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 419653 1256 0 0 93059 56 0 0 25 0 5 0 635524323 261169152 51567 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=255048 CPUtime=942.11 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 295577 1256 0 0 94167 44 0 0 25 0 5 0 635524323 261169152 51567 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=255048 CPUtime=942.29 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 259797 1256 0 0 94192 37 0 0 25 0 5 0 635524323 261169152 51567 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3757.88
Current children cumulated vsize (KiB) 255048

[startup+1002.3 s]
/proc/loadavg: 8.11 8.09 7.92 9/373 16454
/proc/meminfo: memFree=29761612/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=272404 CPUtime=3997.15 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1398695 1256 0 0 399509 206 0 0 25 0 5 0 635524320 278941696 55906 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 68101 55906 180 55 0 65987 0
[pid=16358/tid=16367] ppid=16355 vsize=272404 CPUtime=1002.27 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 372911 1256 0 0 100168 59 0 0 25 0 5 0 635524323 278941696 55906 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=272404 CPUtime=990.44 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 439539 1256 0 0 98985 59 0 0 25 0 5 0 635524323 278941696 55906 33554432000 4194304 4417452 140735490032320 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=272404 CPUtime=1002.11 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 309360 1256 0 0 100165 46 0 0 25 0 5 0 635524323 278941696 55906 33554432000 4194304 4417452 140735490032320 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=272404 CPUtime=1002.29 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 274923 1256 0 0 100189 40 0 0 25 0 5 0 635524323 278941696 55906 33554432000 4194304 4417452 140735490032320 18446744073709551615 4352402 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3997.15
Current children cumulated vsize (KiB) 272404

[startup+1062.3 s]
/proc/loadavg: 8.13 8.10 7.93 9/373 16455
/proc/meminfo: memFree=29775856/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=246496 CPUtime=4236.44 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1455009 1256 0 0 423431 213 0 0 25 0 5 0 635524320 252411904 49429 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 61624 49429 180 55 0 59510 0
[pid=16358/tid=16367] ppid=16355 vsize=246496 CPUtime=1062.26 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 384676 1256 0 0 106166 60 0 0 25 0 5 0 635524323 252411904 49429 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=246496 CPUtime=1049.74 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 460245 1256 0 0 104912 62 0 0 25 0 5 0 635524323 252411904 49429 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=246496 CPUtime=1062.09 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 319192 1256 0 0 106162 47 0 0 25 0 5 0 635524323 252411904 49429 33554432000 4194304 4417452 140735490032320 18446744073709551615 4365150 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=246496 CPUtime=1062.29 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 288934 1256 0 0 106187 42 0 0 25 0 5 0 635524323 252411904 49429 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364809 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4236.44
Current children cumulated vsize (KiB) 246496

[startup+1122.3 s]
/proc/loadavg: 8.08 8.10 7.93 9/373 16457
/proc/meminfo: memFree=29811768/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=228972 CPUtime=4475.71 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1522182 1256 0 0 447348 223 0 0 25 0 5 0 635524320 234467328 45048 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 57243 45048 180 55 0 55129 0
[pid=16358/tid=16367] ppid=16355 vsize=228972 CPUtime=1122.26 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 407540 1256 0 0 112162 64 0 0 25 0 5 0 635524323 234467328 45048 33554432000 4194304 4417452 140735490032320 18446744073709551615 4250476 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=228972 CPUtime=1109.02 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 478188 1256 0 0 110838 64 0 0 25 0 5 0 635524323 234467328 45048 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=228972 CPUtime=1122.09 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 330322 1256 0 0 112159 50 0 0 25 0 5 0 635524323 234467328 45048 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=228972 CPUtime=1122.29 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 304170 1256 0 0 112185 44 0 0 25 0 5 0 635524323 234467328 45048 33554432000 4194304 4417452 140735490032320 18446744073709551615 4366407 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4475.71
Current children cumulated vsize (KiB) 228972

[startup+1182.3 s]
/proc/loadavg: 8.19 8.12 7.95 9/373 16460
/proc/meminfo: memFree=29764340/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=264600 CPUtime=4715.01 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1595612 1256 0 0 471268 233 0 0 25 0 5 0 635524320 270950400 53955 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 66150 53955 180 55 0 64036 0
[pid=16358/tid=16367] ppid=16355 vsize=264600 CPUtime=1182.26 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 433723 1256 0 0 118159 67 0 0 25 0 5 0 635524323 270950400 53955 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=264600 CPUtime=1168.33 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 499889 1256 0 0 116766 67 0 0 25 0 5 0 635524323 270950400 53955 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=264600 CPUtime=1182.08 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 340362 1256 0 0 118157 51 0 0 25 0 5 0 635524323 270950400 53955 33554432000 4194304 4417452 140735490032320 18446744073709551615 4365042 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=264600 CPUtime=1182.29 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 319676 1256 0 0 118183 46 0 0 25 0 5 0 635524323 270950400 53955 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364690 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4715.01
Current children cumulated vsize (KiB) 264600



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.7 s]
/proc/loadavg: 8.14 8.11 7.95 9/373 16460
/proc/meminfo: memFree=29732084/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=268508 CPUtime=4800.37 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1617086 1256 0 0 479802 235 0 0 25 0 5 0 635524320 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 67127 54932 180 55 0 65013 0
[pid=16358/tid=16367] ppid=16355 vsize=268508 CPUtime=1203.68 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 440228 1256 0 0 120300 68 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=268508 CPUtime=1189.46 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 507120 1256 0 0 118879 67 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=268508 CPUtime=1203.48 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 343080 1256 0 0 120297 51 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=268508 CPUtime=1203.7 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 324696 1256 0 0 120324 46 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 268508

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.7 s]
/proc/loadavg: 8.14 8.11 7.95 9/373 16460
/proc/meminfo: memFree=29732084/32951124 swapFree=51873460/67111528
[pid=16358] ppid=16355 vsize=268508 CPUtime=4800.37 cores=0,2,4,6
/proc/16358/stat : 16358 (plingeling) S 16355 16358 8955 0 -1 4202496 1617086 1256 0 0 479802 235 0 0 25 0 5 0 635524320 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 212051458021 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/16358/statm: 67127 54932 180 55 0 65013 0
[pid=16358/tid=16367] ppid=16355 vsize=268508 CPUtime=1203.68 cores=0,2,4,6
/proc/16358/task/16367/stat : 16367 (plingeling) R 16355 16358 8955 0 -1 4202560 440228 1256 0 0 120300 68 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16358/tid=16368] ppid=16355 vsize=268508 CPUtime=1189.46 cores=0,2,4,6
/proc/16358/task/16368/stat : 16368 (plingeling) R 16355 16358 8955 0 -1 4202560 507120 1256 0 0 118879 67 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16358/tid=16370] ppid=16355 vsize=268508 CPUtime=1203.48 cores=0,2,4,6
/proc/16358/task/16370/stat : 16370 (plingeling) R 16355 16358 8955 0 -1 4202560 343080 1256 0 0 120297 51 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16358/tid=16371] ppid=16355 vsize=268508 CPUtime=1203.7 cores=0,2,4,6
/proc/16358/task/16371/stat : 16371 (plingeling) R 16355 16358 8955 0 -1 4202560 324696 1256 0 0 120324 46 0 0 25 0 5 0 635524323 274952192 54932 33554432000 4194304 4417452 140735490032320 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 268508

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.73
CPU time (s): 4800.42
CPU user time (s): 4798.06
CPU system time (s): 2.36864
CPU usage (%): 398.796
Max. virtual memory (cumulated for all children) (KiB): 283556

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4798.06
system time used= 2.36864
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1618342
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 52
involuntary context switches= 23927

runsolver used 4.08938 second user time and 10.1565 second system time

The end

Launcher Data 

Begin job on node146 at 2011-05-15 05:44:41
IDJOB=3371044
IDBENCH=84761
IDSOLVER=1848
FILE ID=node146/3371044-1305431081
RUNJOBID= node146-1305428668-16108
PBS_JOBID= 13324589
Free space on /tmp= 73336 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT04/handmade/marijn/equilarge-l/equilarge_l2.used-as.sat04-894.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3371044-1305431081/watcher-3371044-1305431081 -o /tmp/evaluation-result-3371044-1305431081/solver-3371044-1305431081 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3371044-1305431081.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b3ddb2ee2bb2b83e1c98b2a9018351fc
RANDOM SEED=1257100552

node146.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30201728 kB
Buffers:        101220 kB
Cached:         169860 kB
SwapCached:      56148 kB
Active:         774136 kB
Inactive:      1746352 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30201728 kB
SwapTotal:    67111528 kB
SwapFree:     51873460 kB
Dirty:             412 kB
Writeback:           0 kB
AnonPages:     2200240 kB
Mapped:          17340 kB
Slab:           104024 kB
PageTables:      58892 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 81823756 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73336 MiB
End job on node146 at 2011-05-15 06:04:46