Trace number 3251032

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.44 1202.33

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1504.5-n60-p15-s80450670.cnf
MD5SUM78d8e2b8c0fd15b93f96e38e33701f17
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark5.61315
Satisfiable
(Un)Satisfiability was proved
Number of variables3510
Number of clauses974465
Sum of the clauses size1952320
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2974405
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251032-1304115090/watcher-3251032-1304115090 -o /tmp/evaluation-result-3251032-1304115090/solver-3251032-1304115090 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3251032-1304115090.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.92 7.35 7.70 5/181 16162
/proc/meminfo: memFree=25135568/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=10160 CPUtime=0 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) R 16160 16162 15661 0 -1 4202496 617 1252 0 0 0 0 0 0 18 0 1 0 503925214 10403840 545 33554432000 4194304 4417332 140735165732560 18446744073709551615 4264914 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/16162/statm: 2540 545 146 55 0 426 0

[startup+0.0087839 s]
/proc/loadavg: 5.92 7.35 7.70 5/181 16162
/proc/meminfo: memFree=25135568/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=10352 CPUtime=0 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) R 16160 16162 15661 0 -1 4202496 681 1252 0 0 0 0 0 0 18 0 1 0 503925214 10600448 609 33554432000 4194304 4417332 140735165732560 18446744073709551615 4209566 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/16162/statm: 2588 609 146 55 0 474 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 10352

[startup+0.100797 s]
/proc/loadavg: 5.92 7.35 7.70 5/181 16162
/proc/meminfo: memFree=25135568/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=18432 CPUtime=0.09 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) R 16160 16162 15661 0 -1 4202496 2734 1252 0 0 9 0 0 0 18 0 1 0 503925214 18874368 2630 33554432000 4194304 4417332 140735165732560 18446744073709551615 255106768640 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/16162/statm: 4608 2630 147 55 0 2494 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18432

[startup+0.300787 s]
/proc/loadavg: 5.92 7.35 7.70 5/181 16162
/proc/meminfo: memFree=25135568/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=43008 CPUtime=0.29 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) R 16160 16162 15661 0 -1 4202496 8878 1252 0 0 28 1 0 0 19 0 1 0 503925214 44040192 8774 33554432000 4194304 4417332 140735165732560 18446744073709551615 4209703 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/16162/statm: 10752 8774 147 55 0 8638 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 43008

[startup+0.70065 s]
/proc/loadavg: 5.92 7.35 7.70 5/181 16162
/proc/meminfo: memFree=25135568/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=182780 CPUtime=0.84 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 33596 1252 0 0 80 4 0 0 19 0 5 0 503925214 187166720 33468 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 45695 33468 166 55 0 43581 0
Current children cumulated CPU time (s) 0.84
Current children cumulated vsize (KiB) 182780

[startup+1.50051 s]
/proc/loadavg: 5.92 7.35 7.70 9/186 16172
/proc/meminfo: memFree=24979800/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=279572 CPUtime=4.03 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 62973 1252 0 0 395 8 0 0 19 0 5 0 503925214 286281728 45407 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 69893 45407 177 55 0 67779 0
[pid=16162/tid=16169] ppid=16160 vsize=279572 CPUtime=0.84 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 8250 1252 0 0 83 1 0 0 23 0 5 0 503925279 286281728 45407 33554432000 4194304 4417332 140735165732560 18446744073709551615 4255998 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=279572 CPUtime=0.83 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 6226 1252 0 0 83 0 0 0 23 0 5 0 503925279 286281728 45407 33554432000 4194304 4417332 140735165732560 18446744073709551615 4261238 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=279572 CPUtime=0.83 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 8427 1252 0 0 83 0 0 0 23 0 5 0 503925279 286281728 45407 33554432000 4194304 4417332 140735165732560 18446744073709551615 4255683 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=279572 CPUtime=0.84 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 6604 1252 0 0 84 0 0 0 23 0 5 0 503925279 286281728 45407 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4.03
Current children cumulated vsize (KiB) 279572

[startup+3.10022 s]
/proc/loadavg: 5.92 7.35 7.70 9/186 16172
/proc/meminfo: memFree=24945956/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=316400 CPUtime=10.42 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 85386 1252 0 0 1032 10 0 0 19 0 5 0 503925214 323993600 50536 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 79100 50536 180 55 0 76986 0
[pid=16162/tid=16169] ppid=16160 vsize=316400 CPUtime=2.44 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 11899 1252 0 0 243 1 0 0 25 0 5 0 503925279 323993600 50536 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249958 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=316400 CPUtime=2.43 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 11168 1252 0 0 242 1 0 0 25 0 5 0 503925279 323993600 50536 33554432000 4194304 4417332 140735165732560 18446744073709551615 4250017 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=316400 CPUtime=2.44 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 12514 1252 0 0 243 1 0 0 25 0 5 0 503925279 323993600 50536 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249950 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=316400 CPUtime=2.44 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 16339 1252 0 0 243 1 0 0 25 0 5 0 503925279 323993600 50536 33554432000 4194304 4417332 140735165732560 18446744073709551615 4365991 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 10.42
Current children cumulated vsize (KiB) 316400

[startup+6.30066 s]
/proc/loadavg: 6.09 7.36 7.71 9/186 16172
/proc/meminfo: memFree=24884324/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=342844 CPUtime=23.21 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 99098 1252 0 0 2309 12 0 0 19 0 5 0 503925214 351072256 57147 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 85711 57147 180 55 0 83597 0
[pid=16162/tid=16169] ppid=16160 vsize=342844 CPUtime=5.64 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 11899 1252 0 0 563 1 0 0 25 0 5 0 503925279 351072256 57147 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249292 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=342844 CPUtime=5.62 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 11168 1252 0 0 561 1 0 0 25 0 5 0 503925279 351072256 57147 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=342844 CPUtime=5.64 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 12514 1252 0 0 563 1 0 0 25 0 5 0 503925279 351072256 57147 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=342844 CPUtime=5.64 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 30051 1252 0 0 561 3 0 0 25 0 5 0 503925279 351072256 57147 33554432000 4194304 4417332 140735165732560 18446744073709551615 4356899 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 23.21
Current children cumulated vsize (KiB) 342844

[startup+12.7005 s]
/proc/loadavg: 6.32 7.39 7.71 9/186 16172
/proc/meminfo: memFree=24790480/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=418004 CPUtime=48.79 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 131880 1252 0 0 4862 17 0 0 19 0 5 0 503925214 428036096 75938 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 104501 75938 181 55 0 102387 0
[pid=16162/tid=16169] ppid=16160 vsize=418004 CPUtime=12.04 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 11899 1252 0 0 1203 1 0 0 25 0 5 0 503925279 428036096 75938 33554432000 4194304 4417332 140735165732560 18446744073709551615 4258377 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=418004 CPUtime=11.99 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 11168 1252 0 0 1198 1 0 0 25 0 5 0 503925279 428036096 75938 33554432000 4194304 4417332 140735165732560 18446744073709551615 4258449 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=418004 CPUtime=12.04 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 12515 1252 0 0 1203 1 0 0 25 0 5 0 503925279 428036096 75938 33554432000 4194304 4417332 140735165732560 18446744073709551615 4376577 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=418004 CPUtime=12.04 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 62832 1252 0 0 1196 8 0 0 25 0 5 0 503925279 428036096 75938 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364652 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 48.79
Current children cumulated vsize (KiB) 418004

[startup+25.5003 s]
/proc/loadavg: 6.69 7.42 7.72 9/186 16172
/proc/meminfo: memFree=24680484/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=501324 CPUtime=99.92 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 192099 1252 0 0 9965 27 0 0 19 0 5 0 503925214 513355776 94953 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 125331 94953 181 55 0 123217 0
[pid=16162/tid=16169] ppid=16160 vsize=501324 CPUtime=24.84 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 12131 1252 0 0 2483 1 0 0 25 0 5 0 503925279 513355776 94953 33554432000 4194304 4417332 140735165732560 18446744073709551615 4254433 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=501324 CPUtime=24.72 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 11168 1252 0 0 2471 1 0 0 25 0 5 0 503925279 513355776 94953 33554432000 4194304 4417332 140735165732560 18446744073709551615 4320251 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=501324 CPUtime=24.84 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 13461 1252 0 0 2483 1 0 0 25 0 5 0 503925279 513355776 94953 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=501324 CPUtime=24.84 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 121873 1252 0 0 2466 18 0 0 25 0 5 0 503925279 513355776 94953 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364713 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.92
Current children cumulated vsize (KiB) 501324

[startup+51.1008 s]
/proc/loadavg: 7.19 7.48 7.73 9/186 16173
/proc/meminfo: memFree=24534180/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=592280 CPUtime=202.19 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 272194 1252 0 0 20178 41 0 0 19 0 5 0 503925214 606494720 119001 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 148070 119001 181 55 0 145956 0
[pid=16162/tid=16169] ppid=16160 vsize=592280 CPUtime=50.45 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 16174 1252 0 0 5043 2 0 0 25 0 5 0 503925279 606494720 119001 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=592280 CPUtime=50.2 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 12828 1252 0 0 5019 1 0 0 25 0 5 0 503925279 606494720 119001 33554432000 4194304 4417332 140735165732560 18446744073709551615 4258394 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=592280 CPUtime=50.44 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 18317 1252 0 0 5042 2 0 0 25 0 5 0 503925279 606494720 119001 33554432000 4194304 4417332 140735165732560 18446744073709551615 4236549 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=592280 CPUtime=50.44 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 191409 1252 0 0 5013 31 0 0 25 0 5 0 503925279 606494720 119001 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.19
Current children cumulated vsize (KiB) 592280

[startup+102.307 s]
/proc/loadavg: 7.76 7.59 7.75 9/186 16174
/proc/meminfo: memFree=23906860/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=825568 CPUtime=406.78 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 397670 1252 0 0 40619 59 0 0 19 0 5 0 503925214 845381632 177829 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 206392 177829 181 55 0 204278 0
[pid=16162/tid=16169] ppid=16160 vsize=825568 CPUtime=101.65 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 16174 1252 0 0 10163 2 0 0 25 0 5 0 503925279 845381632 177829 33554432000 4194304 4417332 140735165732560 18446744073709551615 4248238 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=825568 CPUtime=101.16 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 12828 1252 0 0 10115 1 0 0 25 0 5 0 503925279 845381632 177829 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=825568 CPUtime=101.65 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 29477 1252 0 0 10162 3 0 0 25 0 5 0 503925279 845381632 177829 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=825568 CPUtime=101.65 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 305725 1252 0 0 10118 47 0 0 25 0 5 0 503925279 845381632 177829 33554432000 4194304 4417332 140735165732560 18446744073709551615 4320938 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.78
Current children cumulated vsize (KiB) 825568

[startup+162.3 s]
/proc/loadavg: 8.01 7.69 7.77 9/186 16176
/proc/meminfo: memFree=23702864/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=964044 CPUtime=646.48 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 503031 1252 0 0 64574 74 0 0 19 0 5 0 503925214 987181056 212480 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 241011 212480 181 55 0 238897 0
[pid=16162/tid=16169] ppid=16160 vsize=964044 CPUtime=161.65 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 17396 1252 0 0 16163 2 0 0 25 0 5 0 503925279 987181056 212480 33554432000 4194304 4417332 140735165732560 18446744073709551615 4258604 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=964044 CPUtime=160.87 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 12828 1252 0 0 16086 1 0 0 25 0 5 0 503925279 987181056 212480 33554432000 4194304 4417332 140735165732560 18446744073709551615 4258604 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=964044 CPUtime=161.64 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 38158 1252 0 0 16160 4 0 0 25 0 5 0 503925279 987181056 212480 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=964044 CPUtime=161.65 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 401183 1252 0 0 16104 61 0 0 25 0 5 0 503925279 987181056 212480 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.48
Current children cumulated vsize (KiB) 964044


################
# More data... #
################

/proc/meminfo: memFree=20614436/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2182296 CPUtime=3043.65 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 1524867 1252 0 0 304144 221 0 0 19 0 5 0 503925214 2234671104 517011 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 545574 517011 181 55 0 543460 0
[pid=16162/tid=16169] ppid=16160 vsize=2182296 CPUtime=761.65 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 200354 1252 0 0 76139 26 0 0 25 0 5 0 503925279 2234671104 517011 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2182296 CPUtime=758.51 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 108880 1252 0 0 75836 15 0 0 25 0 5 0 503925279 2234671104 517011 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2182296 CPUtime=761.14 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 208829 1252 0 0 76083 31 0 0 25 0 5 0 503925279 2234671104 517011 33554432000 4194304 4417332 140735165732560 18446744073709551615 4255645 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2182296 CPUtime=761.67 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 973338 1252 0 0 76024 143 0 0 25 0 5 0 503925279 2234671104 517011 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3043.65
Current children cumulated vsize (KiB) 2182296

[startup+822.301 s]
/proc/loadavg: 8.27 8.08 7.93 10/187 16197
/proc/meminfo: memFree=20213736/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2365484 CPUtime=3282.82 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 1595808 1252 0 0 328052 230 0 0 19 0 5 0 503925214 2422255616 562807 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 591371 562807 181 55 0 589257 0
[pid=16162/tid=16169] ppid=16160 vsize=2365484 CPUtime=821.65 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 205652 1252 0 0 82138 27 0 0 25 0 5 0 503925279 2422255616 562807 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364734 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2365484 CPUtime=818.36 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 134587 1252 0 0 81817 19 0 0 25 0 5 0 503925279 2422255616 562807 33554432000 4194304 4417332 140735165732560 18446744073709551615 4236319 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2365484 CPUtime=820.47 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 248765 1252 0 0 82012 35 0 0 25 0 5 0 503925279 2422255616 562807 33554432000 4194304 4417332 140735165732560 18446744073709551615 4308204 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2365484 CPUtime=821.66 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 973338 1252 0 0 82023 143 0 0 25 0 5 0 503925279 2422255616 562807 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3282.82
Current children cumulated vsize (KiB) 2365484

[startup+882.301 s]
/proc/loadavg: 8.18 8.09 7.94 9/186 16198
/proc/meminfo: memFree=20566516/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=1988808 CPUtime=3522.23 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 1844297 1252 0 0 351953 270 0 0 19 0 5 0 503925214 2036539392 468639 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 497202 468639 181 55 0 495088 0
[pid=16162/tid=16169] ppid=16160 vsize=1988808 CPUtime=881.66 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 254399 1252 0 0 88130 36 0 0 25 0 5 0 503925279 2036539392 468639 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=1988808 CPUtime=878.35 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 157604 1252 0 0 87812 23 0 0 25 0 5 0 503925279 2036539392 468639 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=1988808 CPUtime=879.88 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 296300 1252 0 0 87943 45 0 0 25 0 5 0 503925279 2036539392 468639 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=1988808 CPUtime=881.66 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1102528 1252 0 0 88006 160 0 0 25 0 5 0 503925279 2036539392 468639 33554432000 4194304 4417332 140735165732560 18446744073709551615 4354410 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3522.23
Current children cumulated vsize (KiB) 1988808

[startup+942.301 s]
/proc/loadavg: 8.11 8.09 7.94 9/186 16200
/proc/meminfo: memFree=21162180/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2071076 CPUtime=3761.95 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 1955407 1252 0 0 375909 286 0 0 19 0 5 0 503925214 2120781824 489206 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 517769 489206 181 55 0 515655 0
[pid=16162/tid=16169] ppid=16160 vsize=2071076 CPUtime=941.65 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 275930 1252 0 0 94127 38 0 0 25 0 5 0 503925279 2120781824 489206 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364565 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2071076 CPUtime=938.36 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 172168 1252 0 0 93811 25 0 0 25 0 5 0 503925279 2120781824 489206 33554432000 4194304 4417332 140735165732560 18446744073709551615 4307511 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2071076 CPUtime=939.59 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 371315 1252 0 0 93903 56 0 0 25 0 5 0 503925279 2120781824 489206 33554432000 4194304 4417332 140735165732560 18446744073709551615 4354410 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2071076 CPUtime=941.66 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1102528 1252 0 0 94006 160 0 0 25 0 5 0 503925279 2120781824 489206 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3761.95
Current children cumulated vsize (KiB) 2071076

[startup+1002.3 s]
/proc/loadavg: 8.08 8.08 7.95 9/186 16201
/proc/meminfo: memFree=20963396/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=1933604 CPUtime=4001.66 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 2044452 1252 0 0 399867 299 0 0 19 0 5 0 503925214 1980010496 454838 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 483401 454838 181 55 0 481287 0
[pid=16162/tid=16169] ppid=16160 vsize=1933604 CPUtime=1001.66 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 296772 1252 0 0 100125 41 0 0 25 0 5 0 503925279 1980010496 454838 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=1933604 CPUtime=998.36 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 209973 1252 0 0 99806 30 0 0 25 0 5 0 503925279 1980010496 454838 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364990 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=1933604 CPUtime=999.31 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 384643 1252 0 0 99872 59 0 0 25 0 5 0 503925279 1980010496 454838 33554432000 4194304 4417332 140735165732560 18446744073709551615 4249584 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=1933604 CPUtime=1001.66 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1119598 1252 0 0 100002 164 0 0 25 0 5 0 503925279 1980010496 454838 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4001.66
Current children cumulated vsize (KiB) 1933604

[startup+1062.3 s]
/proc/loadavg: 8.07 8.08 7.95 9/186 16203
/proc/meminfo: memFree=20707928/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2159980 CPUtime=4241.37 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 2132841 1252 0 0 423822 315 0 0 19 0 5 0 503925214 2211819520 511431 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 539995 511431 181 55 0 537881 0
[pid=16162/tid=16169] ppid=16160 vsize=2159980 CPUtime=1061.66 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 300983 1252 0 0 106124 42 0 0 25 0 5 0 503925279 2211819520 511431 33554432000 4194304 4417332 140735165732560 18446744073709551615 4320251 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2159980 CPUtime=1058.36 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 224945 1252 0 0 105805 31 0 0 25 0 5 0 503925279 2211819520 511431 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2159980 CPUtime=1059.01 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 411610 1252 0 0 105838 63 0 0 25 0 5 0 503925279 2211819520 511431 33554432000 4194304 4417332 140735165732560 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2159980 CPUtime=1061.66 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1161837 1252 0 0 105993 173 0 0 25 0 5 0 503925279 2211819520 511431 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4241.37
Current children cumulated vsize (KiB) 2159980

[startup+1122.3 s]
/proc/loadavg: 8.11 8.09 7.96 9/186 16205
/proc/meminfo: memFree=20467104/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2371704 CPUtime=4481.05 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 2239838 1252 0 0 447775 330 0 0 19 0 5 0 503925214 2428624896 564363 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 592926 564363 181 55 0 590812 0
[pid=16162/tid=16169] ppid=16160 vsize=2371704 CPUtime=1121.66 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 339003 1252 0 0 112119 47 0 0 25 0 5 0 503925279 2428624896 564363 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364465 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2371704 CPUtime=1118.36 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 251128 1252 0 0 111801 35 0 0 25 0 5 0 503925279 2428624896 564363 33554432000 4194304 4417332 140735165732560 18446744073709551615 4311285 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2371704 CPUtime=1118.7 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 445119 1252 0 0 111802 68 0 0 25 0 5 0 503925279 2428624896 564363 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364753 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2371704 CPUtime=1121.66 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1171122 1252 0 0 111992 174 0 0 25 0 5 0 503925279 2428624896 564363 33554432000 4194304 4417332 140735165732560 18446744073709551615 4250432 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4481.05
Current children cumulated vsize (KiB) 2371704

[startup+1182.3 s]
/proc/loadavg: 8.08 8.09 7.97 9/186 16206
/proc/meminfo: memFree=20107724/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2659620 CPUtime=4720.76 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 2399971 1252 0 0 471716 360 0 0 19 0 5 0 503925214 2723450880 636342 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 664905 636342 181 55 0 662791 0
[pid=16162/tid=16169] ppid=16160 vsize=2659620 CPUtime=1181.66 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 359962 1252 0 0 118115 51 0 0 25 0 5 0 503925279 2723450880 636342 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364465 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2659620 CPUtime=1178.36 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 270962 1252 0 0 117797 39 0 0 25 0 5 0 503925279 2723450880 636342 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2659620 CPUtime=1178.4 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 474331 1252 0 0 117767 73 0 0 25 0 5 0 503925279 2723450880 636342 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364713 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2659620 CPUtime=1181.67 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1261250 1252 0 0 117975 192 0 0 25 0 5 0 503925279 2723450880 636342 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4720.76
Current children cumulated vsize (KiB) 2659620



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.12 8.10 7.97 9/186 16244
/proc/meminfo: memFree=22684792/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=2592284 CPUtime=4800.29 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) S 16160 16162 15661 0 -1 4202496 2409187 1252 0 0 479667 362 0 0 19 0 5 0 503925214 2654498816 619508 33554432000 4194304 4417332 140735165732560 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 648071 619508 181 55 0 645957 0
[pid=16162/tid=16169] ppid=16160 vsize=2592284 CPUtime=1201.57 cores=0,2,4,6
/proc/16162/task/16169/stat : 16169 (plingeling) R 16160 16162 15661 0 -1 4202560 367823 1252 0 0 120105 52 0 0 25 0 5 0 503925279 2654498816 619508 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16162/tid=16170] ppid=16160 vsize=2592284 CPUtime=1198.26 cores=0,2,4,6
/proc/16162/task/16170/stat : 16170 (plingeling) R 16160 16162 15661 0 -1 4202560 271714 1252 0 0 119787 39 0 0 25 0 5 0 503925279 2654498816 619508 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=2592284 CPUtime=1198.2 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4202560 474934 1252 0 0 119747 73 0 0 25 0 5 0 503925279 2654498816 619508 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16162/tid=16172] ppid=16160 vsize=2592284 CPUtime=1201.58 cores=0,2,4,6
/proc/16162/task/16172/stat : 16172 (plingeling) R 16160 16162 15661 0 -1 4202560 1261250 1252 0 0 119966 192 0 0 25 0 5 0 503925279 2654498816 619508 33554432000 4194304 4417332 140735165732560 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 2592284

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.32 s]
/proc/loadavg: 8.12 8.10 7.97 9/186 16244
/proc/meminfo: memFree=22684792/32951124 swapFree=67111448/67111528
[pid=16162] ppid=16160 vsize=0 CPUtime=4800.42 cores=0,2,4,6
/proc/16162/stat : 16162 (plingeling) Z 16160 16162 15661 0 -1 4203548 2409187 1252 0 0 479670 372 0 0 18 0 2 0 503925214 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16162/statm: 0 0 0 0 0 0 0
[pid=16162/tid=16171] ppid=16160 vsize=0 CPUtime=1198.31 cores=0,2,4,6
/proc/16162/task/16171/stat : 16171 (plingeling) R 16160 16162 15661 0 -1 4203588 474934 1252 0 0 119748 83 0 0 25 0 2 0 503925279 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.42
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.33
CPU time (s): 4800.44
CPU user time (s): 4796.7
CPU system time (s): 3.74043
CPU usage (%): 399.262
Max. virtual memory (cumulated for all children) (KiB): 2681964

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.7
system time used= 3.74043
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2410439
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 105
involuntary context switches= 35755

runsolver used 1.67674 second user time and 4.06838 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-30 00:11:31
IDJOB=3251032
IDBENCH=88208
IDSOLVER=1605
FILE ID=node144/3251032-1304115090
RUNJOBID= node144-1304115079-16096
PBS_JOBID= 13170685
Free space on /tmp= 72392 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1504.5-n60-p15-s80450670.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251032-1304115090/watcher-3251032-1304115090 -o /tmp/evaluation-result-3251032-1304115090/solver-3251032-1304115090 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3251032-1304115090.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 78d8e2b8c0fd15b93f96e38e33701f17
RANDOM SEED=273020879

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25135848 kB
Buffers:       2108152 kB
Cached:        4951136 kB
SwapCached:         80 kB
Active:         306412 kB
Inactive:      6964464 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25135848 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           19708 kB
Writeback:           0 kB
AnonPages:      211068 kB
Mapped:          15456 kB
Slab:           481256 kB
PageTables:       4892 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   397236 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72392 MiB
End job on node144 at 2011-04-30 00:31:35