Trace number 3371032

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.41 1202.68

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_60_29_30_6-dir.shuffled-as.sat03-130.cnf
MD5SUMe826117d87949a5b285e72732bfa395b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables870
Number of clauses69964
Sum of the clauses size140740
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 269935
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 529

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3371032-1305429968/watcher-3371032-1305429968 -o /tmp/evaluation-result-3371032-1305429968/solver-3371032-1305429968 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3371032-1305429968.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.80 7.70 6.32 2/269 2391
/proc/meminfo: memFree=20972808/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=9328 CPUtime=0 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) R 2389 2391 27712 0 -1 4202496 424 1259 0 0 0 0 0 0 24 0 1 0 654596540 9551872 354 33554432000 4194304 4417452 140735873305104 18446744073709551615 237278879488 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/2391/statm: 2332 354 147 55 0 218 0

[startup+0.0168591 s]
/proc/loadavg: 6.80 7.70 6.32 2/269 2391
/proc/meminfo: memFree=20972808/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=11200 CPUtime=0.01 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) R 2389 2391 27712 0 -1 4202496 941 1259 0 0 1 0 0 0 24 0 1 0 654596540 11468800 823 33554432000 4194304 4417452 140735873305104 18446744073709551615 4262062 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/2391/statm: 2800 823 148 55 0 686 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 11200

[startup+0.157837 s]
/proc/loadavg: 6.80 7.70 6.32 2/269 2391
/proc/meminfo: memFree=20972808/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=61748 CPUtime=0.43 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 4337 1259 0 0 43 0 0 0 24 0 5 0 654596540 63229952 3145 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 15437 3145 176 55 0 13323 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 61748

[startup+0.300811 s]
/proc/loadavg: 6.80 7.70 6.32 2/269 2391
/proc/meminfo: memFree=20972808/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=63476 CPUtime=1 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 5458 1259 0 0 100 0 0 0 24 0 5 0 654596540 64999424 3655 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 15869 3655 179 55 0 13755 0
Current children cumulated CPU time (s) 1
Current children cumulated vsize (KiB) 63476

[startup+0.700739 s]
/proc/loadavg: 6.80 7.70 6.32 2/269 2391
/proc/meminfo: memFree=20972808/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=67076 CPUtime=2.6 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 7295 1259 0 0 260 0 0 0 24 0 5 0 654596540 68685824 4555 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 16769 4555 179 55 0 14655 0
Current children cumulated CPU time (s) 2.6
Current children cumulated vsize (KiB) 67076

[startup+1.5006 s]
/proc/loadavg: 6.80 7.70 6.32 9/281 2416
/proc/meminfo: memFree=20931800/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=76820 CPUtime=5.8 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 11117 1259 0 0 579 1 0 0 24 0 5 0 654596540 78663680 6991 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 19205 6991 179 55 0 17091 0
[pid=2391/tid=2413] ppid=2389 vsize=76820 CPUtime=1.44 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 1091 1259 0 0 144 0 0 0 25 0 5 0 654596545 78663680 6991 33554432000 4194304 4417452 140735873305104 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=76820 CPUtime=1.42 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 166 1259 0 0 142 0 0 0 25 0 5 0 654596545 78663680 6991 33554432000 4194304 4417452 140735873305104 18446744073709551615 4236300 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=76820 CPUtime=1.43 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 565 1259 0 0 143 0 0 0 25 0 5 0 654596545 78663680 6991 33554432000 4194304 4417452 140735873305104 18446744073709551615 4356796 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=76820 CPUtime=1.42 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 6806 1259 0 0 142 0 0 0 25 0 5 0 654596545 78663680 6991 33554432000 4194304 4417452 140735873305104 18446744073709551615 4356995 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 5.8
Current children cumulated vsize (KiB) 76820

[startup+3.10031 s]
/proc/loadavg: 6.80 7.70 6.32 9/281 2416
/proc/meminfo: memFree=20922128/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=85452 CPUtime=12.18 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 18885 1259 0 0 1216 2 0 0 24 0 5 0 654596540 87502848 9149 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 21363 9149 179 55 0 19249 0
[pid=2391/tid=2413] ppid=2389 vsize=85452 CPUtime=3.04 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 1541 1259 0 0 304 0 0 0 25 0 5 0 654596545 87502848 9149 33554432000 4194304 4417452 140735873305104 18446744073709551615 4249193 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=85452 CPUtime=3.02 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 616 1259 0 0 302 0 0 0 25 0 5 0 654596545 87502848 9149 33554432000 4194304 4417452 140735873305104 18446744073709551615 4249151 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=85452 CPUtime=3.03 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 2159 1259 0 0 303 0 0 0 25 0 5 0 654596545 87502848 9149 33554432000 4194304 4417452 140735873305104 18446744073709551615 4310483 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=85452 CPUtime=3.02 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 12080 1259 0 0 301 1 0 0 25 0 5 0 654596545 87502848 9149 33554432000 4194304 4417452 140735873305104 18446744073709551615 4250502 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12.18
Current children cumulated vsize (KiB) 85452

[startup+6.30075 s]
/proc/loadavg: 6.90 7.70 6.33 9/281 2416
/proc/meminfo: memFree=20894784/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=98740 CPUtime=24.95 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 27744 1259 0 0 2492 3 0 0 24 0 5 0 654596540 101109760 12471 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 24685 12471 179 55 0 22571 0
[pid=2391/tid=2413] ppid=2389 vsize=98740 CPUtime=6.24 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 1910 1259 0 0 624 0 0 0 25 0 5 0 654596545 101109760 12471 33554432000 4194304 4417452 140735873305104 18446744073709551615 4357536 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=98740 CPUtime=6.22 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 714 1259 0 0 622 0 0 0 25 0 5 0 654596545 101109760 12471 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=98740 CPUtime=6.23 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 5380 1259 0 0 623 0 0 0 25 0 5 0 654596545 101109760 12471 33554432000 4194304 4417452 140735873305104 18446744073709551615 4366458 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=98740 CPUtime=6.19 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 17251 1259 0 0 617 2 0 0 25 0 5 0 654596545 101109760 12471 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.95
Current children cumulated vsize (KiB) 98740

[startup+12.7006 s]
/proc/loadavg: 6.99 7.71 6.33 9/281 2416
/proc/meminfo: memFree=20824216/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=126512 CPUtime=50.5 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 51964 1259 0 0 5043 7 0 0 24 0 5 0 654596540 129548288 19414 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 31628 19414 179 55 0 29514 0
[pid=2391/tid=2413] ppid=2389 vsize=126512 CPUtime=12.64 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 4585 1259 0 0 1264 0 0 0 25 0 5 0 654596545 129548288 19414 33554432000 4194304 4417452 140735873305104 18446744073709551615 4263224 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=126512 CPUtime=12.61 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 4369 1259 0 0 1261 0 0 0 25 0 5 0 654596545 129548288 19414 33554432000 4194304 4417452 140735873305104 18446744073709551615 4366528 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=126512 CPUtime=12.63 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 11632 1259 0 0 1262 1 0 0 25 0 5 0 654596545 129548288 19414 33554432000 4194304 4417452 140735873305104 18446744073709551615 4305888 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=126512 CPUtime=12.54 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 28889 1259 0 0 1250 4 0 0 25 0 5 0 654596545 129548288 19414 33554432000 4194304 4417452 140735873305104 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.5
Current children cumulated vsize (KiB) 126512

[startup+25.5003 s]
/proc/loadavg: 7.21 7.72 6.36 9/281 2417
/proc/meminfo: memFree=20677048/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=199088 CPUtime=101.6 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 89755 1259 0 0 10147 13 0 0 24 0 5 0 654596540 203866112 37562 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 49772 37562 180 55 0 47658 0
[pid=2391/tid=2413] ppid=2389 vsize=199088 CPUtime=25.44 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 6996 1259 0 0 2544 0 0 0 25 0 5 0 654596545 203866112 37562 33554432000 4194304 4417452 140735873305104 18446744073709551615 4365118 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=199088 CPUtime=25.42 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 15097 1259 0 0 2539 3 0 0 25 0 5 0 654596545 203866112 37562 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=199088 CPUtime=25.43 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 23065 1259 0 0 2541 2 0 0 25 0 5 0 654596545 203866112 37562 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=199088 CPUtime=25.23 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 42108 1259 0 0 2518 5 0 0 25 0 5 0 654596545 203866112 37562 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.6
Current children cumulated vsize (KiB) 199088

[startup+51.1008 s]
/proc/loadavg: 7.63 7.78 6.41 9/281 2417
/proc/meminfo: memFree=20560084/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=262904 CPUtime=203.78 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 149033 1259 0 0 20357 21 0 0 24 0 5 0 654596540 269213696 53513 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 65726 53513 180 55 0 63612 0
[pid=2391/tid=2413] ppid=2389 vsize=262904 CPUtime=51.04 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 20041 1259 0 0 5102 2 0 0 25 0 5 0 654596545 269213696 53513 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=262904 CPUtime=51 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 31596 1259 0 0 5096 4 0 0 25 0 5 0 654596545 269213696 53513 33554432000 4194304 4417452 140735873305104 18446744073709551615 4258194 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=262904 CPUtime=51.03 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 42377 1259 0 0 5097 6 0 0 25 0 5 0 654596545 269213696 53513 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364780 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=262904 CPUtime=50.62 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 52530 1259 0 0 5055 7 0 0 25 0 5 0 654596545 269213696 53513 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.78
Current children cumulated vsize (KiB) 262904

[startup+102.311 s]
/proc/loadavg: 7.84 7.81 6.49 9/281 2419
/proc/meminfo: memFree=20348772/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=349156 CPUtime=408.2 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 247463 1259 0 0 40781 39 0 0 24 0 5 0 654596540 357535744 75076 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 87289 75076 180 55 0 85175 0
[pid=2391/tid=2413] ppid=2389 vsize=349156 CPUtime=102.25 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 36851 1259 0 0 10220 5 0 0 25 0 5 0 654596545 357535744 75076 33554432000 4194304 4417452 140735873305104 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=349156 CPUtime=102.21 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 51467 1259 0 0 10213 8 0 0 25 0 5 0 654596545 357535744 75076 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=349156 CPUtime=102.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 70309 1259 0 0 10213 12 0 0 25 0 5 0 654596545 357535744 75076 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=349156 CPUtime=101.41 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 86347 1259 0 0 10129 12 0 0 25 0 5 0 654596545 357535744 75076 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 408.2
Current children cumulated vsize (KiB) 349156

[startup+162.301 s]
/proc/loadavg: 8.08 7.87 6.60 9/281 2420
/proc/meminfo: memFree=20136704/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=404488 CPUtime=647.67 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 441750 1259 0 0 64698 69 0 0 24 0 5 0 654596540 414195712 88909 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 101122 88909 180 55 0 99008 0
[pid=2391/tid=2413] ppid=2389 vsize=404488 CPUtime=162.25 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 68454 1259 0 0 16213 12 0 0 25 0 5 0 654596545 414195712 88909 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364812 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=404488 CPUtime=162.2 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 95212 1259 0 0 16205 15 0 0 25 0 5 0 654596545 414195712 88909 33554432000 4194304 4417452 140735873305104 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=404488 CPUtime=162.23 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 161234 1259 0 0 16199 24 0 0 25 0 5 0 654596545 414195712 88909 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=404488 CPUtime=160.91 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 114361 1259 0 0 16075 16 0 0 25 0 5 0 654596545 414195712 88909 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364657 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 647.67
Current children cumulated vsize (KiB) 404488


################
# More data... #
################

[pid=2391/tid=2414] ppid=2389 vsize=1033776 CPUtime=702.14 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 437414 1259 0 0 70150 64 0 0 25 0 5 0 654596545 1058586624 246231 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1033776 CPUtime=702.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 768539 1259 0 0 70115 110 0 0 25 0 5 0 654596545 1058586624 246231 33554432000 4194304 4417452 140735873305104 18446744073709551615 4366105 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1033776 CPUtime=696.5 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 236075 1259 0 0 69614 36 0 0 25 0 5 0 654596545 1058586624 246231 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2803.21
Current children cumulated vsize (KiB) 1033776

[startup+762.31 s]
/proc/loadavg: 8.10 8.07 7.30 9/281 2436
/proc/meminfo: memFree=18718200/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1163476 CPUtime=3042.75 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 1849764 1259 0 0 304007 268 0 0 24 0 5 0 654596540 1191399424 278656 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 290869 278656 180 55 0 288755 0
[pid=2391/tid=2413] ppid=2389 vsize=1163476 CPUtime=762.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 188356 1259 0 0 76199 27 0 0 25 0 5 0 654596545 1191399424 278656 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1163476 CPUtime=762.15 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 470182 1259 0 0 76146 69 0 0 25 0 5 0 654596545 1191399424 278656 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1163476 CPUtime=762.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 936278 1259 0 0 76094 131 0 0 25 0 5 0 654596545 1191399424 278656 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1163476 CPUtime=756.01 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 252459 1259 0 0 75563 38 0 0 25 0 5 0 654596545 1191399424 278656 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3042.75
Current children cumulated vsize (KiB) 1163476

[startup+822.3 s]
/proc/loadavg: 8.07 8.07 7.34 9/281 2438
/proc/meminfo: memFree=18831636/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1212760 CPUtime=3282.22 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2128325 1259 0 0 327919 303 0 0 24 0 5 0 654596540 1241866240 290977 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 303190 290977 180 55 0 301076 0
[pid=2391/tid=2413] ppid=2389 vsize=1212760 CPUtime=822.25 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 202623 1259 0 0 82196 29 0 0 25 0 5 0 654596545 1241866240 290977 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1212760 CPUtime=822.12 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 547469 1259 0 0 82134 78 0 0 25 0 5 0 654596545 1241866240 290977 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1212760 CPUtime=822.24 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1108310 1259 0 0 82072 152 0 0 25 0 5 0 654596545 1241866240 290977 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1212760 CPUtime=815.52 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 267434 1259 0 0 81511 41 0 0 25 0 5 0 654596545 1241866240 290977 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3282.22
Current children cumulated vsize (KiB) 1212760

[startup+882.301 s]
/proc/loadavg: 8.17 8.10 7.40 9/281 2440
/proc/meminfo: memFree=19007332/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1106776 CPUtime=3521.73 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2175624 1259 0 0 351861 312 0 0 24 0 5 0 654596540 1133338624 264481 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 276694 264481 180 55 0 274580 0
[pid=2391/tid=2413] ppid=2389 vsize=1106776 CPUtime=882.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 206581 1259 0 0 88196 30 0 0 25 0 5 0 654596545 1133338624 264481 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1106776 CPUtime=882.12 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 560855 1259 0 0 88131 81 0 0 25 0 5 0 654596545 1133338624 264481 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1106776 CPUtime=882.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1120598 1259 0 0 88071 154 0 0 25 0 5 0 654596545 1133338624 264481 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1106776 CPUtime=875.02 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 285101 1259 0 0 87457 45 0 0 25 0 5 0 654596545 1133338624 264481 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3521.73
Current children cumulated vsize (KiB) 1106776

[startup+942.301 s]
/proc/loadavg: 8.21 8.11 7.44 9/281 2441
/proc/meminfo: memFree=18845244/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1151136 CPUtime=3761.24 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2186714 1259 0 0 375812 312 0 0 24 0 5 0 654596540 1178763264 275571 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 287784 275571 180 55 0 285670 0
[pid=2391/tid=2413] ppid=2389 vsize=1151136 CPUtime=942.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 206581 1259 0 0 94196 30 0 0 25 0 5 0 654596545 1178763264 275571 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1151136 CPUtime=942.11 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 560855 1259 0 0 94130 81 0 0 25 0 5 0 654596545 1178763264 275571 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1151136 CPUtime=942.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1120598 1259 0 0 94071 154 0 0 25 0 5 0 654596545 1178763264 275571 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364958 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1151136 CPUtime=934.54 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 296191 1259 0 0 93408 46 0 0 25 0 5 0 654596545 1178763264 275571 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3761.24
Current children cumulated vsize (KiB) 1151136

[startup+1002.3 s]
/proc/loadavg: 8.14 8.10 7.48 9/281 2443
/proc/meminfo: memFree=18746036/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1141552 CPUtime=4000.73 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2192510 1259 0 0 399760 313 0 0 24 0 5 0 654596540 1168949248 273175 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 285388 273175 180 55 0 283274 0
[pid=2391/tid=2413] ppid=2389 vsize=1141552 CPUtime=1002.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 206581 1259 0 0 100196 30 0 0 25 0 5 0 654596545 1168949248 273175 33554432000 4194304 4417452 140735873305104 18446744073709551615 4306366 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1141552 CPUtime=1002.1 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 560855 1259 0 0 100129 81 0 0 25 0 5 0 654596545 1168949248 273175 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1141552 CPUtime=1002.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1120598 1259 0 0 100071 154 0 0 25 0 5 0 654596545 1168949248 273175 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1141552 CPUtime=994.04 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 301987 1259 0 0 99358 46 0 0 25 0 5 0 654596545 1168949248 273175 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4000.73
Current children cumulated vsize (KiB) 1141552

[startup+1062.3 s]
/proc/loadavg: 8.12 8.10 7.51 9/281 2444
/proc/meminfo: memFree=18574532/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1335376 CPUtime=4240.24 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2273429 1259 0 0 423697 327 0 0 24 0 5 0 654596540 1367425024 321630 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 333844 321630 180 55 0 331730 0
[pid=2391/tid=2413] ppid=2389 vsize=1335376 CPUtime=1062.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 252917 1259 0 0 106187 39 0 0 25 0 5 0 654596545 1367425024 321630 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1335376 CPUtime=1062.09 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 587246 1259 0 0 106125 84 0 0 25 0 5 0 654596545 1367425024 321630 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1335376 CPUtime=1062.26 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1120598 1259 0 0 106071 155 0 0 25 0 5 0 654596545 1367425024 321630 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1335376 CPUtime=1053.55 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 310179 1259 0 0 105308 47 0 0 25 0 5 0 654596545 1367425024 321630 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4240.24
Current children cumulated vsize (KiB) 1335376

[startup+1122.3 s]
/proc/loadavg: 8.04 8.08 7.54 9/281 2446
/proc/meminfo: memFree=18303824/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1521820 CPUtime=4479.75 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2333614 1259 0 0 447638 337 0 0 24 0 5 0 654596540 1558343680 368241 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 380455 368241 180 55 0 378341 0
[pid=2391/tid=2413] ppid=2389 vsize=1521820 CPUtime=1122.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 252917 1259 0 0 112187 39 0 0 25 0 5 0 654596545 1558343680 368241 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1521820 CPUtime=1122.09 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 587246 1259 0 0 112125 84 0 0 25 0 5 0 654596545 1558343680 368241 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1521820 CPUtime=1122.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1159396 1259 0 0 112065 160 0 0 25 0 5 0 654596545 1558343680 368241 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1521820 CPUtime=1113.06 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 331566 1259 0 0 111255 51 0 0 25 0 5 0 654596545 1558343680 368241 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4479.75
Current children cumulated vsize (KiB) 1521820

[startup+1182.3 s]
/proc/loadavg: 8.21 8.11 7.58 9/281 2448
/proc/meminfo: memFree=18177204/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1512988 CPUtime=4719.26 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2339598 1259 0 0 471588 338 0 0 24 0 5 0 654596540 1549299712 366033 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 378247 366033 180 55 0 376133 0
[pid=2391/tid=2413] ppid=2389 vsize=1512988 CPUtime=1182.26 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 252917 1259 0 0 118187 39 0 0 25 0 5 0 654596545 1549299712 366033 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1512988 CPUtime=1182.08 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 587246 1259 0 0 118124 84 0 0 25 0 5 0 654596545 1549299712 366033 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1512988 CPUtime=1182.25 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1159396 1259 0 0 118065 160 0 0 25 0 5 0 654596545 1549299712 366033 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364780 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1512988 CPUtime=1172.57 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 337550 1259 0 0 117205 52 0 0 25 0 5 0 654596545 1549299712 366033 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4719.26
Current children cumulated vsize (KiB) 1512988



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.6 s]
/proc/loadavg: 8.15 8.10 7.59 9/281 2448
/proc/meminfo: memFree=18144088/32950904 swapFree=67057400/67111528
[pid=2391] ppid=2389 vsize=1545756 CPUtime=4800.3 cores=0,2,4,6
/proc/2391/stat : 2391 (plingeling) S 2389 2391 27712 0 -1 4202496 2347790 1259 0 0 479691 339 0 0 24 0 5 0 654596540 1582854144 374225 33554432000 4194304 4417452 140735873305104 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/2391/statm: 386439 374225 180 55 0 384325 0
[pid=2391/tid=2413] ppid=2389 vsize=1545756 CPUtime=1202.57 cores=0,2,4,6
/proc/2391/task/2413/stat : 2413 (plingeling) R 2389 2391 27712 0 -1 4202560 252917 1259 0 0 120218 39 0 0 25 0 5 0 654596545 1582854144 374225 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364558 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=2391/tid=2414] ppid=2389 vsize=1545756 CPUtime=1202.39 cores=0,2,4,6
/proc/2391/task/2414/stat : 2414 (plingeling) R 2389 2391 27712 0 -1 4202560 587246 1259 0 0 120155 84 0 0 25 0 5 0 654596545 1582854144 374225 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2391/tid=2415] ppid=2389 vsize=1545756 CPUtime=1202.56 cores=0,2,4,6
/proc/2391/task/2415/stat : 2415 (plingeling) R 2389 2391 27712 0 -1 4202560 1159396 1259 0 0 120096 160 0 0 25 0 5 0 654596545 1582854144 374225 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=2391/tid=2416] ppid=2389 vsize=1545756 CPUtime=1192.7 cores=0,2,4,6
/proc/2391/task/2416/stat : 2416 (plingeling) R 2389 2391 27712 0 -1 4202560 345742 1259 0 0 119217 53 0 0 25 0 5 0 654596545 1582854144 374225 33554432000 4194304 4417452 140735873305104 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 1545756

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.68
CPU time (s): 4800.41
CPU user time (s): 4796.95
CPU system time (s): 3.46147
CPU usage (%): 399.142
Max. virtual memory (cumulated for all children) (KiB): 1545756

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.95
system time used= 3.46147
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2349049
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 69
involuntary context switches= 22864

runsolver used 2.73758 second user time and 7.14991 second system time

The end

Launcher Data 

Begin job on node121 at 2011-05-15 05:26:08
IDJOB=3371032
IDBENCH=83986
IDSOLVER=1848
FILE ID=node121/3371032-1305429968
RUNJOBID= node121-1305429968-2376
PBS_JOBID= 13324576
Free space on /tmp= 73492 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_60_29_30_6-dir.shuffled-as.sat03-130.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3371032-1305429968/watcher-3371032-1305429968 -o /tmp/evaluation-result-3371032-1305429968/solver-3371032-1305429968 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3371032-1305429968.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e826117d87949a5b285e72732bfa395b
RANDOM SEED=1877196460

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      20972964 kB
Buffers:        304956 kB
Cached:        5950124 kB
SwapCached:       7656 kB
Active:        6396320 kB
Inactive:      5230492 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      20972964 kB
SwapTotal:    67111528 kB
SwapFree:     67057400 kB
Dirty:             884 kB
Writeback:           0 kB
AnonPages:     5368692 kB
Mapped:          21300 kB
Slab:           264024 kB
PageTables:      24404 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS: 30645776 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73492 MiB
End job on node121 at 2011-05-15 05:46:13