Trace number 3740981

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 273.68 392.582

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-47.opb
MD5SUM17efc5fe25abdce41d8265867254d297
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables10052
Total number of constraints226117
Number of constraints which are clauses203168
Number of constraints which are cardinality constraints (but not clauses)731
Number of constraints which are nor clauses,nor cardinality constraints22218
Minimum length of a constraint1
Maximum length of a constraint231
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 232
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.37/1.32	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.37/1.32	c INFO: solving HOME/instance-3740981-1338720608.opb
1.96/2.94	c INFO: parsing took 1.57 s
4.87/5.84	c INFO: features took 2.90 CPU seconds
4.87/5.87	c INFO: running wbo1.4b-fixed@0 for 45 with 1795 remaining (b = 0.96)
51.89/52.84	c INFO: running pbct-0.1.2-linear@0 for 45 with 1748 remaining (b = 0.96)
99.84/100.72	c INFO: running wbo1.4a@0 for 45 with 1701 remaining (b = 0.90)
163.78/164.60	c INFO: wrote minion input file to HOME/tmp4aaIkj.minion
163.78/164.60	c INFO: running minion-0.12@0 for 45 with 1652 remaining (b = 0.70)
173.04/295.33	c INFO: running clasp-1.3.7@0 for 45 with 1634 remaining (b = 0.57)
220.22/342.48	c INFO: running sat4j-pb-v20101225@0 for 45 with 1587 remaining (b = 0.45)
269.98/390.27	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1537 remaining (b = 0.29)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3740981-1338720608/watcher-3740981-1338720608 -o /tmp/evaluation-result-3740981-1338720608/solver-3740981-1338720608 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740981-1338720608.opb -seed 248426795 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.94 2.13 2.23 3/184 7906
/proc/meminfo: memFree=28869632/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/7906/stat : 7906 (python) D 7904 7906 3958 0 -1 4202496 672 0 0 0 0 0 0 0 20 0 1 0 338186894 15642624 559 33554432000 4194304 5683644 140733447682928 140733447650872 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 5 0 0 0 0 0
/proc/7906/statm: 3819 559 324 364 0 261 0

[startup+0.0853449 s]
/proc/loadavg: 1.94 2.13 2.23 3/184 7906
/proc/meminfo: memFree=28869632/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=19084 CPUtime=0.02 cores=1,3,5,7
/proc/7906/stat : 7906 (python) D 7904 7906 3958 0 -1 4202496 1356 0 0 0 1 1 0 0 20 0 1 0 338186894 19542016 1115 33554432000 4194304 5683644 140733447682928 140733447627160 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 5 0 0 0 0 0
/proc/7906/statm: 4771 1115 430 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19084

[startup+0.100257 s]
/proc/loadavg: 1.94 2.13 2.23 3/184 7906
/proc/meminfo: memFree=28869632/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=19084 CPUtime=0.04 cores=1,3,5,7
/proc/7906/stat : 7906 (python) R 7904 7906 3958 0 -1 4202496 1398 0 0 0 3 1 0 0 20 0 1 0 338186894 19542016 1152 33554432000 4194304 5683644 140733447682928 140733447628648 217401355559 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7906/statm: 5293 1156 432 364 0 695 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 19084

[startup+0.300241 s]
/proc/loadavg: 1.94 2.13 2.23 3/184 7906
/proc/meminfo: memFree=28869632/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=102080 CPUtime=0.12 cores=1,3,5,7
/proc/7906/stat : 7906 (python) R 7904 7906 3958 0 -1 4202496 4179 1011 1 0 9 3 0 0 20 0 1 0 338186894 104529920 2737 33554432000 4194304 5683644 140733447682928 140733447555528 4539660 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7906/statm: 25520 2737 876 364 0 1730 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 102080

[startup+0.700229 s]
/proc/loadavg: 1.94 2.13 2.23 3/184 7906
/proc/meminfo: memFree=28869632/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=166948 CPUtime=0.22 cores=1,3,5,7
/proc/7906/stat : 7906 (python) D 7904 7906 3958 0 -1 4202496 6543 1011 1 0 16 6 0 0 20 0 1 0 338186894 170954752 4336 33554432000 4194304 5683644 140733447682928 140733447587160 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 1 0 0 0 0 0
/proc/7906/statm: 41737 4336 1215 364 0 3290 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 166948

[startup+1.50015 s]
/proc/loadavg: 1.94 2.13 2.23 2/185 7911
/proc/meminfo: memFree=28851024/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=295496 CPUtime=0.57 cores=1,3,5,7
/proc/7906/stat : 7906 (python) R 7904 7906 3958 0 -1 4202496 12019 1011 1 0 48 9 0 0 20 0 1 0 338186894 302587904 10585 33554432000 4194304 5683644 140733447682928 140733447676168 4445115 0 0 16781312 2 0 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 73874 10585 1811 364 0 9103 0
Current children cumulated CPU time (s) 0.57
Current children cumulated vsize (KiB) 295496

[startup+3.10032 s]
/proc/loadavg: 2.03 2.15 2.23 3/185 7911
/proc/meminfo: memFree=28798280/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=341880 CPUtime=2.16 cores=1,3,5,7
/proc/7906/stat : 7906 (python) R 7904 7906 3958 0 -1 4202496 27905 1011 1 0 203 13 0 0 20 0 1 0 338186894 350085120 22221 33554432000 4194304 5683644 140733447682928 140733447673352 217410143466 0 0 16781312 33554434 0 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 85470 22221 1846 364 0 20699 0
Current children cumulated CPU time (s) 2.16
Current children cumulated vsize (KiB) 341880
heavy processes:

[startup+6.30013 s]
/proc/loadavg: 2.03 2.15 2.23 3/185 7911
/proc/meminfo: memFree=28784884/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=422984 CPUtime=4.94 cores=1,3,5,7
/proc/7906/stat : 7906 (python) S 7904 7906 3958 0 -1 4202496 29510 1011 1 0 480 14 0 0 20 0 2 0 338186894 433135616 23600 33554432000 4194304 5683644 140733447682928 140733447677760 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 105746 23600 1894 364 0 40969 0
Current children cumulated CPU time (s) 4.94
Current children cumulated vsize (KiB) 422984
heavy processes:

[startup+12.7002 s]
/proc/loadavg: 2.02 2.14 2.23 3/188 7914
/proc/meminfo: memFree=28746688/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=422984 CPUtime=4.94 cores=1,3,5,7
/proc/7906/stat : 7906 (python) S 7904 7906 3958 0 -1 4202496 29510 1011 1 0 480 14 0 0 20 0 2 0 338186894 433135616 23600 33554432000 4194304 5683644 140733447682928 140733447677760 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 105746 23600 1894 364 0 40969 0
[pid=7912] ppid=7906 vsize=347204 CPUtime=0 cores=1,3,5,7
/proc/7912/stat : 7912 (python) S 7906 7906 3958 0 -1 4202560 883 0 0 0 0 0 0 0 20 0 1 0 338187481 355536896 22063 33554432000 4194304 5683644 140733447682928 140733447674016 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7912/statm: 86801 22063 363 364 0 22024 0
[pid=7914] ppid=7912 vsize=31900 CPUtime=6.81 cores=1,3,5,7
/proc/7914/stat : 7914 (wbo1.4b-fixed) R 7912 7914 7914 0 -1 4202496 10155 0 0 0 656 25 0 0 20 0 1 0 338187482 32665600 7356 33554432000 134512640 135752315 4293691120 4293689196 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/7914/statm: 7975 7356 146 303 0 7670 0
[pid=7906/tid=7913] ppid=7904 vsize=422984 CPUtime=0 cores=1,3,5,7
/proc/7906/task/7913/stat : 7913 (python) S 7904 7906 3958 0 -1 4202560 26 1011 0 0 0 0 0 0 20 0 2 0 338187481 433135616 23600 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 11.75
Current children cumulated vsize (KiB) 802088

[startup+25.5003 s]
/proc/loadavg: 2.02 2.14 2.22 3/192 7918
/proc/meminfo: memFree=28717860/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=422984 CPUtime=4.94 cores=1,3,5,7
/proc/7906/stat : 7906 (python) S 7904 7906 3958 0 -1 4202496 29510 1011 1 0 480 14 0 0 20 0 2 0 338186894 433135616 23600 33554432000 4194304 5683644 140733447682928 140733447677760 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 105746 23600 1894 364 0 40969 0
[pid=7912] ppid=7906 vsize=347204 CPUtime=0.02 cores=1,3,5,7
/proc/7912/stat : 7912 (python) S 7906 7906 3958 0 -1 4202560 898 0 0 0 1 1 0 0 20 0 1 0 338187481 355536896 22063 33554432000 4194304 5683644 140733447682928 140733447674016 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7912/statm: 86801 22063 363 364 0 22024 0
[pid=7914] ppid=7912 vsize=40516 CPUtime=19.6 cores=1,3,5,7
/proc/7914/stat : 7914 (wbo1.4b-fixed) R 7912 7914 7914 0 -1 4202496 12275 0 0 0 1926 34 0 0 20 0 1 0 338187482 41488384 9476 33554432000 134512640 135752315 4293691120 4293689120 134611201 0 0 16781312 3 0 0 0 17 3 0 0 0 0 0
/proc/7914/statm: 10129 9476 146 303 0 9824 0
[pid=7906/tid=7913] ppid=7904 vsize=422984 CPUtime=0 cores=1,3,5,7
/proc/7906/task/7913/stat : 7913 (python) S 7904 7906 3958 0 -1 4202560 26 1011 0 0 0 0 0 0 20 0 2 0 338187481 433135616 23600 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 24.56
Current children cumulated vsize (KiB) 810704

[startup+51.1003 s]
/proc/loadavg: 2.01 2.12 2.22 3/192 7918
/proc/meminfo: memFree=28670996/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=422984 CPUtime=4.94 cores=1,3,5,7
/proc/7906/stat : 7906 (python) S 7904 7906 3958 0 -1 4202496 29510 1011 1 0 480 14 0 0 20 0 2 0 338186894 433135616 23600 33554432000 4194304 5683644 140733447682928 140733447677760 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 105746 23600 1894 364 0 40969 0
[pid=7912] ppid=7906 vsize=347204 CPUtime=0.06 cores=1,3,5,7
/proc/7912/stat : 7912 (python) S 7906 7906 3958 0 -1 4202560 1007 0 0 0 4 2 0 0 20 0 1 0 338187481 355536896 22068 33554432000 4194304 5683644 140733447682928 140733447674016 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7912/statm: 86801 22068 368 364 0 22024 0
[pid=7914] ppid=7912 vsize=63740 CPUtime=45.19 cores=1,3,5,7
/proc/7914/stat : 7914 (wbo1.4b-fixed) R 7912 7914 7914 0 -1 4202496 18015 0 0 0 4463 56 0 0 20 0 1 0 338187482 65269760 15216 33554432000 134512640 135752315 4293691120 4293689196 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/7914/statm: 15935 15216 146 303 0 15630 0
[pid=7906/tid=7913] ppid=7904 vsize=422984 CPUtime=0 cores=1,3,5,7
/proc/7906/task/7913/stat : 7913 (python) S 7904 7906 3958 0 -1 4202560 26 1011 0 0 0 0 0 0 20 0 2 0 338187481 433135616 23600 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 50.19
Current children cumulated vsize (KiB) 833928

[startup+102.301 s]
/proc/loadavg: 2.00 2.10 2.20 3/203 7931
/proc/meminfo: memFree=24057688/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=574568 CPUtime=4.99 cores=1,3,5,7
/proc/7906/stat : 7906 (python) S 7904 7906 3958 0 -1 4202496 31309 1011 1 0 485 14 0 0 20 0 4 0 338186894 588357632 23625 33554432000 4194304 5683644 140733447682928 140733447677760 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 143642 23625 1907 364 0 78859 0
[pid=7912] ppid=7906 vsize=422984 CPUtime=0.06 cores=1,3,5,7
/proc/7912/stat : 7912 (python) S 7906 7906 3958 0 -1 4202560 1138 0 0 0 4 2 0 0 20 0 2 0 338187481 433135616 22103 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7912/statm: 105746 22103 396 364 0 40969 0
[pid=7914] ppid=7912 vsize=65592 CPUtime=46.92 cores=1,3,5,7
/proc/7914/stat : 7914 (wbo1.4b-fixed) T 7912 7914 7914 0 -1 4202496 18478 0 0 0 4633 59 0 0 20 0 1 0 338187482 67166208 15679 33554432000 134512640 135752315 4293691120 4293689120 134611201 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7914/statm: 16398 15679 146 303 0 16093 0
[pid=7912/tid=7919] ppid=7906 vsize=422984 CPUtime=0 cores=1,3,5,7
/proc/7912/task/7919/stat : 7919 (python) S 7906 7906 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338192176 433135616 22103 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7920] ppid=7906 vsize=422996 CPUtime=0.07 cores=1,3,5,7
/proc/7920/stat : 7920 (python) S 7906 7906 3958 0 -1 4202560 1076 0 0 0 4 3 0 0 20 0 2 0 338192178 433147904 22090 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7920/statm: 105749 22090 383 364 0 40969 0
[pid=7922] ppid=7920 vsize=896492 CPUtime=47.82 cores=1,3,5,7
/proc/7922/stat : 7922 (pbct-0.1.2-linu) T 7920 7922 7922 0 -1 4202496 214979 0 0 0 4710 72 0 0 20 0 1 0 338192179 918007808 218454 33554432000 134512640 136422188 4288428000 4288425616 134862210 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7922/statm: 224123 218454 215 467 0 223654 0
[pid=7920/tid=7928] ppid=7906 vsize=422996 CPUtime=0 cores=1,3,5,7
/proc/7920/task/7928/stat : 7928 (python) S 7906 7906 3958 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338196964 433147904 22090 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7929] ppid=7906 vsize=498788 CPUtime=0 cores=1,3,5,7
/proc/7929/stat : 7929 (python) S 7906 7906 3958 0 -1 4202560 786 0 0 0 0 0 0 0 20 0 1 0 338196966 510758912 22047 33554432000 4194304 5683644 140733447682928 140733447674016 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7929/statm: 124697 22047 335 364 0 59914 0
[pid=7931] ppid=7929 vsize=22656 CPUtime=1.56 cores=1,3,5,7
/proc/7931/stat : 7931 (wbo1.4a) R 7929 7931 7931 0 -1 4202496 7847 0 0 0 149 7 0 0 20 0 1 0 338196966 23199744 5045 33554432000 134512640 135751879 4291666000 4291664252 135270609 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/7931/statm: 5664 5045 143 303 0 5359 0
[pid=7906/tid=7913] ppid=7904 vsize=574568 CPUtime=0 cores=1,3,5,7
/proc/7906/task/7913/stat : 7913 (python) S 7904 7906 3958 0 -1 4202560 26 1011 0 0 0 0 0 0 20 0 4 0 338187481 588357632 23625 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7906/tid=7921] ppid=7904 vsize=574568 CPUtime=0 cores=1,3,5,7
/proc/7906/task/7921/stat : 7921 (python) S 7904 7906 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 4 0 338192179 588357632 23625 33554432000 4194304 5683644 140733447682928 140687194423856 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7906/tid=7930] ppid=7904 vsize=574568 CPUtime=0 cores=1,3,5,7
/proc/7906/task/7930/stat : 7930 (python) S 7904 7906 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 4 0 338196966 588357632 23625 33554432000 4194304 5683644 140733447682928 140687183934000 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 101.42
Current children cumulated vsize (KiB) 2904076

[startup+162.301 s]
/proc/loadavg: 2.00 2.08 2.18 3/208 7936
/proc/meminfo: memFree=23934880/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=574572 CPUtime=16.61 cores=1,3,5,7
/proc/7906/stat : 7906 (python) R 7904 7906 3958 0 -1 4202496 31949 1011 1 0 1643 18 0 0 20 0 4 0 338186894 588361728 23626 33554432000 4194304 5683644 140733447682928 140733447677248 4530629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 14 0 0
/proc/7906/statm: 143643 23626 1907 364 0 78860 0
[pid=7912] ppid=7906 vsize=422984 CPUtime=0.06 cores=1,3,5,7
/proc/7912/stat : 7912 (python) S 7906 7906 3958 0 -1 4202560 1138 0 0 0 4 2 0 0 20 0 2 0 338187481 433135616 22103 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7912/statm: 105746 22103 396 364 0 40969 0
[pid=7914] ppid=7912 vsize=65592 CPUtime=46.92 cores=1,3,5,7
/proc/7914/stat : 7914 (wbo1.4b-fixed) T 7912 7914 7914 0 -1 4202496 18478 0 0 0 4633 59 0 0 20 0 1 0 338187482 67166208 15679 33554432000 134512640 135752315 4293691120 4293689120 134611201 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7914/statm: 16398 15679 146 303 0 16093 0
[pid=7912/tid=7919] ppid=7906 vsize=422984 CPUtime=0 cores=1,3,5,7
/proc/7912/task/7919/stat : 7919 (python) S 7906 7906 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338192176 433135616 22103 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7920] ppid=7906 vsize=422996 CPUtime=0.07 cores=1,3,5,7
/proc/7920/stat : 7920 (python) S 7906 7906 3958 0 -1 4202560 1076 0 0 0 4 3 0 0 20 0 2 0 338192178 433147904 22090 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7920/statm: 105749 22090 383 364 0 40969 0
[pid=7922] ppid=7920 vsize=896492 CPUtime=47.82 cores=1,3,5,7
/proc/7922/stat : 7922 (pbct-0.1.2-linu) T 7920 7922 7922 0 -1 4202496 214979 0 0 0 4710 72 0 0 20 0 1 0 338192179 918007808 218454 33554432000 134512640 136422188 4288428000 4288425616 134862210 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0

################
# More data... #
################

Current children cumulated CPU time (s) 220.11
Current children cumulated vsize (KiB) 3790420
heavy processes:
  %CPU=73 pid=8024 uid=0 cmd=/usr/libexec/abrt-hook-ccpp /var/spool/abrt 8023 11 506 18446744073709551615 



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+390.801 s]
/proc/loadavg: 4.40 3.34 2.67 10/246 8069
/proc/meminfo: memFree=9267296/32873648 swapFree=7628/7628
[pid=7906] ppid=7904 vsize=877736 CPUtime=28.29 cores=1,3,5,7
/proc/7906/stat : 7906 (python) S 7904 7906 3958 0 -1 4202496 34939 1039094 1 0 1881 19 449 480 20 0 8 0 338186894 898801664 24168 33554432000 4194304 5683644 140733447682928 140733447677760 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 7 0 0 14 0 0
/proc/7906/statm: 219434 24168 1915 364 0 154639 0
[pid=7912] ppid=7906 vsize=422984 CPUtime=0.06 cores=1,3,5,7
/proc/7912/stat : 7912 (python) S 7906 7906 3958 0 -1 4202560 1138 0 0 0 4 2 0 0 20 0 2 0 338187481 433135616 22103 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7912/statm: 105746 22103 396 364 0 40969 0
[pid=7914] ppid=7912 vsize=65592 CPUtime=46.92 cores=1,3,5,7
/proc/7914/stat : 7914 (wbo1.4b-fixed) T 7912 7914 7914 0 -1 4202496 18478 0 0 0 4633 59 0 0 20 0 1 0 338187482 67166208 15769 33554432000 134512640 135752315 4293691120 4293689120 134611201 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7914/statm: 16398 15769 146 303 0 16093 0
[pid=7912/tid=7919] ppid=7906 vsize=422984 CPUtime=0 cores=1,3,5,7
/proc/7912/task/7919/stat : 7919 (python) S 7906 7906 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338192176 433135616 22103 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7920] ppid=7906 vsize=422996 CPUtime=0.07 cores=1,3,5,7
/proc/7920/stat : 7920 (python) S 7906 7906 3958 0 -1 4202560 1076 0 0 0 4 3 0 0 20 0 2 0 338192178 433147904 22090 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7920/statm: 105749 22090 383 364 0 40969 0
[pid=7922] ppid=7920 vsize=896492 CPUtime=47.82 cores=1,3,5,7
/proc/7922/stat : 7922 (pbct-0.1.2-linu) T 7920 7922 7922 0 -1 4202496 214979 0 0 0 4710 72 0 0 20 0 1 0 338192179 918007808 218454 33554432000 134512640 136422188 4288428000 4288425616 134862210 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7922/statm: 224123 218454 215 467 0 223654 0
[pid=7920/tid=7928] ppid=7906 vsize=422996 CPUtime=0 cores=1,3,5,7
/proc/7920/task/7928/stat : 7928 (python) S 7906 7906 3958 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338196964 433147904 22090 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7929] ppid=7906 vsize=498788 CPUtime=0.07 cores=1,3,5,7
/proc/7929/stat : 7929 (python) S 7906 7906 3958 0 -1 4202560 1098 0 0 0 4 3 0 0 20 0 2 0 338196966 510758912 22096 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7929/statm: 124697 22096 383 364 0 59914 0
[pid=7931] ppid=7929 vsize=64212 CPUtime=49.93 cores=1,3,5,7
/proc/7931/stat : 7931 (wbo1.4a) T 7929 7931 7931 0 -1 4202496 18097 0 0 0 4928 65 0 0 20 0 1 0 338196966 65753088 15295 33554432000 134512640 135751879 4291666000 4291664176 134616382 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7931/statm: 16053 15295 143 303 0 15748 0
[pid=7929/tid=7936] ppid=7906 vsize=498788 CPUtime=0 cores=1,3,5,7
/proc/7929/task/7936/stat : 7936 (python) S 7906 7906 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338201961 510758912 22096 33554432000 4194304 5683644 140733447682928 140687194423856 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8017] ppid=7906 vsize=650372 CPUtime=0.07 cores=1,3,5,7
/proc/8017/stat : 8017 (python) S 7906 7906 3958 0 -1 4202560 1119 0 0 0 4 3 0 0 20 0 2 0 338216427 665980928 22108 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8017/statm: 162593 22108 383 364 0 97804 0
[pid=8019] ppid=8017 vsize=42964 CPUtime=47.1 cores=1,3,5,7
/proc/8019/stat : 8019 (clasp-1.3.7-x86) T 8017 8019 8019 0 -1 4202496 11467 0 0 0 4708 2 0 0 20 0 1 0 338216427 43995136 10274 33554432000 134512640 136139019 4287431056 4287429920 134608891 0 0 16781312 16386 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/8019/statm: 10741 10274 183 398 0 10341 0
[pid=8017/tid=8025] ppid=7906 vsize=650372 CPUtime=0 cores=1,3,5,7
/proc/8017/task/8025/stat : 8025 (python) S 7906 7906 3958 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338221139 665980928 22108 33554432000 4194304 5683644 140733447682928 140687173444144 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8026] ppid=7906 vsize=726164 CPUtime=0.07 cores=1,3,5,7
/proc/8026/stat : 8026 (python) S 7906 7906 3958 0 -1 4202560 1125 0 0 0 4 3 0 0 20 0 2 0 338221142 743591936 22114 33554432000 4194304 5683644 140733447682928 140733447674576 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8026/statm: 181541 22114 383 364 0 116749 0
[pid=8028] ppid=8026 vsize=9624972 CPUtime=49.66 cores=1,3,5,7
/proc/8028/stat : 8028 (java) T 8026 8028 8028 0 -1 4202496 13229 0 1 0 4917 49 0 0 20 0 19 0 338221142 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 140735616922592 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/8028/statm: 2406243 526546 2277 9 0 2396937 0
[pid=8028/tid=8029] ppid=8026 vsize=9624972 CPUtime=46.21 cores=1,3,5,7
/proc/8028/task/8029/stat : 8029 (java) T 8026 8028 8028 0 -1 4202560 3841 0 1 0 4591 30 0 0 20 0 19 0 338221143 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863695466752 139863547147078 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=8028/tid=8030] ppid=8026 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/8028/task/8030/stat : 8030 (java) T 8026 8028 8028 0 -1 4202560 31 0 0 0 10 0 0 0 20 0 19 0 338221144 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863667919400 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=8028/tid=8031] ppid=8026 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/8028/task/8031/stat : 8031 (java) T 8026 8028 8028 0 -1 4202560 67 0 0 0 10 3 0 0 20 0 19 0 338221144 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863666866600 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=8028/tid=8032] ppid=8026 vsize=9624972 CPUtime=0.16 cores=1,3,5,7
/proc/8028/task/8032/stat : 8032 (java) T 8026 8028 8028 0 -1 4202560 71 0 0 0 13 3 0 0 20 0 19 0 338221144 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863665813800 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=8028/tid=8033] ppid=8026 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/8028/task/8033/stat : 8033 (java) T 8026 8028 8028 0 -1 4202560 39 0 0 0 8 2 0 0 20 0 19 0 338221144 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863535429800 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=8028/tid=8034] ppid=8026 vsize=9624972 CPUtime=0.17 cores=1,3,5,7
/proc/8028/task/8034/stat : 8034 (java) T 8026 8028 8028 0 -1 4202560 61 0 0 0 14 3 0 0 20 0 19 0 338221144 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863534378024 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=8028/tid=8035] ppid=8026 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/8028/task/8035/stat : 8035 (java) T 8026 8028 8028 0 -1 4202560 34 0 0 0 8 0 0 0 20 0 19 0 338221145 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863533325224 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=8028/tid=8036] ppid=8026 vsize=9624972 CPUtime=0.16 cores=1,3,5,7
/proc/8028/task/8036/stat : 8036 (java) T 8026 8028 8028 0 -1 4202560 93 0 0 0 13 3 0 0 20 0 19 0 338221145 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863532272424 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=8028/tid=8037] ppid=8026 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/8028/task/8037/stat : 8037 (java) T 8026 8028 8028 0 -1 4202560 43 0 0 0 9 2 0 0 20 0 19 0 338221145 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863531219624 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=8028/tid=8038] ppid=8026 vsize=9624972 CPUtime=1.15 cores=1,3,5,7
/proc/8028/task/8038/stat : 8038 (java) T 8026 8028 8028 0 -1 4202560 1058 0 0 0 114 1 0 0 20 0 19 0 338221146 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863048706536 217413892009 0 0 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=8028/tid=8039] ppid=8026 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8028/task/8039/stat : 8039 (java) T 8026 8028 8028 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338221146 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863047652600 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=8028/tid=8040] ppid=8026 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8028/task/8040/stat : 8040 (java) T 8026 8028 8028 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338221146 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863046599688 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=8028/tid=8041] ppid=8026 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8028/task/8041/stat : 8041 (java) T 8026 8028 8028 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338221149 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863045548176 217413900064 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=8028/tid=8042] ppid=8026 vsize=9624972 CPUtime=0.5 cores=1,3,5,7
/proc/8028/task/8042/stat : 8042 (java) T 8026 8028 8028 0 -1 4202560 3029 0 0 0 50 0 0 0 20 0 19 0 338221149 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863044496312 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=8028/tid=8043] ppid=8026 vsize=9624972 CPUtime=0.6 cores=1,3,5,7
/proc/8028/task/8043/stat : 8043 (java) T 8026 8028 8028 0 -1 4202560 3244 0 0 0 60 0 0 0 20 0 19 0 338221149 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863043443512 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=8028/tid=8044] ppid=8026 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8028/task/8044/stat : 8044 (java) T 8026 8028 8028 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338221149 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863042390904 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=8028/tid=8045] ppid=8026 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/8028/task/8045/stat : 8045 (java) T 8026 8028 8028 0 -1 4202560 203 0 0 0 0 1 0 0 20 0 19 0 338221149 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863041338312 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=8028/tid=8046] ppid=8026 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8028/task/8046/stat : 8046 (java) T 8026 8028 8028 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338221311 9855971328 526546 33554432000 1073741824 1073778376 140735616931456 139863040283816 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=8026/tid=8048] ppid=7906 vsize=726164 CPUtime=0 cores=1,3,5,7
/proc/8026/task/8048/stat : 8048 (python) S 7906 7906 3958 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338225918 743591936 22114 33554432000 4194304 5683644 140733447682928 140687162954288 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8049] ppid=7906 vsize=801956 CPUtime=0 cores=1,3,5,7
/proc/8049/stat : 8049 (python) S 7906 7906 3958 0 -1 4202560 854 0 0 0 0 0 0 0 20 0 1 0 338225921 821202944 22071 33554432000 4194304 5683644 140733447682928 140733447674016 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8049/statm: 200489 22071 335 364 0 135694 0
[pid=8051] ppid=8049 vsize=9624972 CPUtime=3.62 cores=1,3,5,7
/proc/8051/stat : 8051 (java) S 8049 8051 8051 0 -1 4202496 12393 0 1 0 348 14 0 0 20 0 19 0 338225921 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140735751379792 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/8051/statm: 2406243 90090 2265 9 0 2396937 0
[pid=8051/tid=8052] ppid=8049 vsize=9624972 CPUtime=2.12 cores=1,3,5,7
/proc/8051/task/8052/stat : 8052 (java) S 8049 8051 8051 0 -1 4202560 2563 0 1 0 207 5 0 0 20 0 19 0 338225922 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170746821992 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8051/tid=8053] ppid=8049 vsize=9624972 CPUtime=0.04 cores=1,3,5,7
/proc/8051/task/8053/stat : 8053 (java) R 8049 8051 8051 0 -1 4202560 29 0 0 0 3 1 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170708314920 217410473095 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=8051/tid=8054] ppid=8049 vsize=9624972 CPUtime=0.03 cores=1,3,5,7
/proc/8051/task/8054/stat : 8054 (java) R 8049 8051 8051 0 -1 4202560 25 0 0 0 3 0 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170707262376 217410473095 0 4 0 16800975 0 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8055] ppid=8049 vsize=9624972 CPUtime=0.07 cores=1,3,5,7
/proc/8051/task/8055/stat : 8055 (java) R 8049 8051 8051 0 -1 4202560 44 0 0 0 5 2 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170706209880 140170753721411 0 4 0 16800975 0 0 0 -1 7 0 0 0 0 0
[pid=8051/tid=8056] ppid=8049 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/8051/task/8056/stat : 8056 (java) S 8049 8051 8051 0 -1 4202560 47 0 0 0 7 1 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170705157016 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8051/tid=8057] ppid=8049 vsize=9624972 CPUtime=0.03 cores=1,3,5,7
/proc/8051/task/8057/stat : 8057 (java) R 8049 8051 8051 0 -1 4202560 20 0 0 0 3 0 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170704103720 217410473095 0 4 0 16800975 0 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8058] ppid=8049 vsize=9624972 CPUtime=0.02 cores=1,3,5,7
/proc/8051/task/8058/stat : 8058 (java) R 8049 8051 8051 0 -1 4202560 10 0 0 0 2 0 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170703051176 217410473095 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=8051/tid=8059] ppid=8049 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/8051/task/8059/stat : 8059 (java) R 8049 8051 8051 0 -1 4202560 12 0 0 0 1 0 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170701998680 140170753721592 0 4 0 16800975 0 0 0 -1 7 0 0 0 0 0
[pid=8051/tid=8060] ppid=8049 vsize=9624972 CPUtime=0.02 cores=1,3,5,7
/proc/8051/task/8060/stat : 8060 (java) R 8049 8051 8051 0 -1 4202560 35 0 0 0 1 1 0 0 20 0 19 0 338225923 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170700946088 217410473095 0 4 0 16800975 0 0 0 -1 1 0 0 0 0 0
[pid=8051/tid=8061] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8061/stat : 8061 (java) S 8049 8051 8051 0 -1 4202560 200 0 0 0 0 0 0 0 20 0 19 0 338225924 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170688534024 217413891132 0 0 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8062] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8062/stat : 8062 (java) S 8049 8051 8051 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338225924 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170569442936 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8063] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8063/stat : 8063 (java) S 8049 8051 8051 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338225924 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170568390280 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8064] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8064/stat : 8064 (java) S 8049 8051 8051 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338225926 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170567339024 217413900064 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8065] ppid=8049 vsize=9624972 CPUtime=0.52 cores=1,3,5,7
/proc/8051/task/8065/stat : 8065 (java) S 8049 8051 8051 0 -1 4202560 4103 0 0 0 52 0 0 0 20 0 19 0 338225926 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170566285368 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8066] ppid=8049 vsize=9624972 CPUtime=0.5 cores=1,3,5,7
/proc/8051/task/8066/stat : 8066 (java) S 8049 8051 8051 0 -1 4202560 3886 0 0 0 50 0 0 0 20 0 19 0 338225926 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170565232824 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8051/tid=8067] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8067/stat : 8067 (java) S 8049 8051 8051 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338225926 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170564180472 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8051/tid=8068] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8068/stat : 8068 (java) S 8049 8051 8051 0 -1 4202560 5 0 0 0 0 0 0 0 20 0 19 0 338225926 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170563128136 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8051/tid=8069] ppid=8049 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/8051/task/8069/stat : 8069 (java) S 8049 8051 8051 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338226087 9855971328 90090 33554432000 1073741824 1073778376 140735751388656 140170562073896 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=7906/tid=7913] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/7913/stat : 7913 (python) S 7904 7906 3958 0 -1 4202560 26 1039094 0 0 0 0 449 480 20 0 8 0 338187481 898801664 24168 33554432000 4194304 5683644 140733447682928 140687204913712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7906/tid=7921] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/7921/stat : 7921 (python) S 7904 7906 3958 0 -1 4202560 21 1039094 0 0 0 0 449 480 20 0 8 0 338192179 898801664 24168 33554432000 4194304 5683644 140733447682928 140687194423856 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7906/tid=7930] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/7930/stat : 7930 (python) S 7904 7906 3958 0 -1 4202560 21 1039094 0 0 0 0 449 480 20 0 8 0 338196966 898801664 24168 33554432000 4194304 5683644 140733447682928 140687183934000 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7906/tid=7938] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/7938/stat : 7938 (python) S 7904 7906 3958 0 -1 4202560 22 1039094 0 0 0 0 449 480 20 0 8 0 338203354 898801664 24168 33554432000 4194304 5683644 140733447682928 140687173444144 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7906/tid=8018] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/8018/stat : 8018 (python) S 7904 7906 3958 0 -1 4202560 21 1039094 0 0 0 0 449 480 20 0 8 0 338216427 898801664 24168 33554432000 4194304 5683644 140733447682928 140687162954288 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7906/tid=8027] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/8027/stat : 8027 (python) S 7904 7906 3958 0 -1 4202560 21 1039094 0 0 0 0 449 480 20 0 8 0 338221142 898801664 24168 33554432000 4194304 5683644 140733447682928 140687139938864 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7906/tid=8050] ppid=7904 vsize=877736 CPUtime=9.29 cores=1,3,5,7
/proc/7906/task/8050/stat : 8050 (python) S 7904 7906 3958 0 -1 4202560 22 1039094 0 0 0 0 449 480 20 0 8 0 338225921 898801664 24168 33554432000 4194304 5683644 140733447682928 140687129449008 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 273.68
Current children cumulated vsize (KiB) 24720200
heavy processes:
  %CPU=24 pid=8024 uid=0 cmd=/usr/libexec/abrt-hook-ccpp /var/spool/abrt 8023 11 506 18446744073709551615 

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7906 and gives
#  childrusage.ru_utime.tv_sec=23
#  childrusage.ru_utime.tv_usec=304457
#  childrusage.ru_stime.tv_sec=5
#  childrusage.ru_stime.tv_usec=7238
# CPU time returned by wait4() is 28.3117
# while last known CPU time is 273.68
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 392.582
CPU time (s): 273.68
CPU user time (s): 265.94
CPU system time (s): 7.74
CPU usage (%): 69.7129
Max. virtual memory (cumulated for all children) (KiB): 24720200

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 23.3045
system time used= 5.00724
maximum resident set size= 4158704
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1074033
page faults= 1
swaps= 0
block input operations= 0
block output operations= 28584
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1023500
involuntary context switches= 76

runsolver used 1.20982 second user time and 3.2755 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-03 12:50:08
IDJOB=3740981
IDBENCH=89714
IDSOLVER=2327
FILE ID=node122/3740981-1338720608
RUNJOBID= node122-1338713392-3975
PBS_JOBID= 14636732
Free space on /tmp= 71068 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-47.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3740981-1338720608/watcher-3740981-1338720608 -o /tmp/evaluation-result-3740981-1338720608/solver-3740981-1338720608 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740981-1338720608.opb -seed 248426795 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 17efc5fe25abdce41d8265867254d297
RANDOM SEED=248426795

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        28869904 kB
Buffers:          281428 kB
Cached:          1957320 kB
SwapCached:         1804 kB
Active:          1756700 kB
Inactive:        1420820 kB
Active(anon):     939136 kB
Inactive(anon):     2024 kB
Active(file):     817564 kB
Inactive(file):  1418796 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67100164 kB
Dirty:            111096 kB
Writeback:            16 kB
AnonPages:        944764 kB
Mapped:            16512 kB
Shmem:                36 kB
Slab:             683128 kB
SReclaimable:     125980 kB
SUnreclaim:       557148 kB
KernelStack:        1520 kB
PageTables:         7684 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    1283552 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:     69632 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 63068 MiB
End job on node122 at 2012-06-03 12:56:44