Trace number 3740994

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 350.83 350.865

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-48.opb
MD5SUMbcbd6cca7644150bcdcbd185d4571ad6
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark155.412
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables9956
Total number of constraints224622
Number of constraints which are clauses201792
Number of constraints which are cardinality constraints (but not clauses)738
Number of constraints which are nor clauses,nor cardinality constraints22092
Minimum length of a constraint1
Maximum length of a constraint231
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 232
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.37/1.28	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.37/1.28	c INFO: solving HOME/instance-3740994-1338720628.opb
1.93/2.85	c INFO: parsing took 1.54 s
4.64/5.55	c INFO: features took 2.68 CPU seconds
4.64/5.57	c INFO: running wbo1.4b-fixed@0 for 45 with 1795 remaining (b = 0.96)
51.87/52.79	c INFO: running pbct-0.1.2-linear@0 for 45 with 1749 remaining (b = 0.96)
99.81/100.65	c INFO: running wbo1.4a@0 for 45 with 1701 remaining (b = 0.90)
160.16/160.92	c INFO: wrote minion input file to HOME/tmpBlflSn.minion
160.16/160.92	c INFO: running minion-0.12@0 for 45 with 1655 remaining (b = 0.70)
208.21/208.94	c INFO: running clasp-1.3.7@0 for 45 with 1595 remaining (b = 0.57)
255.48/256.10	c INFO: running bsolo_pb10-l2@0 for 45 with 1548 remaining (b = 0.46)
304.13/304.74	c INFO: running clasp-1.3.7@47 for 45 with 1500 remaining (b = 0.18)
349.30/349.90	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1455 remaining (b = 0.21)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3740994-1338720628/watcher-3740994-1338720628 -o /tmp/evaluation-result-3740994-1338720628/solver-3740994-1338720628 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740994-1338720628.opb -seed 1614251005 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.80 2.77 2.50 2/193 1035
/proc/meminfo: memFree=13171696/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=15016 CPUtime=0 cores=1,3,5,7
/proc/1035/stat : 1035 (python) R 1033 1035 27000 0 -1 4202496 559 0 0 0 0 0 0 0 20 0 1 0 338190346 15376384 459 33554432000 4194304 5683644 140736014534816 140736014521592 4540219 0 0 4096 0 0 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 3754 465 278 364 0 196 0

[startup+0.0707849 s]
/proc/loadavg: 3.80 2.77 2.50 2/193 1035
/proc/meminfo: memFree=13171696/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=16596 CPUtime=0.01 cores=1,3,5,7
/proc/1035/stat : 1035 (python) D 1033 1035 27000 0 -1 4202496 1241 0 0 0 1 0 0 0 20 0 1 0 338190346 16994304 1031 33554432000 4194304 5683644 140736014534816 140736014498840 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 4149 1031 422 364 0 591 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16596

[startup+0.100312 s]
/proc/loadavg: 3.80 2.77 2.50 2/193 1035
/proc/meminfo: memFree=13171696/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=19088 CPUtime=0.02 cores=1,3,5,7
/proc/1035/stat : 1035 (python) R 1033 1035 27000 0 -1 4202496 1394 0 0 0 2 0 0 0 20 0 1 0 338190346 19546112 1149 33554432000 4194304 5683644 140736014534816 140736014465320 219513589652 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 4772 1149 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.300316 s]
/proc/loadavg: 3.80 2.77 2.50 2/193 1035
/proc/meminfo: memFree=13171696/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=96896 CPUtime=0.1 cores=1,3,5,7
/proc/1035/stat : 1035 (python) D 1033 1035 27000 0 -1 4202496 3912 1010 1 0 8 2 0 0 20 0 1 0 338190346 99221504 2647 33554432000 4194304 5683644 140736014534816 140736014434808 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 24224 2647 822 364 0 1715 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 96896

[startup+0.700206 s]
/proc/loadavg: 3.80 2.77 2.50 2/193 1035
/proc/meminfo: memFree=13171696/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=122480 CPUtime=0.2 cores=1,3,5,7
/proc/1035/stat : 1035 (python) D 1033 1035 27000 0 -1 4202496 6051 1010 1 0 15 5 0 0 20 0 1 0 338190346 125419520 3914 33554432000 4194304 5683644 140736014534816 140736014452776 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 30620 3914 1015 364 0 2863 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 122480

[startup+1.50024 s]
/proc/loadavg: 3.80 2.77 2.50 1/194 1040
/proc/meminfo: memFree=13154712/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=296984 CPUtime=0.63 cores=1,3,5,7
/proc/1035/stat : 1035 (python) R 1033 1035 27000 0 -1 4202496 11848 1010 1 0 53 10 0 0 20 0 1 0 338190346 304111616 10926 33554432000 4194304 5683644 140736014534816 140736014448888 139903369802719 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 74246 10926 1811 364 0 9475 0
Current children cumulated CPU time (s) 0.63
Current children cumulated vsize (KiB) 296984

[startup+3.10023 s]
/proc/loadavg: 3.80 2.77 2.50 2/194 1040
/proc/meminfo: memFree=13113420/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=341236 CPUtime=2.22 cores=1,3,5,7
/proc/1035/stat : 1035 (python) R 1033 1035 27000 0 -1 4202496 26646 1010 1 0 209 13 0 0 20 0 1 0 338190346 349425664 22060 33554432000 4194304 5683644 140736014534816 140736014527432 219521509173 0 0 16781312 33554434 0 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 85309 22060 1846 364 0 20538 0
Current children cumulated CPU time (s) 2.22
Current children cumulated vsize (KiB) 341236
heavy processes:
  %CPU=25 pid=969 uid=0 cmd=

[startup+6.30025 s]
/proc/loadavg: 3.90 2.81 2.52 2/194 1040
/proc/meminfo: memFree=13086704/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=422304 CPUtime=4.7 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 28246 1010 1 0 456 14 0 0 20 0 2 0 338190346 432439296 23438 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 105576 23438 1902 364 0 40799 0
Current children cumulated CPU time (s) 4.7
Current children cumulated vsize (KiB) 422304
heavy processes:
  %CPU=23 pid=969 uid=0 cmd=/usr/libexec/abrt-hook-ccpp /var/spool/abrt 968 11 506 18446744073709551615 

[startup+12.7003 s]
/proc/loadavg: 3.99 2.85 2.53 2/197 1043
/proc/meminfo: memFree=13056080/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=422304 CPUtime=4.7 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 28246 1010 1 0 456 14 0 0 20 0 2 0 338190346 432439296 23438 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 105576 23438 1902 364 0 40799 0
[pid=1041] ppid=1035 vsize=346524 CPUtime=0 cores=1,3,5,7
/proc/1041/stat : 1041 (python) S 1035 1035 27000 0 -1 4202560 886 0 0 0 0 0 0 0 20 0 1 0 338190904 354840576 21893 33554432000 4194304 5683644 140736014534816 140736014525904 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1041/statm: 86631 21893 363 364 0 21854 0
[pid=1043] ppid=1041 vsize=29508 CPUtime=7.11 cores=1,3,5,7
/proc/1043/stat : 1043 (wbo1.4b-fixed) R 1041 1043 1043 0 -1 4202496 9538 0 0 0 681 30 0 0 20 0 1 0 338190904 30216192 6761 33554432000 134512640 135752315 4289171248 4289169324 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1043/statm: 7377 6761 146 303 0 7072 0
[pid=1035/tid=1042] ppid=1033 vsize=422304 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1042/stat : 1042 (python) S 1033 1035 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338190904 432439296 23438 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.81
Current children cumulated vsize (KiB) 798336

[startup+25.5004 s]
/proc/loadavg: 4.21 2.95 2.57 2/197 1043
/proc/meminfo: memFree=13041228/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=422304 CPUtime=4.7 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 28246 1010 1 0 456 14 0 0 20 0 2 0 338190346 432439296 23438 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 105576 23438 1902 364 0 40799 0
[pid=1041] ppid=1035 vsize=346524 CPUtime=0.03 cores=1,3,5,7
/proc/1041/stat : 1041 (python) S 1035 1035 27000 0 -1 4202560 898 0 0 0 2 1 0 0 20 0 1 0 338190904 354840576 21893 33554432000 4194304 5683644 140736014534816 140736014525904 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1041/statm: 86631 21893 363 364 0 21854 0
[pid=1043] ppid=1041 vsize=41156 CPUtime=19.9 cores=1,3,5,7
/proc/1043/stat : 1043 (wbo1.4b-fixed) R 1041 1043 1043 0 -1 4202496 12420 0 0 0 1929 61 0 0 20 0 1 0 338190904 42143744 9643 33554432000 134512640 135752315 4289171248 4289169324 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1043/statm: 10289 9643 146 303 0 9984 0
[pid=1035/tid=1042] ppid=1033 vsize=422304 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1042/stat : 1042 (python) S 1033 1035 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338190904 432439296 23438 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.63
Current children cumulated vsize (KiB) 809984

[startup+51.1004 s]
/proc/loadavg: 4.48 3.11 2.63 2/196 1043
/proc/meminfo: memFree=13030572/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=422304 CPUtime=4.7 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 28246 1010 1 0 456 14 0 0 20 0 2 0 338190346 432439296 23438 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1035/statm: 105576 23438 1902 364 0 40799 0
[pid=1041] ppid=1035 vsize=346524 CPUtime=0.07 cores=1,3,5,7
/proc/1041/stat : 1041 (python) S 1035 1035 27000 0 -1 4202560 1001 0 0 0 4 3 0 0 20 0 1 0 338190904 354840576 21898 33554432000 4194304 5683644 140736014534816 140736014525904 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1041/statm: 86631 21898 368 364 0 21854 0
[pid=1043] ppid=1041 vsize=55196 CPUtime=45.5 cores=1,3,5,7
/proc/1043/stat : 1043 (wbo1.4b-fixed) R 1041 1043 1043 0 -1 4202496 15905 0 0 0 4471 79 0 0 20 0 1 0 338190904 56520704 13128 33554432000 134512640 135752315 4289171248 4289169324 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1043/statm: 13799 13128 146 303 0 13494 0
[pid=1035/tid=1042] ppid=1033 vsize=422304 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1042/stat : 1042 (python) S 1033 1035 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338190904 432439296 23438 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.27
Current children cumulated vsize (KiB) 824024

[startup+102.307 s]
/proc/loadavg: 4.07 3.24 2.70 3/205 1059
/proc/meminfo: memFree=20554712/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=573888 CPUtime=4.76 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 30061 1010 1 0 461 15 0 0 20 0 4 0 338190346 587661312 23455 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1035/statm: 143472 23455 1907 364 0 78689 0
[pid=1041] ppid=1035 vsize=422304 CPUtime=0.07 cores=1,3,5,7
/proc/1041/stat : 1041 (python) S 1035 1035 27000 0 -1 4202560 1134 0 0 0 4 3 0 0 20 0 2 0 338190904 432439296 21933 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1041/statm: 105576 21933 396 364 0 40799 0
[pid=1043] ppid=1041 vsize=56284 CPUtime=47.16 cores=1,3,5,7
/proc/1043/stat : 1043 (wbo1.4b-fixed) T 1041 1043 1043 0 -1 4202496 16185 0 0 0 4636 80 0 0 20 0 1 0 338190904 57634816 13408 33554432000 134512640 135752315 4289171248 4289169324 135270849 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1043/statm: 14071 13408 146 303 0 13766 0
[pid=1041/tid=1044] ppid=1035 vsize=422304 CPUtime=0 cores=1,3,5,7
/proc/1041/task/1044/stat : 1044 (python) S 1035 1035 27000 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 338195623 432439296 21933 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1045] ppid=1035 vsize=422316 CPUtime=0.07 cores=1,3,5,7
/proc/1045/stat : 1045 (python) S 1035 1035 27000 0 -1 4202560 1060 0 0 0 4 3 0 0 20 0 2 0 338195625 432451584 21920 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1045/statm: 105579 21920 383 364 0 40799 0
[pid=1047] ppid=1045 vsize=888604 CPUtime=47.81 cores=1,3,5,7
/proc/1047/stat : 1047 (pbct-0.1.2-linu) T 1045 1047 1047 0 -1 4202496 213432 0 0 0 4712 69 0 0 20 0 1 0 338195626 909930496 215926 33554432000 134512640 136422188 4293516864 4293514288 134620377 0 0 16781312 16387 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/1047/statm: 222151 215926 215 467 0 221682 0
[pid=1045/tid=1054] ppid=1035 vsize=422316 CPUtime=0 cores=1,3,5,7
/proc/1045/task/1054/stat : 1054 (python) S 1035 1035 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338200409 432451584 21920 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1055] ppid=1035 vsize=498108 CPUtime=0 cores=1,3,5,7
/proc/1055/stat : 1055 (python) S 1035 1035 27000 0 -1 4202560 786 0 0 0 0 0 0 0 20 0 1 0 338200412 510062592 21877 33554432000 4194304 5683644 140736014534816 140736014525904 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/1055/statm: 124527 21877 335 364 0 59744 0
[pid=1059] ppid=1055 vsize=22468 CPUtime=1.63 cores=1,3,5,7
/proc/1059/stat : 1059 (wbo1.4a) R 1055 1059 1059 0 -1 4202496 7780 0 0 0 154 9 0 0 20 0 1 0 338200412 23007232 5002 33554432000 134512640 135751879 4294577920 4294576172 135270609 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1059/statm: 5617 5002 143 303 0 5312 0
[pid=1035/tid=1042] ppid=1033 vsize=573888 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1042/stat : 1042 (python) S 1033 1035 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 4 0 338190904 587661312 23455 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1046] ppid=1033 vsize=573888 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1046/stat : 1046 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 4 0 338195626 587661312 23455 33554432000 4194304 5683644 140736014534816 139903352676912 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1058] ppid=1033 vsize=573888 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1058/stat : 1058 (python) S 1033 1035 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 4 0 338200412 587661312 23455 33554432000 4194304 5683644 140736014534816 139903342187056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 101.5
Current children cumulated vsize (KiB) 2883972

[startup+162.301 s]
/proc/loadavg: 2.89 3.06 2.67 3/212 1067
/proc/meminfo: memFree=20408140/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=649680 CPUtime=18.18 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 31034 1010 1 0 1798 20 0 0 20 0 5 0 338190346 665272320 23463 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1035/statm: 162420 23463 1909 364 0 97634 0
[pid=1041] ppid=1035 vsize=422304 CPUtime=0.07 cores=1,3,5,7
/proc/1041/stat : 1041 (python) S 1035 1035 27000 0 -1 4202560 1134 0 0 0 4 3 0 0 20 0 2 0 338190904 432439296 21933 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1041/statm: 105576 21933 396 364 0 40799 0
[pid=1043] ppid=1041 vsize=56284 CPUtime=47.16 cores=1,3,5,7
/proc/1043/stat : 1043 (wbo1.4b-fixed) T 1041 1043 1043 0 -1 4202496 16185 0 0 0 4636 80 0 0 20 0 1 0 338190904 57634816 13408 33554432000 134512640 135752315 4289171248 4289169324 135270849 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1043/statm: 14071 13408 146 303 0 13766 0
[pid=1041/tid=1044] ppid=1035 vsize=422304 CPUtime=0 cores=1,3,5,7
/proc/1041/task/1044/stat : 1044 (python) S 1035 1035 27000 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 338195623 432439296 21933 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1045] ppid=1035 vsize=422316 CPUtime=0.07 cores=1,3,5,7
/proc/1045/stat : 1045 (python) S 1035 1035 27000 0 -1 4202560 1060 0 0 0 4 3 0 0 20 0 2 0 338195625 432451584 21920 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1045/statm: 105579 21920 383 364 0 40799 0

################
# More data... #
################

/proc/1065/statm: 143475 21932 383 364 0 78689 0
[pid=1067] ppid=1065 vsize=3783092 CPUtime=47.97 cores=1,3,5,7
/proc/1067/stat : 1067 (minion) T 1065 1067 1067 0 -1 4202496 935980 0 0 0 4668 129 0 0 20 0 1 0 338206439 3873886208 938394 33554432000 134512640 165565024 4292766480 4292763712 159737071 0 0 16781312 8396802 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1067/statm: 945773 938394 358 7582 0 938188 0
[pid=1065/tid=1069] ppid=1035 vsize=573900 CPUtime=0 cores=1,3,5,7
/proc/1065/task/1069/stat : 1069 (python) S 1035 1035 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338211238 587673600 21932 33554432000 4194304 5683644 140736014534816 139903342187056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1070] ppid=1035 vsize=649692 CPUtime=0.15 cores=1,3,5,7
/proc/1070/stat : 1070 (python) S 1035 1035 27000 0 -1 4202560 1142 0 0 0 9 6 0 0 20 0 2 0 338211240 665284608 21938 33554432000 4194304 5683644 140736014534816 140736014525904 219513979363 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1070/statm: 162423 21938 383 364 0 97634 0
[pid=1072] ppid=1070 vsize=46016 CPUtime=84.67 cores=1,3,5,7
/proc/1072/stat : 1072 (clasp-1.3.7-x86) R 1070 1072 1072 0 -1 4202496 12142 0 0 0 8464 3 0 0 20 0 1 0 338211240 47120384 10965 33554432000 134512640 136139019 4288882576 4288881552 134883787 0 0 16781312 16386 0 0 0 17 1 0 0 0 0 0
/proc/1072/statm: 11504 10965 183 398 0 11104 0
[pid=1070/tid=1094] ppid=1035 vsize=649692 CPUtime=0 cores=1,3,5,7
/proc/1070/task/1094/stat : 1094 (python) S 1035 1035 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338215954 665284608 21938 33554432000 4194304 5683644 140736014534816 139903331697200 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1095] ppid=1035 vsize=725484 CPUtime=0.08 cores=1,3,5,7
/proc/1095/stat : 1095 (python) S 1035 1035 27000 0 -1 4202560 1107 0 0 0 4 4 0 0 20 0 2 0 338215957 742895616 21944 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1095/statm: 181371 21944 383 364 0 116579 0
[pid=1097] ppid=1095 vsize=83448 CPUtime=48.59 cores=1,3,5,7
/proc/1097/stat : 1097 (bsolo_pb10) T 1095 1097 1097 0 -1 4202496 21234 0 0 0 4799 60 0 0 20 0 1 0 338215957 85450752 18393 33554432000 134512640 137138936 4287984640 4287983568 134542199 0 0 16781312 16384 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/1097/statm: 20862 18393 192 642 0 20218 0
[pid=1095/tid=1167] ppid=1035 vsize=725484 CPUtime=0 cores=1,3,5,7
/proc/1095/task/1167/stat : 1167 (python) S 1035 1035 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338220818 742895616 21944 33554432000 4194304 5683644 140736014534816 139903321207344 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1042] ppid=1033 vsize=801264 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1042/stat : 1042 (python) S 1033 1035 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 7 0 338190904 820494336 23479 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1046] ppid=1033 vsize=801264 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1046/stat : 1046 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 7 0 338195626 820494336 23479 33554432000 4194304 5683644 140736014534816 139903352676912 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1058] ppid=1033 vsize=801264 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1058/stat : 1058 (python) S 1033 1035 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 7 0 338200412 820494336 23479 33554432000 4194304 5683644 140736014534816 139903342187056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=1035/tid=1066] ppid=1033 vsize=801264 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1066/stat : 1066 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 7 0 338206439 820494336 23479 33554432000 4194304 5683644 140736014534816 139903331697200 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1035/tid=1071] ppid=1033 vsize=801264 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1071/stat : 1071 (python) S 1033 1035 27000 0 -1 4202560 24 1010 0 0 0 0 0 0 20 0 7 0 338211240 820494336 23479 33554432000 4194304 5683644 140736014534816 139903321207344 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1035/tid=1096] ppid=1033 vsize=801264 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1096/stat : 1096 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 7 0 338215957 820494336 23479 33554432000 4194304 5683644 140736014534816 139903298421296 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 341.8
Current children cumulated vsize (KiB) 9002804



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+350.801 s]
/proc/loadavg: 2.12 2.60 2.56 7/228 1191
/proc/meminfo: memFree=19242564/32873648 swapFree=7120/7120
[pid=1035] ppid=1033 vsize=877056 CPUtime=18.29 cores=1,3,5,7
/proc/1035/stat : 1035 (python) S 1033 1035 27000 0 -1 4202496 33895 1010 1 0 1808 21 0 0 20 0 8 0 338190346 898105344 23487 33554432000 4194304 5683644 140736014534816 140736014529648 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/1035/statm: 219264 23487 1915 364 0 154469 0
[pid=1041] ppid=1035 vsize=422304 CPUtime=0.07 cores=1,3,5,7
/proc/1041/stat : 1041 (python) S 1035 1035 27000 0 -1 4202560 1134 0 0 0 4 3 0 0 20 0 2 0 338190904 432439296 21933 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1041/statm: 105576 21933 396 364 0 40799 0
[pid=1043] ppid=1041 vsize=56284 CPUtime=47.16 cores=1,3,5,7
/proc/1043/stat : 1043 (wbo1.4b-fixed) T 1041 1043 1043 0 -1 4202496 16185 0 0 0 4636 80 0 0 20 0 1 0 338190904 57634816 13408 33554432000 134512640 135752315 4289171248 4289169324 135270849 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1043/statm: 14071 13408 146 303 0 13766 0
[pid=1041/tid=1044] ppid=1035 vsize=422304 CPUtime=0 cores=1,3,5,7
/proc/1041/task/1044/stat : 1044 (python) S 1035 1035 27000 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 338195623 432439296 21933 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1045] ppid=1035 vsize=422316 CPUtime=0.07 cores=1,3,5,7
/proc/1045/stat : 1045 (python) S 1035 1035 27000 0 -1 4202560 1060 0 0 0 4 3 0 0 20 0 2 0 338195625 432451584 21920 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1045/statm: 105579 21920 383 364 0 40799 0
[pid=1047] ppid=1045 vsize=888604 CPUtime=47.81 cores=1,3,5,7
/proc/1047/stat : 1047 (pbct-0.1.2-linu) T 1045 1047 1047 0 -1 4202496 213432 0 0 0 4712 69 0 0 20 0 1 0 338195626 909930496 215926 33554432000 134512640 136422188 4293516864 4293514288 134620377 0 0 16781312 16387 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/1047/statm: 222151 215926 215 467 0 221682 0
[pid=1045/tid=1054] ppid=1035 vsize=422316 CPUtime=0 cores=1,3,5,7
/proc/1045/task/1054/stat : 1054 (python) S 1035 1035 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338200409 432451584 21920 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1055] ppid=1035 vsize=498108 CPUtime=0.07 cores=1,3,5,7
/proc/1055/stat : 1055 (python) S 1035 1035 27000 0 -1 4202560 1093 0 0 0 3 4 0 0 20 0 2 0 338200412 510062592 21926 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1055/statm: 124527 21926 383 364 0 59744 0
[pid=1059] ppid=1055 vsize=52292 CPUtime=46.82 cores=1,3,5,7
/proc/1059/stat : 1059 (wbo1.4a) T 1055 1059 1059 0 -1 4202496 15187 0 0 0 4602 80 0 0 20 0 1 0 338200412 53547008 12409 33554432000 134512640 135751879 4294577920 4294576096 134616876 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1059/statm: 13073 12409 143 303 0 12768 0
[pid=1055/tid=1064] ppid=1035 vsize=498108 CPUtime=0 cores=1,3,5,7
/proc/1055/task/1064/stat : 1064 (python) S 1035 1035 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338205096 510062592 21926 33554432000 4194304 5683644 140736014534816 139903352676912 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1065] ppid=1035 vsize=573900 CPUtime=0.08 cores=1,3,5,7
/proc/1065/stat : 1065 (python) S 1035 1035 27000 0 -1 4202560 1089 0 0 0 5 3 0 0 20 0 2 0 338206438 587673600 21932 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1065/statm: 143475 21932 383 364 0 78689 0
[pid=1067] ppid=1065 vsize=3783092 CPUtime=47.97 cores=1,3,5,7
/proc/1067/stat : 1067 (minion) T 1065 1067 1067 0 -1 4202496 935980 0 0 0 4668 129 0 0 20 0 1 0 338206439 3873886208 938394 33554432000 134512640 165565024 4292766480 4292763712 159737071 0 0 16781312 8396802 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1067/statm: 945773 938394 358 7582 0 938188 0
[pid=1065/tid=1069] ppid=1035 vsize=573900 CPUtime=0 cores=1,3,5,7
/proc/1065/task/1069/stat : 1069 (python) S 1035 1035 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338211238 587673600 21932 33554432000 4194304 5683644 140736014534816 139903342187056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1070] ppid=1035 vsize=649692 CPUtime=0.17 cores=1,3,5,7
/proc/1070/stat : 1070 (python) S 1035 1035 27000 0 -1 4202560 1143 0 0 0 10 7 0 0 20 0 2 0 338211240 665284608 21938 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1070/statm: 162423 21938 383 364 0 97634 0
[pid=1072] ppid=1070 vsize=46720 CPUtime=92.24 cores=1,3,5,7
/proc/1072/stat : 1072 (clasp-1.3.7-x86) T 1070 1072 1072 0 -1 4202496 12303 0 0 0 9221 3 0 0 20 0 1 0 338211240 47841280 11126 33554432000 134512640 136139019 4288882576 4288881440 134608891 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1072/statm: 11680 11126 183 398 0 11280 0
[pid=1070/tid=1094] ppid=1035 vsize=649692 CPUtime=0 cores=1,3,5,7
/proc/1070/task/1094/stat : 1094 (python) S 1035 1035 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338215954 665284608 21938 33554432000 4194304 5683644 140736014534816 139903331697200 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1095] ppid=1035 vsize=725484 CPUtime=0.08 cores=1,3,5,7
/proc/1095/stat : 1095 (python) S 1035 1035 27000 0 -1 4202560 1107 0 0 0 4 4 0 0 20 0 2 0 338215957 742895616 21944 33554432000 4194304 5683644 140736014534816 140736014526464 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1095/statm: 181371 21944 383 364 0 116579 0
[pid=1097] ppid=1095 vsize=83448 CPUtime=48.59 cores=1,3,5,7
/proc/1097/stat : 1097 (bsolo_pb10) T 1095 1097 1097 0 -1 4202496 21234 0 0 0 4799 60 0 0 20 0 1 0 338215957 85450752 18393 33554432000 134512640 137138936 4287984640 4287983568 134542199 0 0 16781312 16384 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/1097/statm: 20862 18393 192 642 0 20218 0
[pid=1095/tid=1167] ppid=1035 vsize=725484 CPUtime=0 cores=1,3,5,7
/proc/1095/task/1167/stat : 1167 (python) S 1035 1035 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338220818 742895616 21944 33554432000 4194304 5683644 140736014534816 139903321207344 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1172] ppid=1035 vsize=801276 CPUtime=0 cores=1,3,5,7
/proc/1172/stat : 1172 (python) S 1035 1035 27000 0 -1 4202560 842 0 0 0 0 0 0 0 20 0 1 0 338225336 820506624 21901 33554432000 4194304 5683644 140736014534816 140736014525904 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 7 0 0 0 0 0
/proc/1172/statm: 200319 21901 335 364 0 135524 0
[pid=1174] ppid=1172 vsize=9558408 CPUtime=1.41 cores=1,3,5,7
/proc/1174/stat : 1174 (java) S 1172 1174 1174 0 -1 4202496 8300 0 1 0 137 4 0 0 20 0 18 0 338225336 9787809792 43459 33554432000 1073741824 1073778376 140734777165152 140734777156288 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/1174/statm: 2389602 43459 2247 9 0 2380296 0
[pid=1174/tid=1175] ppid=1172 vsize=9558408 CPUtime=0.84 cores=1,3,5,7
/proc/1174/task/1175/stat : 1175 (java) S 1172 1174 1174 0 -1 4202560 2487 0 1 0 81 3 0 0 20 0 18 0 338225338 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753462767176 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1174/tid=1176] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1176/stat : 1176 (java) R 1172 1174 1174 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753424259544 139753465239233 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=1174/tid=1177] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1177/stat : 1177 (java) R 1172 1174 1174 0 -1 4202560 5 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753423206568 139753469666352 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=1174/tid=1178] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1178/stat : 1178 (java) S 1172 1174 1174 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753422154280 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1174/tid=1179] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1179/stat : 1179 (java) R 1172 1174 1174 0 -1 4202560 11 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753421101480 219517334588 0 4 0 16800975 0 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1180] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1180/stat : 1180 (java) R 1172 1174 1174 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753420049192 139753469664388 0 4 0 16800975 0 0 0 -1 1 0 0 0 0 0
[pid=1174/tid=1181] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1181/stat : 1181 (java) S 1172 1174 1174 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753418996392 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1182] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1182/stat : 1182 (java) R 1172 1174 1174 0 -1 4202560 5 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753417944104 139753466524812 0 4 0 16800975 0 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1183] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1183/stat : 1183 (java) S 1172 1174 1174 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338225339 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753416891304 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1174/tid=1184] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1184/stat : 1184 (java) S 1172 1174 1174 0 -1 4202560 70 0 0 0 0 0 0 0 20 0 18 0 338225340 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753286526088 219517334588 0 0 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1185] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1185/stat : 1185 (java) S 1172 1174 1174 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 18 0 338225340 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753285473272 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1186] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1186/stat : 1186 (java) S 1172 1174 1174 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 18 0 338225340 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753284420872 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1187] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1187/stat : 1187 (java) S 1172 1174 1174 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338225343 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753283369360 219517343520 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1188] ppid=1172 vsize=9558408 CPUtime=0.25 cores=1,3,5,7
/proc/1174/task/1188/stat : 1188 (java) S 1172 1174 1174 0 -1 4202560 2122 0 0 0 25 0 0 0 20 0 18 0 338225343 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753282316984 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1174/tid=1189] ppid=1172 vsize=9558408 CPUtime=0.19 cores=1,3,5,7
/proc/1174/task/1189/stat : 1189 (java) S 1172 1174 1174 0 -1 4202560 2166 0 0 0 19 0 0 0 20 0 18 0 338225343 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753281264184 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1174/tid=1190] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1190/stat : 1190 (java) S 1172 1174 1174 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338225343 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753280212088 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1174/tid=1191] ppid=1172 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1174/task/1191/stat : 1191 (java) S 1172 1174 1174 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338225343 9787809792 43971 33554432000 1073741824 1073778376 140734777165152 139753279159496 219517335465 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1042] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1042/stat : 1042 (python) S 1033 1035 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 8 0 338190904 898105344 23487 33554432000 4194304 5683644 140736014534816 139903363166768 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1046] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1046/stat : 1046 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338195626 898105344 23487 33554432000 4194304 5683644 140736014534816 139903352676912 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1035/tid=1058] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1058/stat : 1058 (python) S 1033 1035 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 8 0 338200412 898105344 23487 33554432000 4194304 5683644 140736014534816 139903342187056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=1035/tid=1066] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1066/stat : 1066 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338206439 898105344 23487 33554432000 4194304 5683644 140736014534816 139903331697200 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1035/tid=1071] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1071/stat : 1071 (python) S 1033 1035 27000 0 -1 4202560 24 1010 0 0 0 0 0 0 20 0 8 0 338211240 898105344 23487 33554432000 4194304 5683644 140736014534816 139903321207344 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1035/tid=1096] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1096/stat : 1096 (python) S 1033 1035 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338215957 898105344 23487 33554432000 4194304 5683644 140736014534816 139903298421296 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1035/tid=1173] ppid=1033 vsize=877056 CPUtime=0 cores=1,3,5,7
/proc/1035/task/1173/stat : 1173 (python) S 1033 1035 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 8 0 338225336 898105344 23487 33554432000 4194304 5683644 140736014534816 139903287931440 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 350.83
Current children cumulated vsize (KiB) 19438984

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1035 and gives
#  childrusage.ru_utime.tv_sec=18
#  childrusage.ru_utime.tv_usec=80251
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=219966
# CPU time returned by wait4() is 18.3002
# while last known CPU time is 350.83
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 350.865
CPU time (s): 350.83
CPU user time (s): 346.13
CPU system time (s): 4.7
CPU usage (%): 99.99
Max. virtual memory (cumulated for all children) (KiB): 19438984

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 18.0803
system time used= 0.219966
maximum resident set size= 103076
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 34905
page faults= 1
swaps= 0
block input operations= 0
block output operations= 28344
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2082
involuntary context switches= 35

runsolver used 1.37279 second user time and 2.89456 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 12:50:29
IDJOB=3740994
IDBENCH=89726
IDSOLVER=2327
FILE ID=node114/3740994-1338720628
RUNJOBID= node114-1338713908-27017
PBS_JOBID= 14636748
Free space on /tmp= 65976 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-48.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3740994-1338720628/watcher-3740994-1338720628 -o /tmp/evaluation-result-3740994-1338720628/solver-3740994-1338720628 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740994-1338720628.opb -seed 1614251005 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= bcbd6cca7644150bcdcbd185d4571ad6
RANDOM SEED=1614251005

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        13171580 kB
Buffers:         1007404 kB
Cached:         12097148 kB
SwapCached:         2180 kB
Active:          8350644 kB
Inactive:       10031788 kB
Active(anon):    5262688 kB
Inactive(anon):    10708 kB
Active(file):    3087956 kB
Inactive(file): 10021080 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:           5263132 kB
Writeback:         16000 kB
AnonPages:       5276164 kB
Mapped:            20672 kB
Shmem:               160 kB
Slab:            1168152 kB
SReclaimable:     605304 kB
SUnreclaim:       562848 kB
KernelStack:        1576 kB
PageTables:        17928 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    5877592 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    120832 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 69900 MiB
End job on node114 at 2012-06-03 12:56:23