Trace number 3742330

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 283.65 281.156

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/
ShortestPathNG/normalized-NG_K139.opb
MD5SUM76516381ba4bc043f2da87444e256707
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables16541
Total number of constraints16821
Number of constraints which are clauses16682
Number of constraints which are cardinality constraints (but not clauses)139
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint139
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 140
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.42/1.21	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.42/1.21	c INFO: solving HOME/instance-3742330-1338724475.opb
0.53/1.39	c INFO: parsing took 0.18 s
0.73/1.71	c INFO: features took 0.32 CPU seconds
0.73/1.74	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.60)
50.36/51.17	c INFO: running clasp-1.3.7@0 for 45 with 1750 remaining (b = 0.44)
97.60/98.33	c INFO: running scip-2.0.1-spx@0 for 45 with 1703 remaining (b = 0.05)
147.98/148.61	c INFO: running scip-2.0.1-clp@48 for 45 with 1653 remaining (b = 0.01)
191.15/191.75	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1610 remaining (b = 0.04)
237.38/235.46	c INFO: running clasp-1.3.7@47 for 45 with 1564 remaining (b = 0.01)
282.66/280.61	c INFO: running sat4j-pb-v20101225@0 for 45 with 1519 remaining (b = 0.15)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742330-1338724475/watcher-3742330-1338724475 -o /tmp/evaluation-result-3742330-1338724475/solver-3742330-1338724475 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742330-1338724475.opb -seed 2090913201 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.93 2.00 2.05 3/183 2653
/proc/meminfo: memFree=26379916/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/2653/stat : 2653 (python) D 2651 2653 27000 0 -1 4202496 671 0 0 0 0 0 0 0 20 0 1 0 338575020 15642624 559 33554432000 4194304 5683644 140733490026896 140733489994840 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 3819 559 324 364 0 261 0

[startup+0.013336 s]
/proc/loadavg: 1.93 2.00 2.05 3/183 2653
/proc/meminfo: memFree=26379916/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=15380 CPUtime=0 cores=1,3,5,7
/proc/2653/stat : 2653 (python) D 2651 2653 27000 0 -1 4202496 757 0 0 0 0 0 0 0 20 0 1 0 338575020 15749120 631 33554432000 4194304 5683644 140733490026896 140733489967384 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 3845 631 334 364 0 287 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15380

[startup+0.100291 s]
/proc/loadavg: 1.93 2.00 2.05 3/183 2653
/proc/meminfo: memFree=26379916/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=19088 CPUtime=0.03 cores=1,3,5,7
/proc/2653/stat : 2653 (python) R 2651 2653 27000 0 -1 4202496 1398 0 0 0 2 1 0 0 20 0 1 0 338575020 19546112 1153 33554432000 4194304 5683644 140733490026896 140733489957400 4873208 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 4772 1153 430 364 0 696 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 19088

[startup+0.300242 s]
/proc/loadavg: 1.93 2.00 2.05 3/183 2653
/proc/meminfo: memFree=26379916/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=102080 CPUtime=0.12 cores=1,3,5,7
/proc/2653/stat : 2653 (python) R 2651 2653 27000 0 -1 4202496 4134 1010 1 0 9 3 0 0 20 0 1 0 338575020 104529920 2732 33554432000 4194304 5683644 140733490026896 140733489867864 219513963408 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 25520 2732 876 364 0 1730 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 102080

[startup+0.700251 s]
/proc/loadavg: 1.93 2.00 2.05 3/183 2653
/proc/meminfo: memFree=26379916/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=162444 CPUtime=0.23 cores=1,3,5,7
/proc/2653/stat : 2653 (python) D 2651 2653 27000 0 -1 4202496 6447 1010 1 0 16 7 0 0 20 0 1 0 338575020 166342656 4260 33554432000 4194304 5683644 140733490026896 140733489926888 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 40611 4260 1192 364 0 3256 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 162444

[startup+1.50023 s]
/proc/loadavg: 1.93 2.00 2.05 2/184 2658
/proc/meminfo: memFree=26360432/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=288428 CPUtime=0.73 cores=1,3,5,7
/proc/2653/stat : 2653 (python) R 2651 2653 27000 0 -1 4202496 12949 1010 1 0 61 12 0 0 20 0 1 0 338575020 295350272 8858 33554432000 4194304 5683644 140733490026896 140733490013720 4963040 0 0 16781312 33554434 0 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 72107 8858 1846 364 0 7336 0
Current children cumulated CPU time (s) 0.73
Current children cumulated vsize (KiB) 288428

[startup+3.10035 s]
/proc/loadavg: 2.10 2.03 2.06 3/187 2661
/proc/meminfo: memFree=26226876/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=364628 CPUtime=0.97 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 13343 1010 1 0 85 12 0 0 20 0 2 0 338575020 373379072 9019 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 91157 9019 1902 364 0 26380 0
[pid=2659] ppid=2653 vsize=288848 CPUtime=0 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 767 0 0 0 0 0 0 0 20 0 1 0 338575195 295780352 7451 33554432000 4194304 5683644 140733490026896 140733490017984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/2659/statm: 72212 7451 340 364 0 7435 0
[pid=2661] ppid=2659 vsize=154600 CPUtime=1.34 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) R 2659 2661 2661 0 -1 4202496 35567 0 0 0 126 8 0 0 20 0 1 0 338575195 158310400 34854 33554432000 4194304 15243407 140736541887168 140736541882552 13229099 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 38650 34854 682 2698 0 35945 0
[pid=2653/tid=2660] ppid=2651 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338575195 373379072 9019 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 2.31
Current children cumulated vsize (KiB) 808076
heavy processes:

[startup+6.30047 s]
/proc/loadavg: 2.10 2.03 2.06 3/187 2661
/proc/meminfo: memFree=26149004/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=364628 CPUtime=0.97 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 13343 1010 1 0 85 12 0 0 20 0 2 0 338575020 373379072 9019 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 91157 9019 1902 364 0 26380 0
[pid=2659] ppid=2653 vsize=288848 CPUtime=0 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 895 0 0 0 0 0 0 0 20 0 1 0 338575195 295780352 7474 33554432000 4194304 5683644 140733490026896 140733490017984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2659/statm: 72212 7474 363 364 0 7435 0
[pid=2661] ppid=2659 vsize=222436 CPUtime=4.54 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) R 2659 2661 2661 0 -1 4202496 51517 0 0 0 442 12 0 0 20 0 1 0 338575195 227774464 50804 33554432000 4194304 15243407 140736541887168 140736541882552 4520304 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 55609 50804 682 2698 0 52904 0
[pid=2653/tid=2660] ppid=2651 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338575195 373379072 9019 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 5.51
Current children cumulated vsize (KiB) 875912

[startup+12.7004 s]
/proc/loadavg: 2.08 2.03 2.06 3/187 2661
/proc/meminfo: memFree=26102672/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=364628 CPUtime=0.97 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 13343 1010 1 0 85 12 0 0 20 0 2 0 338575020 373379072 9019 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 91157 9019 1902 364 0 26380 0
[pid=2659] ppid=2653 vsize=288848 CPUtime=0 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 898 0 0 0 0 0 0 0 20 0 1 0 338575195 295780352 7474 33554432000 4194304 5683644 140733490026896 140733490017984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2659/statm: 72212 7474 363 364 0 7435 0
[pid=2661] ppid=2659 vsize=272796 CPUtime=10.95 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) R 2659 2661 2661 0 -1 4202496 63163 0 0 0 1081 14 0 0 20 0 1 0 338575195 279343104 62450 33554432000 4194304 15243407 140736541887168 140736541883480 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 68199 62450 682 2698 0 65494 0
[pid=2653/tid=2660] ppid=2651 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338575195 373379072 9019 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.92
Current children cumulated vsize (KiB) 926272

[startup+25.5004 s]
/proc/loadavg: 2.07 2.03 2.06 3/187 2661
/proc/meminfo: memFree=25951540/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=364628 CPUtime=0.97 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 13343 1010 1 0 85 12 0 0 20 0 2 0 338575020 373379072 9019 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 91157 9019 1902 364 0 26380 0
[pid=2659] ppid=2653 vsize=288848 CPUtime=0.03 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 1005 0 0 0 2 1 0 0 20 0 1 0 338575195 295780352 7479 33554432000 4194304 5683644 140733490026896 140733490017984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2659/statm: 72212 7479 368 364 0 7435 0
[pid=2661] ppid=2659 vsize=428112 CPUtime=23.74 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) R 2659 2661 2661 0 -1 4202496 101738 0 0 0 2346 28 0 0 20 0 1 0 338575195 438386688 101025 33554432000 4194304 15243407 140736541887168 140736541883480 6711570 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 107028 101025 686 2698 0 104323 0
[pid=2653/tid=2660] ppid=2651 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338575195 373379072 9019 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.74
Current children cumulated vsize (KiB) 1081588

[startup+51.1004 s]
/proc/loadavg: 2.12 2.04 2.06 3/191 2665
/proc/meminfo: memFree=25470512/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=364628 CPUtime=0.97 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 13343 1010 1 0 85 12 0 0 20 0 2 0 338575020 373379072 9019 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 91157 9019 1902 364 0 26380 0
[pid=2659] ppid=2653 vsize=288848 CPUtime=0.06 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 1023 0 0 0 4 2 0 0 20 0 1 0 338575195 295780352 7479 33554432000 4194304 5683644 140733490026896 140733490017984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2659/statm: 72212 7479 368 364 0 7435 0
[pid=2661] ppid=2659 vsize=583480 CPUtime=49.33 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) R 2659 2661 2661 0 -1 4202496 140197 0 0 0 4887 46 0 0 20 0 1 0 338575195 597483520 139485 33554432000 4194304 15243407 140736541887168 140736541883464 6707872 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 145870 139500 686 2698 0 143165 0
[pid=2653/tid=2660] ppid=2651 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338575195 373379072 9019 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.36
Current children cumulated vsize (KiB) 1236956

[startup+102.301 s]
/proc/loadavg: 2.09 2.05 2.06 3/200 2674
/proc/meminfo: memFree=24887804/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=516212 CPUtime=1.02 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 15164 1010 1 0 89 13 0 0 20 0 4 0 338575020 528601088 9036 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 129053 9036 1907 364 0 64270 0
[pid=2659] ppid=2653 vsize=364628 CPUtime=0.07 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 1164 0 0 0 4 3 0 0 20 0 2 0 338575195 373379072 7514 33554432000 4194304 5683644 140733490026896 140733490018544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2659/statm: 91157 7514 396 364 0 26380 0
[pid=2661] ppid=2659 vsize=583480 CPUtime=49.38 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) T 2659 2661 2661 0 -1 4202496 140274 0 0 0 4892 46 0 0 20 0 1 0 338575195 597483520 139561 33554432000 4194304 15243407 140736541887168 140736541883480 4880923 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 145870 139561 686 2698 0 143165 0
[pid=2659/tid=2666] ppid=2653 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2659/task/2666/stat : 2666 (python) S 2653 2653 27000 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338580136 373379072 7514 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2667] ppid=2653 vsize=364640 CPUtime=0.07 cores=1,3,5,7
/proc/2667/stat : 2667 (python) S 2653 2653 27000 0 -1 4202560 1097 0 0 0 4 3 0 0 20 0 2 0 338580138 373391360 7501 33554432000 4194304 5683644 140733490026896 140733490018544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2667/statm: 91160 7501 383 364 0 26380 0
[pid=2669] ppid=2667 vsize=18456 CPUtime=47.11 cores=1,3,5,7
/proc/2669/stat : 2669 (clasp-1.3.7-x86) T 2667 2669 2669 0 -1 4202496 4879 0 0 0 4711 0 0 0 20 0 1 0 338580138 18898944 4370 33554432000 134512640 136139019 4288499552 4288498528 134883415 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2669/statm: 4614 4370 182 398 0 4214 0
[pid=2667/tid=2671] ppid=2653 vsize=364640 CPUtime=0 cores=1,3,5,7
/proc/2667/task/2671/stat : 2671 (python) S 2653 2653 27000 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338584851 373391360 7501 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2672] ppid=2653 vsize=440432 CPUtime=0 cores=1,3,5,7
/proc/2672/stat : 2672 (python) S 2653 2653 27000 0 -1 4202560 933 0 0 0 0 0 0 0 20 0 1 0 338584854 451002368 7481 33554432000 4194304 5683644 140733490026896 140733490017984 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2672/statm: 110108 7481 358 364 0 45325 0
[pid=2674] ppid=2672 vsize=221204 CPUtime=3.96 cores=1,3,5,7
/proc/2674/stat : 2674 (scip-2.0.1.linu) R 2672 2674 2674 0 -1 4202496 51314 0 0 0 383 13 0 0 20 0 1 0 338584854 226512896 50590 33554432000 4194304 13684479 140735820230608 140735820225992 4880960 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2674/statm: 55301 50590 635 2317 0 52979 0
[pid=2653/tid=2660] ppid=2651 vsize=516212 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 4 0 338575195 528601088 9036 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2653/tid=2668] ppid=2651 vsize=516212 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2668/stat : 2668 (python) S 2651 2653 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 4 0 338580138 528601088 9036 33554432000 4194304 5683644 140733490026896 140704303792688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2653/tid=2673] ppid=2651 vsize=516212 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2673/stat : 2673 (python) S 2651 2653 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 4 0 338584854 528601088 9036 33554432000 4194304 5683644 140733490026896 140704293302832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.61
Current children cumulated vsize (KiB) 2509052

[startup+162.301 s]
/proc/loadavg: 2.03 2.04 2.06 3/222 2696
/proc/meminfo: memFree=22145404/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=516212 CPUtime=1.05 cores=1,3,5,7

################
# More data... #
################

[pid=2701/tid=2715] ppid=2699 vsize=9624972 CPUtime=1.03 cores=1,3,5,7
/proc/2701/task/2715/stat : 2715 (java) S 2699 2701 2701 0 -1 4202560 8804 0 0 0 101 2 0 0 20 0 19 0 338594203 9855971328 267434 33554432000 1073741824 1073778376 140736244831408 139989052799800 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2701/tid=2716] ppid=2699 vsize=9624972 CPUtime=0.88 cores=1,3,5,7
/proc/2701/task/2716/stat : 2716 (java) S 2699 2701 2701 0 -1 4202560 7210 0 0 0 88 0 0 0 20 0 19 0 338594203 9855971328 267434 33554432000 1073741824 1073778376 140736244831408 139989051747256 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2701/tid=2717] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2717/stat : 2717 (java) S 2699 2701 2701 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338594203 9855971328 267434 33554432000 1073741824 1073778376 140736244831408 139989050694392 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2701/tid=2718] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2718/stat : 2718 (java) S 2699 2701 2701 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338594203 9855971328 267434 33554432000 1073741824 1073778376 140736244831408 139989049642056 219517335465 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2701/tid=2719] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2719/stat : 2719 (java) S 2699 2701 2701 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338594255 9855971328 267434 33554432000 1073741824 1073778376 140736244831408 139989048587304 219517335465 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2653/tid=2660] ppid=2651 vsize=592004 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 5 0 338575195 606212096 9044 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2653/tid=2668] ppid=2651 vsize=592004 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2668/stat : 2668 (python) S 2651 2653 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 5 0 338580138 606212096 9044 33554432000 4194304 5683644 140733490026896 140704303792688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2653/tid=2673] ppid=2651 vsize=592004 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2673/stat : 2673 (python) S 2651 2653 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 5 0 338584854 606212096 9044 33554432000 4194304 5683644 140733490026896 140704293302832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2653/tid=2700] ppid=2651 vsize=592004 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2700/stat : 2700 (python) S 2651 2653 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 5 0 338594196 606212096 9044 33554432000 4194304 5683644 140733490026896 140704282812976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 224.12
Current children cumulated vsize (KiB) 13339240



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+281.101 s]
/proc/loadavg: 2.02 2.03 2.05 5/238 2812
/proc/meminfo: memFree=23789928/32873648 swapFree=7120/7120
[pid=2653] ppid=2651 vsize=667796 CPUtime=1.11 cores=1,3,5,7
/proc/2653/stat : 2653 (python) S 2651 2653 27000 0 -1 4202496 17115 1010 1 0 98 13 0 0 20 0 6 0 338575020 683823104 9560 33554432000 4194304 5683644 140733490026896 140733490021728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2653/statm: 166949 9560 1911 364 0 102160 0
[pid=2659] ppid=2653 vsize=364628 CPUtime=0.15 cores=1,3,5,7
/proc/2659/stat : 2659 (python) S 2653 2653 27000 0 -1 4202560 1192 0 0 0 10 5 0 0 20 0 2 0 338575195 373379072 7515 33554432000 4194304 5683644 140733490026896 140733490018544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2659/statm: 91157 7515 397 364 0 26380 0
[pid=2661] ppid=2659 vsize=826516 CPUtime=92.49 cores=1,3,5,7
/proc/2661/stat : 2661 (scip-2.0.1.linu) T 2659 2661 2661 0 -1 4202496 201412 0 0 0 9174 75 0 0 20 0 1 0 338575195 846352384 200702 33554432000 4194304 15243407 140736541887168 140736541883464 4843100 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2661/statm: 206629 200702 686 2698 0 203924 0
[pid=2659/tid=2666] ppid=2653 vsize=364628 CPUtime=0 cores=1,3,5,7
/proc/2659/task/2666/stat : 2666 (python) S 2653 2653 27000 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338580136 373379072 7515 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2667] ppid=2653 vsize=364640 CPUtime=0.15 cores=1,3,5,7
/proc/2667/stat : 2667 (python) S 2653 2653 27000 0 -1 4202560 1157 0 0 0 9 6 0 0 20 0 2 0 338580138 373391360 7501 33554432000 4194304 5683644 140733490026896 140733490018544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2667/statm: 91160 7501 383 364 0 26380 0
[pid=2669] ppid=2667 vsize=31856 CPUtime=92.23 cores=1,3,5,7
/proc/2669/stat : 2669 (clasp-1.3.7-x86) T 2667 2669 2669 0 -1 4202496 8228 0 0 0 9222 1 0 0 20 0 1 0 338580138 32620544 7719 33554432000 134512640 136139019 4288499552 4288498176 134599833 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2669/statm: 7964 7719 182 398 0 7564 0
[pid=2667/tid=2671] ppid=2653 vsize=364640 CPUtime=0 cores=1,3,5,7
/proc/2667/task/2671/stat : 2671 (python) S 2653 2653 27000 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338584851 373391360 7501 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2672] ppid=2653 vsize=440432 CPUtime=0.08 cores=1,3,5,7
/proc/2672/stat : 2672 (python) S 2653 2653 27000 0 -1 4202560 1121 0 0 0 4 4 0 0 20 0 2 0 338584854 451002368 7507 33554432000 4194304 5683644 140733490026896 140733490018544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2672/statm: 110108 7507 383 364 0 45325 0
[pid=2674] ppid=2672 vsize=591368 CPUtime=50.24 cores=1,3,5,7
/proc/2674/stat : 2674 (scip-2.0.1.linu) T 2672 2674 2674 0 -1 4202496 142732 0 0 0 4978 46 0 0 20 0 1 0 338584854 605560832 142008 33554432000 4194304 13684479 140735820230608 140735820226920 4477584 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2674/statm: 147842 142008 639 2317 0 145520 0
[pid=2672/tid=2696] ppid=2653 vsize=440432 CPUtime=0 cores=1,3,5,7
/proc/2672/task/2696/stat : 2696 (python) S 2653 2653 27000 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338589879 451002368 7507 33554432000 4194304 5683644 140733490026896 140704303792688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2699] ppid=2653 vsize=516224 CPUtime=0.07 cores=1,3,5,7
/proc/2699/stat : 2699 (python) S 2653 2653 27000 0 -1 4202560 1143 0 0 0 4 3 0 0 20 0 2 0 338594196 528613376 7513 33554432000 4194304 5683644 140733490026896 140733490018544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2699/statm: 129056 7513 383 364 0 64270 0
[pid=2701] ppid=2699 vsize=9624972 CPUtime=46.15 cores=1,3,5,7
/proc/2701/stat : 2701 (java) T 2699 2701 2701 0 -1 4202496 21127 0 1 0 4585 30 0 0 20 0 19 0 338594196 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 140736244822544 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2701/statm: 2406243 258811 2269 9 0 2396937 0
[pid=2701/tid=2702] ppid=2699 vsize=9624972 CPUtime=43.37 cores=1,3,5,7
/proc/2701/task/2702/stat : 2702 (java) T 2699 2701 2701 0 -1 4202560 2956 0 1 0 4320 17 0 0 20 0 19 0 338594198 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989233077312 139989105997045 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2703] ppid=2699 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/2701/task/2703/stat : 2703 (java) T 2699 2701 2701 0 -1 4202560 49 0 0 0 9 2 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989076576680 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2701/tid=2704] ppid=2699 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2701/task/2704/stat : 2704 (java) T 2699 2701 2701 0 -1 4202560 57 0 0 0 7 1 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989075524136 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2705] ppid=2699 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2701/task/2705/stat : 2705 (java) T 2699 2701 2701 0 -1 4202560 31 0 0 0 7 2 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989074471080 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2706] ppid=2699 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2701/task/2706/stat : 2706 (java) T 2699 2701 2701 0 -1 4202560 28 0 0 0 8 0 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989073418536 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2701/tid=2707] ppid=2699 vsize=9624972 CPUtime=0.07 cores=1,3,5,7
/proc/2701/task/2707/stat : 2707 (java) T 2699 2701 2701 0 -1 4202560 15 0 0 0 7 0 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989072366504 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2708] ppid=2699 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2701/task/2708/stat : 2708 (java) T 2699 2701 2701 0 -1 4202560 26 0 0 0 9 0 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989071313960 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2701/tid=2709] ppid=2699 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2701/task/2709/stat : 2709 (java) T 2699 2701 2701 0 -1 4202560 52 0 0 0 7 2 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989070260904 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2701/tid=2710] ppid=2699 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/2701/task/2710/stat : 2710 (java) T 2699 2701 2701 0 -1 4202560 42 0 0 0 9 2 0 0 20 0 19 0 338594199 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989069208360 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2711] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2711/stat : 2711 (java) T 2699 2701 2701 0 -1 4202560 438 0 0 0 0 0 0 0 20 0 19 0 338594200 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989057010024 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2712] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2712/stat : 2712 (java) T 2699 2701 2701 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338594200 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989055956344 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2713] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2713/stat : 2713 (java) T 2699 2701 2701 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 19 0 338594200 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989054903176 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2714] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2714/stat : 2714 (java) T 2699 2701 2701 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338594203 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989053851920 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2701/tid=2715] ppid=2699 vsize=9624972 CPUtime=1.04 cores=1,3,5,7
/proc/2701/task/2715/stat : 2715 (java) T 2699 2701 2701 0 -1 4202560 8805 0 0 0 102 2 0 0 20 0 19 0 338594203 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989052799800 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2701/tid=2716] ppid=2699 vsize=9624972 CPUtime=0.88 cores=1,3,5,7
/proc/2701/task/2716/stat : 2716 (java) T 2699 2701 2701 0 -1 4202560 7210 0 0 0 88 0 0 0 20 0 19 0 338594203 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989051747256 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2701/tid=2717] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2717/stat : 2717 (java) T 2699 2701 2701 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338594203 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989050694392 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2701/tid=2718] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2718/stat : 2718 (java) T 2699 2701 2701 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338594203 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989049642056 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2701/tid=2719] ppid=2699 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2701/task/2719/stat : 2719 (java) T 2699 2701 2701 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338594255 9855971328 258811 33554432000 1073741824 1073778376 140736244831408 139989048587304 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2699/tid=2788] ppid=2653 vsize=516224 CPUtime=0 cores=1,3,5,7
/proc/2699/task/2788/stat : 2788 (python) S 2653 2653 27000 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338598564 528613376 7513 33554432000 4194304 5683644 140733490026896 140704293302832 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2793] ppid=2653 vsize=592016 CPUtime=0 cores=1,3,5,7
/proc/2793/stat : 2793 (python) S 2653 2653 27000 0 -1 4202560 868 0 0 0 0 0 0 0 20 0 1 0 338603082 606224384 7470 33554432000 4194304 5683644 140733490026896 140733490017984 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 7 0 0 0 0 0
/proc/2793/statm: 148004 7470 335 364 0 83215 0
[pid=2795] ppid=2793 vsize=9558408 CPUtime=0.98 cores=1,3,5,7
/proc/2795/stat : 2795 (java) S 2793 2795 2795 0 -1 4202496 10337 0 1 0 95 3 0 0 20 0 18 0 338603082 9787809792 26272 33554432000 1073741824 1073778376 140734142950992 140734142942128 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2795/statm: 2389602 26272 2235 9 0 2380296 0
[pid=2795/tid=2796] ppid=2793 vsize=9558408 CPUtime=0.46 cores=1,3,5,7
/proc/2795/task/2796/stat : 2796 (java) R 2793 2795 2795 0 -1 4202560 2463 0 1 0 44 2 0 0 20 0 18 0 338603083 9787809792 26272 33554432000 1073741824 1073778376 140734142950992 140045998078896 140046005844819 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=2795/tid=2797] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2797/stat : 2797 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26272 33554432000 1073741824 1073778376 140734142950992 140045936810664 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2798] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2798/stat : 2798 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26272 33554432000 1073741824 1073778376 140734142950992 140045935758120 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2795/tid=2799] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2799/stat : 2799 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045934705576 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2795/tid=2800] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2800/stat : 2800 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045933653032 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2795/tid=2801] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2801/stat : 2801 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045932599464 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2802] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2802/stat : 2802 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045836826920 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2795/tid=2803] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2803/stat : 2803 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045835774376 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2804] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2804/stat : 2804 (java) S 2793 2795 2795 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338603084 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045834721832 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2805] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2805/stat : 2805 (java) S 2793 2795 2795 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338603085 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045822311016 219517335465 0 0 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2806] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2806/stat : 2806 (java) S 2793 2795 2795 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338603086 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045821257336 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2807] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2807/stat : 2807 (java) S 2793 2795 2795 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338603086 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045820204680 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2808] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2808/stat : 2808 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603088 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045819153424 219517343520 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2795/tid=2809] ppid=2793 vsize=9558408 CPUtime=0.22 cores=1,3,5,7
/proc/2795/task/2809/stat : 2809 (java) R 2793 2795 2795 0 -1 4202560 2697 0 0 0 22 0 0 0 20 0 18 0 338603088 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045818087048 140046000947824 0 4 0 16800975 0 0 0 -1 3 0 0 0 0 0
[pid=2795/tid=2810] ppid=2793 vsize=9558408 CPUtime=0.26 cores=1,3,5,7
/proc/2795/task/2810/stat : 2810 (java) R 2793 2795 2795 0 -1 4202560 3762 0 0 0 26 0 0 0 20 0 18 0 338603088 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045817024472 140046005054840 0 4 0 16800975 0 0 0 -1 1 0 0 0 0 0
[pid=2795/tid=2811] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2811/stat : 2811 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603088 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045815994872 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2795/tid=2812] ppid=2793 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/2795/task/2812/stat : 2812 (java) S 2793 2795 2795 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338603088 9787809792 26784 33554432000 1073741824 1073778376 140734142950992 140045814942536 219517335465 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2653/tid=2660] ppid=2651 vsize=667796 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2660/stat : 2660 (python) S 2651 2653 27000 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 6 0 338575195 683823104 9560 33554432000 4194304 5683644 140733490026896 140704314282544 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2653/tid=2668] ppid=2651 vsize=667796 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2668/stat : 2668 (python) S 2651 2653 27000 0 -1 4202560 28 1010 0 0 0 0 0 0 20 0 6 0 338580138 683823104 9560 33554432000 4194304 5683644 140733490026896 140704303792688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2653/tid=2673] ppid=2651 vsize=667796 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2673/stat : 2673 (python) S 2651 2653 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 6 0 338584854 683823104 9560 33554432000 4194304 5683644 140733490026896 140704293302832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2653/tid=2700] ppid=2651 vsize=667796 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2700/stat : 2700 (python) S 2651 2653 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 6 0 338594196 683823104 9560 33554432000 4194304 5683644 140733490026896 140704282812976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2653/tid=2794] ppid=2651 vsize=667796 CPUtime=0 cores=1,3,5,7
/proc/2653/task/2794/stat : 2794 (python) S 2651 2653 27000 0 -1 4202560 20 1010 0 0 0 0 0 0 20 0 6 0 338603082 683823104 9560 33554432000 4194304 5683644 140733490026896 140704068133424 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 283.65
Current children cumulated vsize (KiB) 23578856

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2653 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=989849
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=139978
# CPU time returned by wait4() is 1.12983
# while last known CPU time is 283.65
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 281.156
CPU time (s): 283.65
CPU user time (s): 281.79
CPU system time (s): 1.86
CPU usage (%): 100.887
Max. virtual memory (cumulated for all children) (KiB): 23578856

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.989849
system time used= 0.139978
maximum resident set size= 38240
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18125
page faults= 1
swaps= 0
block input operations= 0
block output operations= 448
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2070
involuntary context switches= 9

runsolver used 1.14083 second user time and 2.36664 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:54:35
IDJOB=3742330
IDBENCH=90159
IDSOLVER=2327
FILE ID=node114/3742330-1338724475
RUNJOBID= node114-1338713908-27017
PBS_JOBID= 14636748
Free space on /tmp= 70040 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathNG/normalized-NG_K139.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742330-1338724475/watcher-3742330-1338724475 -o /tmp/evaluation-result-3742330-1338724475/solver-3742330-1338724475 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742330-1338724475.opb -seed 2090913201 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 76516381ba4bc043f2da87444e256707
RANDOM SEED=2090913201

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        26380320 kB
Buffers:         1040576 kB
Cached:          3821496 kB
SwapCached:         2180 kB
Active:          3708000 kB
Inactive:        1774648 kB
Active(anon):     605388 kB
Inactive(anon):    10672 kB
Active(file):    3102612 kB
Inactive(file):  1763976 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:            102876 kB
Writeback:            44 kB
AnonPages:        619100 kB
Mapped:            20428 kB
Shmem:               136 kB
Slab:             870108 kB
SReclaimable:     312320 kB
SUnreclaim:       557788 kB
KernelStack:        1504 kB
PageTables:         7144 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     851652 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 69940 MiB
End job on node114 at 2012-06-03 13:59:20