Trace number 3742174

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 614.51 612.865

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K109.opb
MD5SUM3a5151d7fc286e1b50737dee685a4276
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark28.7146
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables7412
Total number of constraints7632
Number of constraints which are clauses7523
Number of constraints which are cardinality constraints (but not clauses)109
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint109
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 110
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.40/1.60	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.40/1.60	c INFO: solving HOME/instance-3742174-1338723795.opb
0.40/1.68	c INFO: parsing took 0.08 s
0.64/1.83	c INFO: features took 0.15 CPU seconds
0.64/1.86	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.67)
50.89/52.07	c INFO: running clasp-1.3.7@0 for 45 with 1750 remaining (b = 0.36)
98.15/99.23	c INFO: running scip-2.0.1-spx@0 for 45 with 1703 remaining (b = 0.06)
147.80/148.81	c INFO: running clasp-1.3.7@47 for 45 with 1653 remaining (b = 0.01)
193.68/194.63	c INFO: wrote minion input file to HOME/tmpsvi485.minion
193.68/194.63	c INFO: running minion-0.12@0 for 45 with 1608 remaining (b = 0.09)
241.12/242.04	c INFO: running wbo1.4a@0 for 45 with 1560 remaining (b = 0.02)
291.39/292.24	c INFO: running scip-2.0.1-clp@49 for 45 with 1511 remaining (b = 0.01)
334.05/334.88	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1468 remaining (b = 0.03)
383.72/382.47	c INFO: running clasp-1.3.7@92 for 136 with 1419 remaining (b = 0.02)
521.35/519.88	c INFO: running scip-2.0.1-spx@49 for 45 with 1282 remaining (b = 0.01)
566.64/565.04	c INFO: running bsolo_pb10-l3@0 for 45 with 1237 remaining (b = 0.02)
614.41/612.70	c INFO: running bsolo_pb10-l1@0 for 91 with 1189 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742174-1338723795/watcher-3742174-1338723795 -o /tmp/evaluation-result-3742174-1338723795/solver-3742174-1338723795 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742174-1338723795.opb -seed 722012330 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.08 2.05 2.23 2/191 7443
/proc/meminfo: memFree=30472736/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=15308 CPUtime=0 cores=1,3,5,7
/proc/7443/stat : 7443 (python) R 7441 7443 2614 0 -1 4202496 641 0 0 0 0 0 0 0 20 0 1 0 338504588 15675392 536 33554432000 4194304 5683644 140733627799264 140733627785176 240564801428 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7443/statm: 3827 541 314 364 0 269 0

[startup+0.0803489 s]
/proc/loadavg: 2.08 2.05 2.23 2/191 7443
/proc/meminfo: memFree=30472736/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=17012 CPUtime=0.01 cores=1,3,5,7
/proc/7443/stat : 7443 (python) D 7441 7443 2614 0 -1 4202496 1332 0 0 0 1 0 0 0 20 0 1 0 338504588 17420288 1097 33554432000 4194304 5683644 140733627799264 140733627725528 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/7443/statm: 4253 1097 426 364 0 695 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 17012

[startup+0.100281 s]
/proc/loadavg: 2.08 2.05 2.23 2/191 7443
/proc/meminfo: memFree=30472736/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=19084 CPUtime=0.02 cores=1,3,5,7
/proc/7443/stat : 7443 (python) D 7441 7443 2614 0 -1 4202496 1398 0 0 0 2 0 0 0 20 0 1 0 338504588 19542016 1153 33554432000 4194304 5683644 140733627799264 140733627739256 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/7443/statm: 4771 1153 430 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19084

[startup+0.30026 s]
/proc/loadavg: 2.08 2.05 2.23 2/191 7443
/proc/meminfo: memFree=30472736/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=102076 CPUtime=0.11 cores=1,3,5,7
/proc/7443/stat : 7443 (python) D 7441 7443 2614 0 -1 4202496 4144 1011 1 0 8 3 0 0 20 0 1 0 338504588 104525824 2733 33554432000 4194304 5683644 140733627799264 140733627640232 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/7443/statm: 25519 2733 876 364 0 1729 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 102076

[startup+0.700234 s]
/proc/loadavg: 2.08 2.05 2.23 2/191 7443
/proc/meminfo: memFree=30472736/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=166944 CPUtime=0.23 cores=1,3,5,7
/proc/7443/stat : 7443 (python) R 7441 7443 2614 0 -1 4202496 6519 1011 1 0 16 7 0 0 20 0 1 0 338504588 170950656 4322 33554432000 4194304 5683644 140733627799264 140733627699256 240565175184 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7443/statm: 41736 4322 1215 364 0 3289 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 166944

[startup+1.50025 s]
/proc/loadavg: 2.08 2.05 2.23 2/192 7448
/proc/meminfo: memFree=30454740/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=260628 CPUtime=0.4 cores=1,3,5,7
/proc/7443/stat : 7443 (python) D 7441 7443 2614 0 -1 4202496 9248 1011 1 0 29 11 0 0 20 0 1 0 338504588 266883072 6531 33554432000 4194304 5683644 140733627799264 140733627735256 240565180263 0 0 16781312 2 18446744071580543392 0 0 17 1 0 0 0 0 0
/proc/7443/statm: 65157 6531 1706 364 0 5209 0
Current children cumulated CPU time (s) 0.4
Current children cumulated vsize (KiB) 260628

[startup+3.10042 s]
/proc/loadavg: 2.15 2.06 2.23 4/195 7451
/proc/meminfo: memFree=30381300/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=360180 CPUtime=0.7 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 11521 1011 1 0 58 12 0 0 20 0 2 0 338504588 368824320 7929 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 40 0 0
/proc/7443/statm: 90045 7929 1893 364 0 25268 0
[pid=7449] ppid=7443 vsize=284400 CPUtime=0 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 762 0 0 0 0 0 0 0 20 0 1 0 338504774 291225600 6371 33554432000 4194304 5683644 140733627799264 140733627790352 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/7449/statm: 71100 6371 341 364 0 6323 0
[pid=7451] ppid=7449 vsize=110128 CPUtime=1.22 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) R 7449 7451 7451 0 -1 4202496 23797 0 0 0 117 5 0 0 20 0 1 0 338504775 112771072 23196 33554432000 4194304 15243407 140734761315344 140734761310728 4861259 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7451/statm: 27532 23196 682 2698 0 24827 0
[pid=7443/tid=7450] ppid=7441 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338504775 368824320 7929 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 1.92
Current children cumulated vsize (KiB) 754708
heavy processes:

[startup+6.30043 s]
/proc/loadavg: 2.15 2.06 2.23 4/195 7451
/proc/meminfo: memFree=30308264/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=360180 CPUtime=0.7 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 11521 1011 1 0 58 12 0 0 20 0 2 0 338504588 368824320 7929 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 40 0 0
/proc/7443/statm: 90045 7929 1893 364 0 25268 0
[pid=7449] ppid=7443 vsize=284400 CPUtime=0 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 891 0 0 0 0 0 0 0 20 0 1 0 338504774 291225600 6394 33554432000 4194304 5683644 140733627799264 140733627790352 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7449/statm: 71100 6394 364 364 0 6323 0
[pid=7451] ppid=7449 vsize=170600 CPUtime=4.41 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) R 7449 7451 7451 0 -1 4202496 38200 0 0 0 433 8 0 0 20 0 1 0 338504775 174694400 37599 33554432000 4194304 15243407 140734761315344 140734761311656 13229099 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7451/statm: 42650 37599 682 2698 0 39945 0
[pid=7443/tid=7450] ppid=7441 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338504775 368824320 7929 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 5.11
Current children cumulated vsize (KiB) 815180

[startup+12.7004 s]
/proc/loadavg: 2.13 2.06 2.23 4/195 7451
/proc/meminfo: memFree=30250048/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=360180 CPUtime=0.7 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 11521 1011 1 0 58 12 0 0 20 0 2 0 338504588 368824320 7929 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 40 0 0
/proc/7443/statm: 90045 7929 1893 364 0 25268 0
[pid=7449] ppid=7443 vsize=284400 CPUtime=0.01 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 895 0 0 0 1 0 0 0 20 0 1 0 338504774 291225600 6394 33554432000 4194304 5683644 140733627799264 140733627790352 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7449/statm: 71100 6394 364 364 0 6323 0
[pid=7451] ppid=7449 vsize=222908 CPUtime=10.82 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) R 7449 7451 7451 0 -1 4202496 51370 0 0 0 1065 17 0 0 20 0 1 0 338504775 228257792 50769 33554432000 4194304 15243407 140734761315344 140734761311656 13277498 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7451/statm: 55727 50769 686 2698 0 53022 0
[pid=7443/tid=7450] ppid=7441 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338504775 368824320 7929 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.53
Current children cumulated vsize (KiB) 867488

[startup+25.5078 s]
/proc/loadavg: 2.19 2.07 2.23 4/195 7451
/proc/meminfo: memFree=30165664/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=360180 CPUtime=0.7 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 11521 1011 1 0 58 12 0 0 20 0 2 0 338504588 368824320 7929 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 40 0 0
/proc/7443/statm: 90045 7929 1893 364 0 25268 0
[pid=7449] ppid=7443 vsize=284400 CPUtime=0.03 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 996 0 0 0 2 1 0 0 20 0 1 0 338504774 291225600 6399 33554432000 4194304 5683644 140733627799264 140733627790352 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7449/statm: 71100 6399 369 364 0 6323 0
[pid=7451] ppid=7449 vsize=305336 CPUtime=23.62 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) R 7449 7451 7451 0 -1 4202496 71842 0 0 0 2330 32 0 0 20 0 1 0 338504775 312664064 71241 33554432000 4194304 15243407 140734761315344 140734761311656 4865653 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7451/statm: 76334 71241 686 2698 0 73629 0
[pid=7443/tid=7450] ppid=7441 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338504775 368824320 7929 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.35
Current children cumulated vsize (KiB) 949916

[startup+51.1003 s]
/proc/loadavg: 2.12 2.07 2.23 4/199 7455
/proc/meminfo: memFree=30062500/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=360180 CPUtime=0.7 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 11521 1011 1 0 58 12 0 0 20 0 2 0 338504588 368824320 7929 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 40 0 0
/proc/7443/statm: 90045 7929 1893 364 0 25268 0
[pid=7449] ppid=7443 vsize=284400 CPUtime=0.08 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 1021 0 0 0 5 3 0 0 20 0 1 0 338504774 291225600 6399 33554432000 4194304 5683644 140733627799264 140733627790352 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7449/statm: 71100 6399 369 364 0 6323 0
[pid=7451] ppid=7449 vsize=361816 CPUtime=49.21 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) R 7449 7451 7451 0 -1 4202496 84627 0 0 0 4827 94 0 0 20 0 1 0 338504775 370499584 85018 33554432000 4194304 15243407 140734761315344 140734761310056 13354455 0 0 16781312 2 0 0 0 17 5 0 0 0 0 0
/proc/7451/statm: 90454 85018 885 2698 0 87749 0
[pid=7443/tid=7450] ppid=7441 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338504775 368824320 7929 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 49.99
Current children cumulated vsize (KiB) 1006396

[startup+102.301 s]
/proc/loadavg: 2.05 2.05 2.21 4/208 7464
/proc/meminfo: memFree=29845968/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=511764 CPUtime=0.76 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 13320 1011 1 0 63 13 0 0 20 0 4 0 338504588 524046336 7954 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 40 0 0
/proc/7443/statm: 127941 7954 1906 364 0 63158 0
[pid=7449] ppid=7443 vsize=360180 CPUtime=0.08 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 1156 0 0 0 5 3 0 0 20 0 2 0 338504774 368824320 6434 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7449/statm: 90045 6434 397 364 0 25268 0
[pid=7451] ppid=7449 vsize=361816 CPUtime=50.16 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) T 7449 7451 7451 0 -1 4202496 84632 0 0 0 4922 94 0 0 20 0 1 0 338504775 370499584 85023 33554432000 4194304 15243407 140734761315344 140734761310056 8115387 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/7451/statm: 90454 85023 885 2698 0 87749 0
[pid=7449/tid=7456] ppid=7443 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7449/task/7456/stat : 7456 (python) S 7443 7443 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338509793 368824320 6434 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7457] ppid=7443 vsize=360192 CPUtime=0.07 cores=1,3,5,7
/proc/7457/stat : 7457 (python) S 7443 7443 2614 0 -1 4202560 1085 0 0 0 5 2 0 0 20 0 2 0 338509795 368836608 6421 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7457/statm: 90048 6421 384 364 0 25268 0
[pid=7459] ppid=7457 vsize=19996 CPUtime=47.11 cores=1,3,5,7
/proc/7459/stat : 7459 (clasp-1.3.7-x86) T 7457 7459 7459 0 -1 4202496 5105 0 0 0 4710 1 0 0 20 0 1 0 338509795 20475904 4738 33554432000 134512640 136139019 4293111712 4293110336 134599005 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7459/statm: 4999 4738 182 398 0 4599 0
[pid=7457/tid=7461] ppid=7443 vsize=360192 CPUtime=0 cores=1,3,5,7
/proc/7457/task/7461/stat : 7461 (python) S 7443 7443 2614 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338514509 368836608 6421 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7462] ppid=7443 vsize=435984 CPUtime=0 cores=1,3,5,7
/proc/7462/stat : 7462 (python) S 7443 7443 2614 0 -1 4202560 793 0 0 0 0 0 0 0 20 0 1 0 338514511 446447616 6378 33554432000 4194304 5683644 140733627799264 140733627790352 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7462/statm: 108996 6378 336 364 0 44213 0
[pid=7464] ppid=7462 vsize=139220 CPUtime=3.05 cores=1,3,5,7
/proc/7464/stat : 7464 (scip-2.0.1.linu) R 7462 7464 7464 0 -1 4202496 30755 0 0 0 298 7 0 0 20 0 1 0 338514512 142561280 30141 33554432000 4194304 13684479 140736698496160 140736698493080 5554632 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7464/statm: 34805 30142 635 2317 0 32483 0
[pid=7443/tid=7450] ppid=7441 vsize=511764 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 4 0 338504775 524046336 7954 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7443/tid=7458] ppid=7441 vsize=511764 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7458/stat : 7458 (python) S 7441 7443 2614 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 4 0 338509795 524046336 7954 33554432000 4194304 5683644 140733627799264 140370426053168 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7443/tid=7463] ppid=7441 vsize=511764 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7463/stat : 7463 (python) S 7441 7443 2614 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 4 0 338514511 524046336 7954 33554432000 4194304 5683644 140733627799264 140370415563312 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.23
Current children cumulated vsize (KiB) 2189152

[startup+162.301 s]
/proc/loadavg: 2.02 2.04 2.19 4/230 7486
/proc/meminfo: memFree=28724668/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=511764 CPUtime=0.78 cores=1,3,5,7

################
# More data... #
################

[pid=7497/tid=7521] ppid=7443 vsize=663984 CPUtime=0 cores=1,3,5,7
/proc/7497/task/7521/stat : 7521 (python) S 7443 7443 2614 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338542832 679919616 7079 33554432000 4194304 5683644 140733627799264 140370191045168 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7586] ppid=7443 vsize=739776 CPUtime=0.02 cores=1,3,5,7
/proc/7586/stat : 7586 (python) S 7443 7443 2614 0 -1 4202560 996 0 0 0 1 1 0 0 20 0 1 0 338561092 757530624 7059 33554432000 4194304 5683644 140733627799264 140733627790352 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/7586/statm: 184944 7059 359 364 0 120149 0
[pid=7588] ppid=7586 vsize=56156 CPUtime=17.24 cores=1,3,5,7
/proc/7588/stat : 7588 (bsolo_pb10) R 7586 7588 7588 0 -1 4202496 12135 0 0 0 1716 8 0 0 20 0 1 0 338561092 57503744 11618 33554432000 134512640 137138936 4291455552 4291454048 134618214 0 0 16781312 16384 0 0 0 17 3 0 0 0 0 0
/proc/7588/statm: 14039 11618 192 642 0 13395 0
[pid=7443/tid=7450] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 32 1011 0 0 0 0 0 0 20 0 8 0 338504775 835129344 8620 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7443/tid=7458] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7458/stat : 7458 (python) S 7441 7443 2614 0 -1 4202560 30 1011 0 0 0 0 0 0 20 0 8 0 338509795 835129344 8620 33554432000 4194304 5683644 140733627799264 140370426053168 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7443/tid=7463] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7463/stat : 7463 (python) S 7441 7443 2614 0 -1 4202560 25 1011 0 0 0 0 0 0 20 0 8 0 338514511 835129344 8620 33554432000 4194304 5683644 140733627799264 140370415563312 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7443/tid=7489] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7489/stat : 7489 (python) S 7441 7443 2614 0 -1 4202560 20 1011 0 0 0 0 0 0 20 0 8 0 338524051 835129344 8620 33554432000 4194304 5683644 140733627799264 140370201535024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7493] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7493/stat : 7493 (python) S 7441 7443 2614 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 8 0 338528793 835129344 8620 33554432000 4194304 5683644 140733627799264 140370191045168 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7498] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7498/stat : 7498 (python) S 7441 7443 2614 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 8 0 338538076 835129344 8620 33554432000 4194304 5683644 140733627799264 140370180555312 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7587] ppid=7441 vsize=815556 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7587/stat : 7587 (python) S 7441 7443 2614 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 8 0 338561092 835129344 8620 33554432000 4194304 5683644 140733627799264 140370170065456 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 583.95
Current children cumulated vsize (KiB) 15126120



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+612.802 s]
/proc/loadavg: 2.14 2.05 2.12 4/259 7613
/proc/meminfo: memFree=26311048/32873844 swapFree=7116/7116
[pid=7443] ppid=7441 vsize=891348 CPUtime=1.65 cores=1,3,5,7
/proc/7443/stat : 7443 (python) S 7441 7443 2614 0 -1 4202496 18345 1011 1 0 150 15 0 0 20 0 9 0 338504588 912740352 8628 33554432000 4194304 5683644 140733627799264 140733627794096 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 40 0 0
/proc/7443/statm: 222837 8628 1916 364 0 158039 0
[pid=7449] ppid=7443 vsize=360180 CPUtime=0.17 cores=1,3,5,7
/proc/7449/stat : 7449 (python) S 7443 7443 2614 0 -1 4202560 1187 0 0 0 10 7 0 0 20 0 2 0 338504774 368824320 6435 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7449/statm: 90045 6435 398 364 0 25268 0
[pid=7451] ppid=7449 vsize=375944 CPUtime=92.76 cores=1,3,5,7
/proc/7451/stat : 7451 (scip-2.0.1.linu) T 7449 7451 7451 0 -1 4202496 120945 0 0 0 9169 107 0 0 20 0 1 0 338504775 384966656 88279 33554432000 4194304 15243407 140734761315344 140734761309736 6958470 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/7451/statm: 93986 88279 947 2698 0 91281 0
[pid=7449/tid=7456] ppid=7443 vsize=360180 CPUtime=0 cores=1,3,5,7
/proc/7449/task/7456/stat : 7456 (python) S 7443 7443 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338509793 368824320 6435 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7457] ppid=7443 vsize=360192 CPUtime=0.4 cores=1,3,5,7
/proc/7457/stat : 7457 (python) S 7443 7443 2614 0 -1 4202560 1145 0 0 0 23 17 0 0 20 0 2 0 338509795 368836608 6421 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7457/statm: 90048 6421 384 364 0 25268 0
[pid=7459] ppid=7457 vsize=34924 CPUtime=229.6 cores=1,3,5,7
/proc/7459/stat : 7459 (clasp-1.3.7-x86) T 7457 7459 7459 0 -1 4202496 8838 0 0 0 22958 2 0 0 20 0 1 0 338509795 35762176 8471 33554432000 134512640 136139019 4293111712 4293110576 134608323 0 0 16781312 16386 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7459/statm: 8731 8471 182 398 0 8331 0
[pid=7457/tid=7461] ppid=7443 vsize=360192 CPUtime=0 cores=1,3,5,7
/proc/7457/task/7461/stat : 7461 (python) S 7443 7443 2614 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338514509 368836608 6421 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7462] ppid=7443 vsize=435984 CPUtime=0.19 cores=1,3,5,7
/proc/7462/stat : 7462 (python) S 7443 7443 2614 0 -1 4202560 1141 0 0 0 10 9 0 0 20 0 2 0 338514511 446447616 6427 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7462/statm: 108996 6427 384 364 0 44213 0
[pid=7464] ppid=7462 vsize=349516 CPUtime=94.66 cores=1,3,5,7
/proc/7464/stat : 7464 (scip-2.0.1.linu) T 7462 7464 7464 0 -1 4202496 85127 0 0 0 9418 48 0 0 20 0 1 0 338514512 357904384 82810 33554432000 4194304 13684479 140736698496160 140736698488072 7410502 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/7464/statm: 87379 82810 759 2317 0 85057 0
[pid=7462/tid=7486] ppid=7443 vsize=435984 CPUtime=0 cores=1,3,5,7
/proc/7462/task/7486/stat : 7486 (python) S 7443 7443 2614 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338519466 446447616 6427 33554432000 4194304 5683644 140733627799264 140370426053168 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7488] ppid=7443 vsize=511776 CPUtime=0.07 cores=1,3,5,7
/proc/7488/stat : 7488 (python) S 7443 7443 2614 0 -1 4202560 1129 0 0 0 4 3 0 0 20 0 2 0 338524051 524058624 6433 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7488/statm: 127944 6433 384 364 0 63158 0
[pid=7490] ppid=7488 vsize=95216 CPUtime=47.36 cores=1,3,5,7
/proc/7490/stat : 7490 (minion) T 7488 7490 7490 0 -1 4202496 10230 0 0 0 4734 2 0 0 20 0 1 0 338524052 97501184 16497 33554432000 134512640 165565024 4291959552 4291956720 134640912 0 0 16781312 8396802 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7490/statm: 23804 16497 336 7582 0 16219 0
[pid=7488/tid=7491] ppid=7443 vsize=511776 CPUtime=0 cores=1,3,5,7
/proc/7488/task/7491/stat : 7491 (python) S 7443 7443 2614 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338528790 524058624 6433 33554432000 4194304 5683644 140733627799264 140370415563312 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7492] ppid=7443 vsize=587568 CPUtime=0.07 cores=1,3,5,7
/proc/7492/stat : 7492 (python) S 7443 7443 2614 0 -1 4202560 1129 0 0 0 4 3 0 0 20 0 2 0 338528792 601669632 6946 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7492/statm: 146892 6946 384 364 0 82103 0
[pid=7494] ppid=7492 vsize=114376 CPUtime=50.15 cores=1,3,5,7
/proc/7494/stat : 7494 (wbo1.4a) T 7492 7494 7494 0 -1 4202496 28506 0 0 0 4938 77 0 0 20 0 1 0 338528793 117121024 28129 33554432000 134512640 135751879 4292210176 4292208352 134616273 0 0 16781312 3 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7494/statm: 28594 28129 143 303 0 28289 0
[pid=7492/tid=7496] ppid=7443 vsize=587568 CPUtime=0 cores=1,3,5,7
/proc/7492/task/7496/stat : 7496 (python) S 7443 7443 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338533809 601669632 6946 33554432000 4194304 5683644 140733627799264 140370201535024 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7497] ppid=7443 vsize=663984 CPUtime=0.09 cores=1,3,5,7
/proc/7497/stat : 7497 (python) S 7443 7443 2614 0 -1 4202560 1146 0 0 0 6 3 0 0 20 0 2 0 338538076 679919616 7079 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7497/statm: 165996 7079 384 364 0 101204 0
[pid=7499] ppid=7497 vsize=9624972 CPUtime=49.54 cores=1,3,5,7
/proc/7499/stat : 7499 (java) T 7497 7499 7499 0 -1 4202496 16576 0 1 0 4914 40 0 0 20 0 19 0 338538076 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 140736692488192 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/7499/statm: 2406243 427212 2268 9 0 2396937 0
[pid=7499/tid=7500] ppid=7497 vsize=9624972 CPUtime=47.14 cores=1,3,5,7
/proc/7499/task/7500/stat : 7500 (java) T 7497 7499 7499 0 -1 4202560 3668 0 1 0 4687 27 0 0 20 0 19 0 338538078 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967438621728 139967297459492 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7499/tid=7501] ppid=7497 vsize=9624972 CPUtime=0.12 cores=1,3,5,7
/proc/7499/task/7501/stat : 7501 (java) T 7497 7499 7499 0 -1 4202560 53 0 0 0 10 2 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967416724136 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7499/tid=7502] ppid=7497 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/7499/task/7502/stat : 7502 (java) T 7497 7499 7499 0 -1 4202560 49 0 0 0 8 2 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967415671592 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7503] ppid=7497 vsize=9624972 CPUtime=0.06 cores=1,3,5,7
/proc/7499/task/7503/stat : 7503 (java) T 7497 7499 7499 0 -1 4202560 18 0 0 0 6 0 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967212809128 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7499/tid=7504] ppid=7497 vsize=9624972 CPUtime=0.07 cores=1,3,5,7
/proc/7499/task/7504/stat : 7504 (java) T 7497 7499 7499 0 -1 4202560 50 0 0 0 7 0 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967211756584 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7505] ppid=7497 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/7499/task/7505/stat : 7505 (java) T 7497 7499 7499 0 -1 4202560 31 0 0 0 7 2 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967210703016 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7506] ppid=7497 vsize=9624972 CPUtime=0.05 cores=1,3,5,7
/proc/7499/task/7506/stat : 7506 (java) T 7497 7499 7499 0 -1 4202560 45 0 0 0 5 0 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967209650472 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7499/tid=7507] ppid=7497 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/7499/task/7507/stat : 7507 (java) T 7497 7499 7499 0 -1 4202560 41 0 0 0 8 0 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967208597928 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7508] ppid=7497 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/7499/task/7508/stat : 7508 (java) T 7497 7499 7499 0 -1 4202560 57 0 0 0 6 2 0 0 20 0 19 0 338538079 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967207545384 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7509] ppid=7497 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7499/task/7509/stat : 7509 (java) T 7497 7499 7499 0 -1 4202560 738 0 0 0 0 0 0 0 20 0 19 0 338538080 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967195347560 240568547241 0 0 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7510] ppid=7497 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7499/task/7510/stat : 7510 (java) T 7497 7499 7499 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338538081 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967194293880 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7499/tid=7511] ppid=7497 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7499/task/7511/stat : 7511 (java) T 7497 7499 7499 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338538081 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967193241224 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7499/tid=7512] ppid=7497 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7499/task/7512/stat : 7512 (java) T 7497 7499 7499 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338538084 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967192189968 240568555296 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7513] ppid=7497 vsize=9624972 CPUtime=0.79 cores=1,3,5,7
/proc/7499/task/7513/stat : 7513 (java) T 7497 7499 7499 0 -1 4202560 6048 0 0 0 77 2 0 0 20 0 19 0 338538084 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967191136312 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7499/tid=7514] ppid=7497 vsize=9624972 CPUtime=0.68 cores=1,3,5,7
/proc/7499/task/7514/stat : 7514 (java) T 7497 7499 7499 0 -1 4202560 4354 0 0 0 68 0 0 0 20 0 19 0 338538084 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967190083768 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7499/tid=7515] ppid=7497 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7499/task/7515/stat : 7515 (java) T 7497 7499 7499 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338538084 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967189031416 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7499/tid=7516] ppid=7497 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/7499/task/7516/stat : 7516 (java) T 7497 7499 7499 0 -1 4202560 3 0 0 0 1 0 0 0 20 0 19 0 338538084 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967187979080 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7499/tid=7517] ppid=7497 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7499/task/7517/stat : 7517 (java) T 7497 7499 7499 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338538120 9855971328 427212 33554432000 1073741824 1073778376 140736692497056 139967186924840 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7497/tid=7521] ppid=7443 vsize=663984 CPUtime=0 cores=1,3,5,7
/proc/7497/task/7521/stat : 7521 (python) S 7443 7443 2614 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338542832 679919616 7079 33554432000 4194304 5683644 140733627799264 140370191045168 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7586] ppid=7443 vsize=739776 CPUtime=0.08 cores=1,3,5,7
/proc/7586/stat : 7586 (python) S 7443 7443 2614 0 -1 4202560 1178 0 0 0 4 4 0 0 20 0 2 0 338561092 757530624 7085 33554432000 4194304 5683644 140733627799264 140733627790912 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7586/statm: 184944 7085 384 364 0 120149 0
[pid=7588] ppid=7586 vsize=101760 CPUtime=47.62 cores=1,3,5,7
/proc/7588/stat : 7588 (bsolo_pb10) T 7586 7588 7588 0 -1 4202496 23509 0 0 0 4748 14 0 0 20 0 1 0 338561092 104202240 22992 33554432000 134512640 137138936 4291455552 4291454504 136515987 0 0 16781312 16384 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7588/statm: 25440 22992 192 642 0 24796 0
[pid=7586/tid=7610] ppid=7443 vsize=739776 CPUtime=0 cores=1,3,5,7
/proc/7586/task/7610/stat : 7610 (python) S 7443 7443 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338565856 757530624 7085 33554432000 4194304 5683644 140733627799264 140370180555312 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7611] ppid=7443 vsize=815568 CPUtime=0 cores=1,3,5,7
/proc/7611/stat : 7611 (python) S 7443 7443 2614 0 -1 4202560 867 0 0 0 0 0 0 0 20 0 1 0 338565858 835141632 7042 33554432000 4194304 5683644 140733627799264 140733627790352 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 7 0 0 0 0 0
/proc/7611/statm: 203892 7042 336 364 0 139094 0
[pid=7613] ppid=7611 vsize=13420 CPUtime=0.1 cores=1,3,5,7
/proc/7613/stat : 7613 (bsolo_pb10) R 7611 7613 7613 0 -1 4202496 1297 0 0 0 10 0 0 0 20 0 1 0 338565859 13742080 912 33554432000 134512640 137138936 4293134352 4293044108 136570369 0 0 16781312 16384 0 0 0 17 3 0 0 0 0 0
/proc/7613/statm: 3355 912 184 642 0 2711 0
[pid=7443/tid=7450] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7450/stat : 7450 (python) S 7441 7443 2614 0 -1 4202560 32 1011 0 0 0 0 0 0 20 0 9 0 338504775 912740352 8628 33554432000 4194304 5683644 140733627799264 140370436543024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7443/tid=7458] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7458/stat : 7458 (python) S 7441 7443 2614 0 -1 4202560 30 1011 0 0 0 0 0 0 20 0 9 0 338509795 912740352 8628 33554432000 4194304 5683644 140733627799264 140370426053168 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7443/tid=7463] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7463/stat : 7463 (python) S 7441 7443 2614 0 -1 4202560 25 1011 0 0 0 0 0 0 20 0 9 0 338514511 912740352 8628 33554432000 4194304 5683644 140733627799264 140370415563312 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7443/tid=7489] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7489/stat : 7489 (python) S 7441 7443 2614 0 -1 4202560 20 1011 0 0 0 0 0 0 20 0 9 0 338524051 912740352 8628 33554432000 4194304 5683644 140733627799264 140370201535024 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7493] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7493/stat : 7493 (python) S 7441 7443 2614 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 9 0 338528793 912740352 8628 33554432000 4194304 5683644 140733627799264 140370191045168 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7498] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7498/stat : 7498 (python) S 7441 7443 2614 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 9 0 338538076 912740352 8628 33554432000 4194304 5683644 140733627799264 140370180555312 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7587] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7587/stat : 7587 (python) S 7441 7443 2614 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 9 0 338561092 912740352 8628 33554432000 4194304 5683644 140733627799264 140370170065456 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7443/tid=7612] ppid=7441 vsize=891348 CPUtime=0 cores=1,3,5,7
/proc/7443/task/7612/stat : 7612 (python) S 7441 7443 2614 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 9 0 338565858 912740352 8628 33554432000 4194304 5683644 140733627799264 140370159575600 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 614.51
Current children cumulated vsize (KiB) 16076504

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7443 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=512770
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=157975
# CPU time returned by wait4() is 1.67075
# while last known CPU time is 614.51
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 612.865
CPU time (s): 614.51
CPU user time (s): 611
CPU system time (s): 3.51
CPU usage (%): 100.268
Max. virtual memory (cumulated for all children) (KiB): 16076504

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.51277
system time used= 0.157975
maximum resident set size= 34512
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19837
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1736
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2091
involuntary context switches= 14

runsolver used 2.6536 second user time and 6.84196 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 13:43:15
IDJOB=3742174
IDBENCH=90137
IDSOLVER=2327
FILE ID=node146/3742174-1338723795
RUNJOBID= node146-1338713700-2631
PBS_JOBID= 14636739
Free space on /tmp= 70808 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K109.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742174-1338723795/watcher-3742174-1338723795 -o /tmp/evaluation-result-3742174-1338723795/solver-3742174-1338723795 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742174-1338723795.opb -seed 722012330 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3a5151d7fc286e1b50737dee685a4276
RANDOM SEED=722012330

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        30473000 kB
Buffers:          284376 kB
Cached:           613524 kB
SwapCached:         1792 kB
Active:          1269548 kB
Inactive:         303584 kB
Active(anon):     675936 kB
Inactive(anon):     1712 kB
Active(file):     593612 kB
Inactive(file):   301872 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            102868 kB
Writeback:             0 kB
AnonPages:        681424 kB
Mapped:            23796 kB
Shmem:                60 kB
Slab:             683056 kB
SReclaimable:     121348 kB
SUnreclaim:       561708 kB
KernelStack:        1552 kB
PageTables:         8196 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    1022944 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:     28672 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70708 MiB
End job on node146 at 2012-06-03 13:53:32