Trace number 3742083

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 615.45 613.364

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K108.opb
MD5SUMa293d07a49a0e06ee574b1553fbf5b2b
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark105.854
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables7344
Total number of constraints7562
Number of constraints which are clauses7454
Number of constraints which are cardinality constraints (but not clauses)108
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint108
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 109
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.37/1.26	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.37/1.26	c INFO: solving HOME/instance-3742083-1338723195.opb
0.45/1.35	c INFO: parsing took 0.08 s
0.65/1.50	c INFO: features took 0.15 CPU seconds
0.65/1.53	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.67)
51.61/52.43	c INFO: running clasp-1.3.7@0 for 45 with 1749 remaining (b = 0.36)
98.71/99.58	c INFO: running scip-2.0.1-spx@0 for 45 with 1702 remaining (b = 0.06)
145.14/145.97	c INFO: running clasp-1.3.7@47 for 45 with 1656 remaining (b = 0.01)
191.12/191.84	c INFO: wrote minion input file to HOME/tmpeAntza.minion
191.12/191.84	c INFO: running minion-0.12@0 for 45 with 1611 remaining (b = 0.09)
238.56/239.27	c INFO: running wbo1.4a@0 for 45 with 1563 remaining (b = 0.02)
287.73/288.37	c INFO: running scip-2.0.1-clp@49 for 45 with 1514 remaining (b = 0.01)
330.40/330.95	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1472 remaining (b = 0.03)
380.23/378.52	c INFO: running clasp-1.3.7@92 for 136 with 1423 remaining (b = 0.02)
517.87/515.95	c INFO: running scip-2.0.1-spx@45 for 45 with 1285 remaining (b = 0.01)
567.18/565.12	c INFO: running bsolo_pb10-l3@0 for 45 with 1236 remaining (b = 0.02)
614.93/612.84	c INFO: running bsolo_pb10-l1@0 for 91 with 1189 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742083-1338723195/watcher-3742083-1338723195 -o /tmp/evaluation-result-3742083-1338723195/solver-3742083-1338723195 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742083-1338723195.opb -seed 375834651 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.09 3.20 3.00 2/218 6470
/proc/meminfo: memFree=25546144/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=15308 CPUtime=0 cores=0,2,4,6
/proc/6470/stat : 6470 (python) R 6468 6470 3621 0 -1 4202496 650 0 0 0 0 0 0 0 20 0 1 0 338444025 15675392 543 33554432000 4194304 5683644 140735002635392 140735002621304 4539741 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 3827 543 316 364 0 269 0

[startup+0.010525 s]
/proc/loadavg: 2.09 3.20 3.00 2/218 6470
/proc/meminfo: memFree=25546144/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=15380 CPUtime=0 cores=0,2,4,6
/proc/6470/stat : 6470 (python) D 6468 6470 3621 0 -1 4202496 740 0 0 0 0 0 0 0 20 0 1 0 338444025 15749120 617 33554432000 4194304 5683644 140735002635392 140735002593848 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 3845 617 331 364 0 287 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15380

[startup+0.100237 s]
/proc/loadavg: 2.09 3.20 3.00 2/218 6470
/proc/meminfo: memFree=25546144/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=19084 CPUtime=0.02 cores=0,2,4,6
/proc/6470/stat : 6470 (python) R 6468 6470 3621 0 -1 4202496 1393 0 0 0 2 0 0 0 20 0 1 0 338444025 19542016 1147 33554432000 4194304 5683644 140735002635392 140735002565896 4864831 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 4771 1147 430 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19084

[startup+0.300236 s]
/proc/loadavg: 2.09 3.20 3.00 2/218 6470
/proc/meminfo: memFree=25546144/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=96896 CPUtime=0.11 cores=0,2,4,6
/proc/6470/stat : 6470 (python) D 6468 6470 3621 0 -1 4202496 3899 1010 1 0 9 2 0 0 20 0 1 0 338444025 99221504 2647 33554432000 4194304 5683644 140735002635392 140735002553416 213419638453 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 24224 2647 822 364 0 1715 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 96896

[startup+0.700201 s]
/proc/loadavg: 2.09 3.20 3.00 2/218 6470
/proc/meminfo: memFree=25546144/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=122348 CPUtime=0.22 cores=0,2,4,6
/proc/6470/stat : 6470 (python) D 6468 6470 3621 0 -1 4202496 5973 1010 1 0 16 6 0 0 20 0 1 0 338444025 125284352 3849 33554432000 4194304 5683644 140735002635392 140735002550792 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 30587 3849 1015 364 0 2830 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 122348

[startup+1.50018 s]
/proc/loadavg: 2.09 3.20 3.00 3/219 6475
/proc/meminfo: memFree=25527288/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=284252 CPUtime=0.65 cores=0,2,4,6
/proc/6470/stat : 6470 (python) R 6468 6470 3621 0 -1 4202496 11192 1010 1 0 56 9 0 0 20 0 1 0 338444025 291074048 7830 33554432000 4194304 5683644 140735002635392 140735002627288 213427185433 0 0 16781312 33554434 0 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 71063 7830 1847 364 0 6292 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 284252

[startup+3.10038 s]
/proc/loadavg: 2.25 3.22 3.01 3/222 6478
/proc/meminfo: memFree=25431056/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=360232 CPUtime=0.68 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 11521 1010 1 0 59 9 0 0 20 0 2 0 338444025 368877568 7934 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 90058 7934 1901 364 0 25281 0
[pid=6476] ppid=6470 vsize=284452 CPUtime=0 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 765 0 0 0 0 0 0 0 20 0 1 0 338444179 291278848 6368 33554432000 4194304 5683644 140735002635392 140735002626480 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 71113 6368 341 364 0 6336 0
[pid=6478] ppid=6476 vsize=116352 CPUtime=1.55 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) R 6476 6478 6478 0 -1 4202496 25145 0 0 0 150 5 0 0 20 0 1 0 338444179 119144448 24542 33554432000 4194304 15243407 140735469710656 140735469706040 4841856 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 29088 24542 682 2698 0 26383 0
[pid=6470/tid=6477] ppid=6468 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338444179 368877568 7934 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 2.23
Current children cumulated vsize (KiB) 761036
heavy processes:

[startup+6.30044 s]
/proc/loadavg: 2.25 3.22 3.01 3/222 6478
/proc/meminfo: memFree=25365832/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=360232 CPUtime=0.68 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 11521 1010 1 0 59 9 0 0 20 0 2 0 338444025 368877568 7934 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 90058 7934 1901 364 0 25281 0
[pid=6476] ppid=6470 vsize=284452 CPUtime=0 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 891 0 0 0 0 0 0 0 20 0 1 0 338444179 291278848 6391 33554432000 4194304 5683644 140735002635392 140735002626480 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 71113 6391 364 364 0 6336 0
[pid=6478] ppid=6476 vsize=178700 CPUtime=4.75 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) R 6476 6478 6478 0 -1 4202496 40418 0 0 0 464 11 0 0 20 0 1 0 338444179 182988800 39815 33554432000 4194304 15243407 140735469710656 140735469706968 13277498 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 44675 39815 686 2698 0 41970 0
[pid=6470/tid=6477] ppid=6468 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338444179 368877568 7934 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 5.43
Current children cumulated vsize (KiB) 823384

[startup+12.7004 s]
/proc/loadavg: 2.21 3.17 3.00 3/222 6478
/proc/meminfo: memFree=25306932/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=360232 CPUtime=0.68 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 11521 1010 1 0 59 9 0 0 20 0 2 0 338444025 368877568 7934 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 90058 7934 1901 364 0 25281 0
[pid=6476] ppid=6470 vsize=284452 CPUtime=0.01 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 897 0 0 0 1 0 0 0 20 0 1 0 338444179 291278848 6391 33554432000 4194304 5683644 140735002635392 140735002626480 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 71113 6391 364 364 0 6336 0
[pid=6478] ppid=6476 vsize=235888 CPUtime=11.15 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) R 6476 6478 6478 0 -1 4202496 54734 0 0 0 1096 19 0 0 20 0 1 0 338444179 241549312 54131 33554432000 4194304 15243407 140735469710656 140735469706968 13277498 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 58972 54131 686 2698 0 56267 0
[pid=6470/tid=6477] ppid=6468 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338444179 368877568 7934 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 11.84
Current children cumulated vsize (KiB) 880572

[startup+25.5004 s]
/proc/loadavg: 2.17 3.14 2.98 3/225 6481
/proc/meminfo: memFree=25163696/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=360232 CPUtime=0.68 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 11521 1010 1 0 59 9 0 0 20 0 2 0 338444025 368877568 7934 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 90058 7934 1901 364 0 25281 0
[pid=6476] ppid=6470 vsize=284452 CPUtime=0.03 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 997 0 0 0 2 1 0 0 20 0 1 0 338444179 291278848 6396 33554432000 4194304 5683644 140735002635392 140735002626480 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 71113 6396 369 364 0 6336 0
[pid=6478] ppid=6476 vsize=331388 CPUtime=23.95 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) R 6476 6478 6478 0 -1 4202496 78171 0 0 0 2358 37 0 0 20 0 1 0 338444179 339341312 77548 33554432000 4194304 15243407 140735469710656 140735469706744 13277498 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 82847 77548 874 2698 0 80142 0
[pid=6470/tid=6477] ppid=6468 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338444179 368877568 7934 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 24.66
Current children cumulated vsize (KiB) 976072

[startup+51.1003 s]
/proc/loadavg: 2.11 3.04 2.96 3/225 6482
/proc/meminfo: memFree=25072120/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=360232 CPUtime=0.68 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 11521 1010 1 0 59 9 0 0 20 0 2 0 338444025 368877568 7934 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 90058 7934 1901 364 0 25281 0
[pid=6476] ppid=6470 vsize=284452 CPUtime=0.08 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 1015 0 0 0 5 3 0 0 20 0 1 0 338444179 291278848 6396 33554432000 4194304 5683644 140735002635392 140735002626480 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 71113 6396 369 364 0 6336 0
[pid=6478] ppid=6476 vsize=358840 CPUtime=49.54 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) R 6476 6478 6478 0 -1 4202496 84490 0 0 0 4863 91 0 0 20 0 1 0 338444179 367452160 84376 33554432000 4194304 15243407 140735469710656 140735469705320 6953008 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 89710 84376 899 2698 0 87005 0
[pid=6470/tid=6477] ppid=6468 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338444179 368877568 7934 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 50.3
Current children cumulated vsize (KiB) 1003524

[startup+102.301 s]
/proc/loadavg: 2.04 2.87 2.90 3/233 6490
/proc/meminfo: memFree=24858348/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=511816 CPUtime=0.74 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 13293 1010 1 0 64 10 0 0 20 0 4 0 338444025 524099584 7951 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 127954 7951 1906 364 0 63171 0
[pid=6476] ppid=6470 vsize=360232 CPUtime=0.09 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 1152 0 0 0 5 4 0 0 20 0 2 0 338444179 368877568 6431 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 90058 6431 397 364 0 25281 0
[pid=6478] ppid=6476 vsize=359884 CPUtime=50.84 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) T 6476 6478 6478 0 -1 4202496 84521 0 0 0 4993 91 0 0 20 0 1 0 338444179 368521216 84407 33554432000 4194304 15243407 140735469710656 140735469705368 8134397 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 89971 84407 899 2698 0 87266 0
[pid=6476/tid=6483] ppid=6470 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6476/task/6483/stat : 6483 (python) S 6470 6470 3621 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338449266 368877568 6431 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6484] ppid=6470 vsize=360244 CPUtime=0.07 cores=0,2,4,6
/proc/6484/stat : 6484 (python) S 6470 6470 3621 0 -1 4202560 1094 0 0 0 4 3 0 0 20 0 2 0 338449268 368889856 6418 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6484/statm: 90061 6418 384 364 0 25281 0
[pid=6486] ppid=6484 vsize=15900 CPUtime=47.05 cores=0,2,4,6
/proc/6486/stat : 6486 (clasp-1.3.7-x86) T 6484 6486 6486 0 -1 4202496 4097 0 0 0 4705 0 0 0 20 0 1 0 338449268 16281600 3731 33554432000 134512640 136139019 4291993632 4291992464 134832239 0 0 16781312 16386 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/6486/statm: 3975 3731 182 398 0 3575 0
[pid=6484/tid=6487] ppid=6470 vsize=360244 CPUtime=0 cores=0,2,4,6
/proc/6484/task/6487/stat : 6487 (python) S 6470 6470 3621 0 -1 4202560 53 0 0 0 0 0 0 0 20 0 2 0 338453981 368889856 6418 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6488] ppid=6470 vsize=436036 CPUtime=0 cores=0,2,4,6
/proc/6488/stat : 6488 (python) S 6470 6470 3621 0 -1 4202560 801 0 0 0 0 0 0 0 20 0 1 0 338453983 446500864 6375 33554432000 4194304 5683644 140735002635392 140735002626480 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6488/statm: 109009 6375 336 364 0 44226 0
[pid=6490] ppid=6488 vsize=135428 CPUtime=2.7 cores=0,2,4,6
/proc/6490/stat : 6490 (scip-2.0.1.linu) R 6488 6490 6490 0 -1 4202496 29868 0 0 0 263 7 0 0 20 0 1 0 338453984 138678272 29254 33554432000 4194304 13684479 140733348147760 140733348144056 4880923 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6490/statm: 33857 29254 635 2317 0 31535 0
[pid=6470/tid=6477] ppid=6468 vsize=511816 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 4 0 338444179 524099584 7951 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6470/tid=6485] ppid=6468 vsize=511816 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6485/stat : 6485 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 4 0 338449268 524099584 7951 33554432000 4194304 5683644 140735002635392 139851643040304 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6470/tid=6489] ppid=6468 vsize=511816 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6489/stat : 6489 (python) S 6468 6470 3621 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 4 0 338453984 524099584 7951 33554432000 4194304 5683644 140735002635392 139851632550448 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 101.49
Current children cumulated vsize (KiB) 2179540

[startup+162.301 s]
/proc/loadavg: 2.01 2.71 2.84 3/235 6492
/proc/meminfo: memFree=24594240/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=511816 CPUtime=0.76 cores=0,2,4,6

################
# More data... #
################

[pid=6501/tid=6523] ppid=6470 vsize=663976 CPUtime=0 cores=0,2,4,6
/proc/6501/task/6523/stat : 6523 (python) S 6470 6470 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338481875 679911424 6568 33554432000 4194304 5683644 140735002635392 139851611570736 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6585] ppid=6470 vsize=739768 CPUtime=0.02 cores=0,2,4,6
/proc/6585/stat : 6585 (python) S 6470 6470 3621 0 -1 4202560 993 0 0 0 1 1 0 0 20 0 1 0 338500538 757522432 6548 33554432000 4194304 5683644 140735002635392 140735002626480 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6585/statm: 184942 6548 359 364 0 120147 0
[pid=6587] ppid=6585 vsize=57480 CPUtime=17.16 cores=0,2,4,6
/proc/6587/stat : 6587 (bsolo_pb10) R 6585 6587 6587 0 -1 4202496 12451 0 0 0 1706 10 0 0 20 0 1 0 338500538 58859520 11933 33554432000 134512640 137138936 4292124272 4292123200 134542497 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/6587/statm: 14370 11933 192 642 0 13726 0
[pid=6470/tid=6477] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 32 1010 0 0 0 0 0 0 20 0 8 0 338444179 835121152 8617 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6470/tid=6485] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6485/stat : 6485 (python) S 6468 6470 3621 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 8 0 338449268 835121152 8617 33554432000 4194304 5683644 140735002635392 139851643040304 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6470/tid=6489] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6489/stat : 6489 (python) S 6468 6470 3621 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 8 0 338453984 835121152 8617 33554432000 4194304 5683644 140735002635392 139851632550448 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6470/tid=6494] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6494/stat : 6494 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338463209 835121152 8617 33554432000 4194304 5683644 140735002635392 139851622060592 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6470/tid=6498] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6498/stat : 6498 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338467952 835121152 8617 33554432000 4194304 5683644 140735002635392 139851611570736 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6470/tid=6502] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6502/stat : 6502 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338477121 835121152 8617 33554432000 4194304 5683644 140735002635392 139851601080880 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6470/tid=6586] ppid=6468 vsize=815548 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6586/stat : 6586 (python) S 6468 6470 3621 0 -1 4202560 20 1010 0 0 0 0 0 0 20 0 8 0 338500538 835121152 8617 33554432000 4194304 5683644 140735002635392 139851248689712 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 584.38
Current children cumulated vsize (KiB) 15117240



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+613.302 s]
/proc/loadavg: 2.03 2.18 2.52 3/238 6592
/proc/meminfo: memFree=24710748/32873844 swapFree=6608/6608
[pid=6470] ppid=6468 vsize=891340 CPUtime=1.68 cores=0,2,4,6
/proc/6470/stat : 6470 (python) S 6468 6470 3621 0 -1 4202496 18227 1010 1 0 156 12 0 0 20 0 9 0 338444025 912732160 8625 33554432000 4194304 5683644 140735002635392 140735002630224 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6470/statm: 222835 8625 1916 364 0 158037 0
[pid=6476] ppid=6470 vsize=360232 CPUtime=0.17 cores=0,2,4,6
/proc/6476/stat : 6476 (python) S 6470 6470 3621 0 -1 4202560 1188 0 0 0 10 7 0 0 20 0 2 0 338444179 368877568 6432 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6476/statm: 90058 6432 398 364 0 25281 0
[pid=6478] ppid=6476 vsize=370912 CPUtime=93.4 cores=0,2,4,6
/proc/6478/stat : 6478 (scip-2.0.1.linu) T 6476 6478 6478 0 -1 4202496 109526 0 0 0 9236 104 0 0 20 0 1 0 338444179 379813888 87694 33554432000 4194304 15243407 140735469710656 140735469705048 7426922 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6478/statm: 92728 87694 956 2698 0 90023 0
[pid=6476/tid=6483] ppid=6470 vsize=360232 CPUtime=0 cores=0,2,4,6
/proc/6476/task/6483/stat : 6483 (python) S 6470 6470 3621 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338449266 368877568 6432 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6484] ppid=6470 vsize=360244 CPUtime=0.42 cores=0,2,4,6
/proc/6484/stat : 6484 (python) S 6470 6470 3621 0 -1 4202560 1196 0 0 0 24 18 0 0 20 0 2 0 338449268 368889856 6427 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6484/statm: 90061 6427 393 364 0 25281 0
[pid=6486] ppid=6484 vsize=39204 CPUtime=229.58 cores=0,2,4,6
/proc/6486/stat : 6486 (clasp-1.3.7-x86) T 6484 6486 6486 0 -1 4202496 9916 0 0 0 22956 2 0 0 20 0 1 0 338449268 40144896 9550 33554432000 134512640 136139019 4291993632 4291992608 134883787 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6486/statm: 9801 9550 182 398 0 9401 0
[pid=6484/tid=6487] ppid=6470 vsize=360244 CPUtime=0 cores=0,2,4,6
/proc/6484/task/6487/stat : 6487 (python) S 6470 6470 3621 0 -1 4202560 53 0 0 0 0 0 0 0 20 0 2 0 338453981 368889856 6427 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6488] ppid=6470 vsize=436036 CPUtime=0.17 cores=0,2,4,6
/proc/6488/stat : 6488 (python) S 6470 6470 3621 0 -1 4202560 1154 0 0 0 9 8 0 0 20 0 2 0 338453983 446500864 6424 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6488/statm: 109009 6424 384 364 0 44226 0
[pid=6490] ppid=6488 vsize=331260 CPUtime=95.46 cores=0,2,4,6
/proc/6490/stat : 6490 (scip-2.0.1.linu) T 6488 6490 6490 0 -1 4202496 82240 0 0 0 9475 71 0 0 20 0 1 0 338453984 339210240 78356 33554432000 4194304 13684479 140733348147760 140733348143848 6392692 0 0 16781312 2 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/6490/statm: 82815 78356 776 2317 0 80493 0
[pid=6488/tid=6492] ppid=6470 vsize=436036 CPUtime=0 cores=0,2,4,6
/proc/6488/task/6492/stat : 6492 (python) S 6470 6470 3621 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338458620 446500864 6424 33554432000 4194304 5683644 140735002635392 139851643040304 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6493] ppid=6470 vsize=511828 CPUtime=0.08 cores=0,2,4,6
/proc/6493/stat : 6493 (python) S 6470 6470 3621 0 -1 4202560 1130 0 0 0 5 3 0 0 20 0 2 0 338463209 524111872 6430 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6493/statm: 127957 6430 384 364 0 63171 0
[pid=6495] ppid=6493 vsize=94632 CPUtime=47.37 cores=0,2,4,6
/proc/6495/stat : 6495 (minion) T 6493 6495 6495 0 -1 4202496 10094 0 0 0 4735 2 0 0 20 0 1 0 338463210 96903168 16365 33554432000 134512640 165565024 4289951456 4289948704 159737071 0 0 16781312 8396802 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6495/statm: 23658 16365 336 7582 0 16073 0
[pid=6493/tid=6496] ppid=6470 vsize=511828 CPUtime=0 cores=0,2,4,6
/proc/6493/task/6496/stat : 6496 (python) S 6470 6470 3621 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338467950 524111872 6430 33554432000 4194304 5683644 140735002635392 139851632550448 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6497] ppid=6470 vsize=587620 CPUtime=0.08 cores=0,2,4,6
/proc/6497/stat : 6497 (python) S 6470 6470 3621 0 -1 4202560 1133 0 0 0 5 3 0 0 20 0 2 0 338467952 601722880 6436 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6497/statm: 146905 6436 384 364 0 82116 0
[pid=6499] ppid=6497 vsize=123528 CPUtime=49.06 cores=0,2,4,6
/proc/6499/stat : 6499 (wbo1.4a) T 6497 6499 6499 0 -1 4202496 30810 0 0 0 4836 70 0 0 20 0 1 0 338467953 126492672 30433 33554432000 134512640 135751879 4293554528 4293552144 134605036 0 0 16781312 3 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/6499/statm: 30882 30433 143 303 0 30577 0
[pid=6497/tid=6500] ppid=6470 vsize=587620 CPUtime=0 cores=0,2,4,6
/proc/6497/task/6500/stat : 6500 (python) S 6470 6470 3621 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338472860 601722880 6436 33554432000 4194304 5683644 140735002635392 139851622060592 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6501] ppid=6470 vsize=663976 CPUtime=0.09 cores=0,2,4,6
/proc/6501/stat : 6501 (python) S 6470 6470 3621 0 -1 4202560 1149 0 0 0 5 4 0 0 20 0 2 0 338477121 679911424 6568 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6501/statm: 165994 6568 384 364 0 101202 0
[pid=6503] ppid=6501 vsize=9624972 CPUtime=49.67 cores=0,2,4,6
/proc/6503/stat : 6503 (java) T 6501 6503 6503 0 -1 4202496 14798 0 1 0 4934 33 0 0 20 0 19 0 338477121 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140733951725728 213422997565 0 0 0 16800975 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/6503/statm: 2406243 264704 2268 9 0 2396937 0
[pid=6503/tid=6504] ppid=6501 vsize=9624972 CPUtime=47.07 cores=0,2,4,6
/proc/6503/task/6504/stat : 6504 (java) T 6501 6503 6503 0 -1 4202560 2997 0 1 0 4689 18 0 0 20 0 19 0 338477123 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561563786384 140561412285681 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6505] ppid=6501 vsize=9624972 CPUtime=0.13 cores=0,2,4,6
/proc/6503/task/6505/stat : 6505 (java) T 6501 6503 6503 0 -1 4202560 134 0 0 0 11 2 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561536239784 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6503/tid=6506] ppid=6501 vsize=9624972 CPUtime=0.1 cores=0,2,4,6
/proc/6503/task/6506/stat : 6506 (java) T 6501 6503 6503 0 -1 4202560 124 0 0 0 8 2 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561535187240 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6507] ppid=6501 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/6503/task/6507/stat : 6507 (java) T 6501 6503 6503 0 -1 4202560 216 0 0 0 9 3 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561534134696 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6508] ppid=6501 vsize=9624972 CPUtime=0.13 cores=0,2,4,6
/proc/6503/task/6508/stat : 6508 (java) T 6501 6503 6503 0 -1 4202560 228 0 0 0 11 2 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561533082152 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6509] ppid=6501 vsize=9624972 CPUtime=0.1 cores=0,2,4,6
/proc/6503/task/6509/stat : 6509 (java) T 6501 6503 6503 0 -1 4202560 35 0 0 0 10 0 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561532029608 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6510] ppid=6501 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/6503/task/6510/stat : 6510 (java) T 6501 6503 6503 0 -1 4202560 129 0 0 0 9 0 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561530977064 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6511] ppid=6501 vsize=9624972 CPUtime=0.07 cores=0,2,4,6
/proc/6503/task/6511/stat : 6511 (java) T 6501 6503 6503 0 -1 4202560 15 0 0 0 7 0 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561400507304 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6512] ppid=6501 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/6503/task/6512/stat : 6512 (java) T 6501 6503 6503 0 -1 4202560 189 0 0 0 7 2 0 0 20 0 19 0 338477124 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140561399454760 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6503/tid=6513] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6513/stat : 6513 (java) T 6501 6503 6503 0 -1 4202560 436 0 0 0 0 0 0 0 20 0 19 0 338477125 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560916506728 213423011753 0 0 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6514] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6514/stat : 6514 (java) T 6501 6503 6503 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338477125 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560915453048 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6503/tid=6515] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6515/stat : 6515 (java) T 6501 6503 6503 0 -1 4202560 10 0 0 0 0 0 0 0 20 0 19 0 338477125 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560914400392 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6516] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6516/stat : 6516 (java) T 6501 6503 6503 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338477128 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560913349136 213423019808 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6503/tid=6517] ppid=6501 vsize=9624972 CPUtime=0.81 cores=0,2,4,6
/proc/6503/task/6517/stat : 6517 (java) T 6501 6503 6503 0 -1 4202560 4201 0 0 0 81 0 0 0 20 0 19 0 338477128 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560912296504 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6518] ppid=6501 vsize=9624972 CPUtime=0.7 cores=0,2,4,6
/proc/6503/task/6518/stat : 6518 (java) T 6501 6503 6503 0 -1 4202560 4671 0 0 0 70 0 0 0 20 0 19 0 338477128 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560911243960 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6519] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6519/stat : 6519 (java) T 6501 6503 6503 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338477128 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560910191608 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6503/tid=6520] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6520/stat : 6520 (java) T 6501 6503 6503 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338477128 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560909139272 213423011753 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6503/tid=6521] ppid=6501 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6503/task/6521/stat : 6521 (java) T 6501 6503 6503 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338477162 9855971328 264704 33554432000 1073741824 1073778376 140733951734592 140560908084008 213423011753 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6501/tid=6523] ppid=6470 vsize=663976 CPUtime=0 cores=0,2,4,6
/proc/6501/task/6523/stat : 6523 (python) S 6470 6470 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338481875 679911424 6568 33554432000 4194304 5683644 140735002635392 139851611570736 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6585] ppid=6470 vsize=739768 CPUtime=0.08 cores=0,2,4,6
/proc/6585/stat : 6585 (python) S 6470 6470 3621 0 -1 4202560 1165 0 0 0 5 3 0 0 20 0 2 0 338500538 757522432 6574 33554432000 4194304 5683644 140735002635392 140735002627040 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6585/statm: 184942 6574 384 364 0 120147 0
[pid=6587] ppid=6585 vsize=97248 CPUtime=47.68 cores=0,2,4,6
/proc/6587/stat : 6587 (bsolo_pb10) T 6585 6587 6587 0 -1 4202496 22390 0 0 0 4753 15 0 0 20 0 1 0 338500538 99581952 21872 33554432000 134512640 137138936 4292124272 4292123200 134542516 0 0 16781312 16384 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6587/statm: 24312 21872 192 642 0 23668 0
[pid=6585/tid=6589] ppid=6470 vsize=739768 CPUtime=0 cores=0,2,4,6
/proc/6585/task/6589/stat : 6589 (python) S 6470 6470 3621 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338505307 757522432 6574 33554432000 4194304 5683644 140735002635392 139851601080880 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6590] ppid=6470 vsize=815560 CPUtime=0 cores=0,2,4,6
/proc/6590/stat : 6590 (python) S 6470 6470 3621 0 -1 4202560 853 0 0 0 0 0 0 0 20 0 1 0 338505310 835133440 7039 33554432000 4194304 5683644 140735002635392 140735002626480 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 4 0 0 0 0 0
/proc/6590/statm: 203890 7039 336 364 0 139092 0
[pid=6592] ppid=6590 vsize=13476 CPUtime=0.46 cores=0,2,4,6
/proc/6592/stat : 6592 (bsolo_pb10) R 6590 6592 6592 0 -1 4202496 1286 0 0 0 46 0 0 0 20 0 1 0 338505310 13799424 904 33554432000 134512640 137138936 4294468304 4294378876 136570369 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/6592/statm: 3369 904 184 642 0 2725 0
[pid=6470/tid=6477] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6477/stat : 6477 (python) S 6468 6470 3621 0 -1 4202560 32 1010 0 0 0 0 0 0 20 0 9 0 338444179 912732160 8625 33554432000 4194304 5683644 140735002635392 139851653530160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6470/tid=6485] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6485/stat : 6485 (python) S 6468 6470 3621 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 9 0 338449268 912732160 8625 33554432000 4194304 5683644 140735002635392 139851643040304 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6470/tid=6489] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6489/stat : 6489 (python) S 6468 6470 3621 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 9 0 338453984 912732160 8625 33554432000 4194304 5683644 140735002635392 139851632550448 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6470/tid=6494] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6494/stat : 6494 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338463209 912732160 8625 33554432000 4194304 5683644 140735002635392 139851622060592 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6470/tid=6498] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6498/stat : 6498 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338467952 912732160 8625 33554432000 4194304 5683644 140735002635392 139851611570736 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6470/tid=6502] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6502/stat : 6502 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338477121 912732160 8625 33554432000 4194304 5683644 140735002635392 139851601080880 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6470/tid=6586] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6586/stat : 6586 (python) S 6468 6470 3621 0 -1 4202560 20 1010 0 0 0 0 0 0 20 0 9 0 338500538 912732160 8625 33554432000 4194304 5683644 140735002635392 139851248689712 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6470/tid=6591] ppid=6468 vsize=891340 CPUtime=0 cores=0,2,4,6
/proc/6470/task/6591/stat : 6591 (python) S 6468 6470 3621 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338505310 912732160 8625 33554432000 4194304 5683644 140735002635392 139851238199856 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 615.45
Current children cumulated vsize (KiB) 16061836

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6470 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=564762
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=124981
# CPU time returned by wait4() is 1.68974
# while last known CPU time is 615.45
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 613.364
CPU time (s): 615.45
CPU user time (s): 611.9
CPU system time (s): 3.55
CPU usage (%): 100.34
Max. virtual memory (cumulated for all children) (KiB): 16061836

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.56476
system time used= 0.124981
maximum resident set size= 34500
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19237
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1720
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2082
involuntary context switches= 12

runsolver used 2.74958 second user time and 6.37703 second system time

The end

Launcher Data 

Begin job on node140 at 2012-06-03 13:33:10
IDJOB=3742083
IDBENCH=90130
IDSOLVER=2327
FILE ID=node140/3742083-1338723195
RUNJOBID= node140-1338713222-3637
PBS_JOBID= 14636725
Free space on /tmp= 70524 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K108.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742083-1338723195/watcher-3742083-1338723195 -o /tmp/evaluation-result-3742083-1338723195/solver-3742083-1338723195 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742083-1338723195.opb -seed 375834651 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a293d07a49a0e06ee574b1553fbf5b2b
RANDOM SEED=375834651

node140.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.88
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        25546680 kB
Buffers:          479196 kB
Cached:          4124996 kB
SwapCached:         1620 kB
Active:          3432076 kB
Inactive:        2951092 kB
Active(anon):    1769916 kB
Inactive(anon):    11364 kB
Active(file):    1662160 kB
Inactive(file):  2939728 kB
Unevictable:        6608 kB
Mlocked:            6608 kB
SwapTotal:      67108856 kB
SwapFree:       67105264 kB
Dirty:            102048 kB
Writeback:             0 kB
AnonPages:       1784080 kB
Mapped:            34404 kB
Shmem:                92 kB
Slab:             793188 kB
SReclaimable:     233344 kB
SUnreclaim:       559844 kB
KernelStack:        1776 kB
PageTables:        11940 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2587572 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1060864 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70424 MiB
End job on node140 at 2012-06-03 13:43:27