Trace number 3739931

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 1528.07 1524.76

General information on the benchmark

Name/DEC-SMALLINT-LIN/oliveras/j90/
normalized-j9045_1-sat.opb
MD5SUMd4d1ca36c8af6075d3c9b6c16eca4a6b
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables26754
Total number of constraints93164
Number of constraints which are clauses92580
Number of constraints which are cardinality constraints (but not clauses)0
Number of constraints which are nor clauses,nor cardinality constraints584
Minimum length of a constraint1
Maximum length of a constraint90
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 23
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 530
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.31/0.49	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.31/0.49	c INFO: solving HOME/instance-3739931-1338719458.opb
0.91/1.07	c INFO: parsing took 0.56 s
1.80/1.95	c INFO: features took 0.88 CPU seconds
1.80/1.97	c INFO: running clasp-1.3.7@0 for 45 with 1798 remaining (b = 0.54)
53.64/53.77	c INFO: wrote minion input file to HOME/tmpel325F.minion
53.64/53.77	c INFO: running minion-0.12@0 for 45 with 1751 remaining (b = 0.09)
101.30/101.38	c INFO: running scip-2.0.1-spx@0 for 45 with 1698 remaining (b = 0.03)
151.26/151.22	c INFO: running scip-2.0.1-clp@0 for 45 with 1649 remaining (b = 0.02)
201.12/201.04	c INFO: running clasp-1.3.7@47 for 45 with 1600 remaining (b = 0.01)
246.29/246.19	c INFO: running wbo1.4b-fixed@0 for 45 with 1555 remaining (b = 0.12)
293.03/292.85	c INFO: running clasp-1.3.7@92 for 136 with 1508 remaining (b = 0.03)
430.67/430.25	c INFO: running wbo1.4a@0 for 45 with 1371 remaining (b = 0.02)
477.31/476.89	c INFO: running scip-2.0.1-clp@49 for 45 with 1324 remaining (b = 0.01)
520.10/519.57	c INFO: running scip-2.0.1-spx@49 for 45 with 1282 remaining (b = 0.01)
563.16/562.56	c INFO: running scip-2.0.1-spx@92 for 410 with 1239 remaining (b = 0.06)
978.33/977.04	c INFO: running scip-2.0.1-spx@504 for 547 with 826 remaining (b = 0.07)
1525.89/1523.63	c INFO: running sat4j-pb-v20101225@0 for 45 with 281 remaining (b = 0.01)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3739931-1338719458/watcher-3739931-1338719458 -o /tmp/evaluation-result-3739931-1338719458/solver-3739931-1338719458 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3739931-1338719458.opb -seed 1996976953 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.40 2.11 2.05 3/201 6222
/proc/meminfo: memFree=29864868/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=15380 CPUtime=0 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 755 0 0 0 0 0 0 0 20 0 1 0 338070740 15749120 629 33554432000 4194304 5683644 140737222906944 140737222847432 240565175184 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 3845 629 333 364 0 287 0

[startup+0.099966 s]
/proc/loadavg: 2.40 2.11 2.05 3/201 6222
/proc/meminfo: memFree=29864868/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=92388 CPUtime=0.07 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 3220 504 1 0 5 2 0 0 20 0 1 0 338070740 94605312 2540 33554432000 4194304 5683644 140737222906944 140737222859416 5178094 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/6222/statm: 23097 2540 816 364 0 1619 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 92388

[startup+0.10028 s]
/proc/loadavg: 2.40 2.11 2.05 3/201 6222
/proc/meminfo: memFree=29864868/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=92388 CPUtime=0.07 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 3220 504 1 0 5 2 0 0 20 0 1 0 338070740 94605312 2540 33554432000 4194304 5683644 140737222906944 140737222859416 5178094 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/6222/statm: 23097 2540 816 364 0 1619 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 92388

[startup+0.300208 s]
/proc/loadavg: 2.40 2.11 2.05 3/201 6222
/proc/meminfo: memFree=29864868/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=169408 CPUtime=0.23 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 6630 1009 1 0 18 5 0 0 20 0 1 0 338070740 173473792 4415 33554432000 4194304 5683644 140737222906944 140737222829080 4540219 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 42352 4417 1253 364 0 3387 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 169408

[startup+0.700184 s]
/proc/loadavg: 2.40 2.11 2.05 3/201 6222
/proc/meminfo: memFree=29864868/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=290488 CPUtime=0.6 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 11772 1009 1 0 52 8 0 0 20 0 1 0 338070740 297459712 9317 33554432000 4194304 5683644 140737222906944 140737222900472 139640367875297 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 72622 9317 1811 364 0 7851 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 290488

[startup+1.50018 s]
/proc/loadavg: 2.40 2.11 2.05 3/202 6227
/proc/meminfo: memFree=29825620/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=301980 CPUtime=1.4 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 18133 1009 1 0 130 10 0 0 20 0 1 0 338070740 309227520 12248 33554432000 4194304 5683644 140737222906944 140737222893768 139640534609954 0 0 16781312 33554434 0 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 75495 12248 1847 364 0 10724 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 301980

[startup+3.1003 s]
/proc/loadavg: 2.40 2.11 2.05 3/205 6230
/proc/meminfo: memFree=29811772/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=379240 CPUtime=1.88 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 18786 1009 1 0 178 10 0 0 20 0 2 0 338070740 388341760 12673 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 94810 12673 1902 364 0 30033 0
[pid=6228] ppid=6222 vsize=303460 CPUtime=0 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 739 0 0 0 0 0 0 0 20 0 1 0 338070937 310743040 11105 33554432000 4194304 5683644 140737222906944 140737222898032 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 75865 11105 340 364 0 11088 0
[pid=6230] ppid=6228 vsize=9248 CPUtime=1.11 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) R 6228 6230 6230 0 -1 4202496 2890 0 0 0 111 0 0 0 20 0 1 0 338070938 9469952 2048 33554432000 134512640 136139019 4293285232 4293283984 134871026 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 2312 2048 183 398 0 1912 0
[pid=6222/tid=6229] ppid=6220 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338070937 388341760 12673 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 2.99
Current children cumulated vsize (KiB) 691948

[startup+6.30029 s]
/proc/loadavg: 2.45 2.13 2.05 4/204 6230
/proc/meminfo: memFree=29811284/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=379240 CPUtime=1.88 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 18786 1009 1 0 178 10 0 0 20 0 2 0 338070740 388341760 12673 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 94810 12673 1902 364 0 30033 0
[pid=6228] ppid=6222 vsize=303460 CPUtime=0 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 893 0 0 0 0 0 0 0 20 0 1 0 338070937 310743040 11127 33554432000 4194304 5683644 140737222906944 140737222898032 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 75865 11127 362 364 0 11088 0
[pid=6230] ppid=6228 vsize=11624 CPUtime=4.31 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) R 6228 6230 6230 0 -1 4202496 3503 0 0 0 431 0 0 0 20 0 1 0 338070938 11902976 2661 33554432000 134512640 136139019 4293285232 4293284064 134832230 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 2906 2661 183 398 0 2506 0
[pid=6222/tid=6229] ppid=6220 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338070937 388341760 12673 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 6.19
Current children cumulated vsize (KiB) 694324

[startup+12.7003 s]
/proc/loadavg: 2.41 2.13 2.05 3/203 6230
/proc/meminfo: memFree=29808808/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=379240 CPUtime=1.88 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 18786 1009 1 0 178 10 0 0 20 0 2 0 338070740 388341760 12673 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 94810 12673 1902 364 0 30033 0
[pid=6228] ppid=6222 vsize=303460 CPUtime=0 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 897 0 0 0 0 0 0 0 20 0 1 0 338070937 310743040 11127 33554432000 4194304 5683644 140737222906944 140737222898032 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 75865 11127 362 364 0 11088 0
[pid=6230] ppid=6228 vsize=15188 CPUtime=10.71 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) R 6228 6230 6230 0 -1 4202496 4382 0 0 0 1070 1 0 0 20 0 1 0 338070938 15552512 3540 33554432000 134512640 136139019 4293285232 4293284208 134882962 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 3797 3540 183 398 0 3397 0
[pid=6222/tid=6229] ppid=6220 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338070937 388341760 12673 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 12.59
Current children cumulated vsize (KiB) 697888

[startup+25.507 s]
/proc/loadavg: 2.47 2.15 2.06 3/204 6231
/proc/meminfo: memFree=29803244/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=379240 CPUtime=1.88 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 18786 1009 1 0 178 10 0 0 20 0 2 0 338070740 388341760 12673 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 94810 12673 1902 364 0 30033 0
[pid=6228] ppid=6222 vsize=303460 CPUtime=0.03 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 911 0 0 0 2 1 0 0 20 0 1 0 338070937 310743040 11128 33554432000 4194304 5683644 140737222906944 140737222898032 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 75865 11128 363 364 0 11088 0
[pid=6230] ppid=6228 vsize=20600 CPUtime=23.51 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) R 6228 6230 6230 0 -1 4202496 5743 0 0 0 2350 1 0 0 20 0 1 0 338070938 21094400 4901 33554432000 134512640 136139019 4293285232 4293284096 134608333 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 5150 4901 183 398 0 4750 0
[pid=6222/tid=6229] ppid=6220 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338070937 388341760 12673 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 25.42
Current children cumulated vsize (KiB) 703300

[startup+51.1003 s]
/proc/loadavg: 2.31 2.14 2.06 3/205 6232
/proc/meminfo: memFree=29792920/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=379244 CPUtime=3.87 cores=0,2,4,6
/proc/6222/stat : 6222 (python) R 6220 6222 2614 0 -1 4202496 19440 1009 1 0 376 11 0 0 20 0 2 0 338070740 388345856 12675 33554432000 4194304 5683644 140737222906944 140737222901264 4952101 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 94811 12675 1903 364 0 30034 0
[pid=6228] ppid=6222 vsize=379240 CPUtime=0.07 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 1146 0 0 0 4 3 0 0 20 0 2 0 338070937 388341760 11168 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 94810 11168 396 364 0 30033 0
[pid=6230] ppid=6228 vsize=27388 CPUtime=47.11 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) T 6228 6230 6230 0 -1 4202496 7418 0 0 0 4709 2 0 0 20 0 1 0 338070938 28045312 6576 33554432000 134512640 136139019 4293285232 4293284208 134883791 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 6847 6576 183 398 0 6447 0
[pid=6228/tid=6232] ppid=6222 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6228/task/6232/stat : 6232 (python) S 6222 6222 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338075650 388341760 11168 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6222/tid=6229] ppid=6220 vsize=379244 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338070937 388345856 12675 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 51.05
Current children cumulated vsize (KiB) 785872

[startup+102.3 s]
/proc/loadavg: 2.13 2.11 2.06 3/208 6235
/proc/meminfo: memFree=29695120/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=530824 CPUtime=6.57 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 20613 1009 1 0 645 12 0 0 20 0 4 0 338070740 543563776 13198 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 132706 13198 1907 364 0 67923 0
[pid=6228] ppid=6222 vsize=379240 CPUtime=0.07 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 1146 0 0 0 4 3 0 0 20 0 2 0 338070937 388341760 11168 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 94810 11168 396 364 0 30033 0
[pid=6230] ppid=6228 vsize=27388 CPUtime=47.11 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) T 6228 6230 6230 0 -1 4202496 7418 0 0 0 4709 2 0 0 20 0 1 0 338070938 28045312 6576 33554432000 134512640 136139019 4293285232 4293284208 134883791 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 6847 6576 183 398 0 6447 0
[pid=6228/tid=6232] ppid=6222 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6228/task/6232/stat : 6232 (python) S 6222 6222 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338075650 388341760 11168 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6233] ppid=6222 vsize=379252 CPUtime=0.07 cores=0,2,4,6
/proc/6233/stat : 6233 (python) S 6222 6222 2614 0 -1 4202560 1080 0 0 0 4 3 0 0 20 0 2 0 338076117 388354048 11155 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6233/statm: 94813 11155 383 364 0 30033 0
[pid=6235] ppid=6233 vsize=122252 CPUtime=47.56 cores=0,2,4,6
/proc/6235/stat : 6235 (minion) T 6233 6235 6235 0 -1 4202496 19242 0 0 0 4753 3 0 0 20 0 1 0 338076117 125186048 23169 33554432000 134512640 165565024 4288414032 4288411312 142424553 0 0 16781312 8396802 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6235/statm: 30563 23169 345 7582 0 22978 0
[pid=6222/tid=6229] ppid=6220 vsize=530824 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 4 0 338070937 543563776 13198 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6234] ppid=6220 vsize=530824 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6234/stat : 6234 (python) S 6220 6222 2614 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 4 0 338076117 543563776 13198 33554432000 4194304 5683644 140737222906944 139640341975600 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 101.38
Current children cumulated vsize (KiB) 1438956

[startup+162.301 s]
/proc/loadavg: 2.05 2.09 2.05 3/216 6243
/proc/meminfo: memFree=28950324/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=606616 CPUtime=6.59 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 21519 1009 1 0 647 12 0 0 20 0 5 0 338070740 621174784 13206 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 151654 13206 1909 364 0 86868 0
[pid=6228] ppid=6222 vsize=379240 CPUtime=0.07 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 1146 0 0 0 4 3 0 0 20 0 2 0 338070937 388341760 11168 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 94810 11168 396 364 0 30033 0
[pid=6230] ppid=6228 vsize=27388 CPUtime=47.11 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) T 6228 6230 6230 0 -1 4202496 7418 0 0 0 4709 2 0 0 20 0 1 0 338070938 28045312 6576 33554432000 134512640 136139019 4293285232 4293284208 134883791 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 6847 6576 183 398 0 6447 0
[pid=6228/tid=6232] ppid=6222 vsize=379240 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

[pid=6349/tid=6357] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6357/stat : 6357 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760994056872 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6358] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6358/stat : 6358 (java) S 6347 6349 6349 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760993004328 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6359] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6359/stat : 6359 (java) S 6347 6349 6349 0 -1 4202560 73 0 0 0 0 0 0 0 20 0 19 0 338223107 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760862460264 240568547241 0 0 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6360] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6360/stat : 6360 (java) S 6347 6349 6349 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338223108 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760861406584 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6361] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6361/stat : 6361 (java) S 6347 6349 6349 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 19 0 338223108 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760860353416 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6362] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6362/stat : 6362 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223110 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760859302160 240568555296 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6363] ppid=6347 vsize=9624972 CPUtime=0.5 cores=0,2,4,6
/proc/6349/task/6363/stat : 6363 (java) R 6347 6349 6349 0 -1 4202560 3931 0 0 0 50 0 0 0 20 0 19 0 338223110 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760858249608 240568550550 0 4 0 16800975 0 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6364] ppid=6347 vsize=9624972 CPUtime=0.52 cores=0,2,4,6
/proc/6349/task/6364/stat : 6364 (java) R 6347 6349 6349 0 -1 4202560 6825 0 0 0 50 2 0 0 20 0 19 0 338223111 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760857174664 139761040690262 0 4 0 16800975 0 0 0 -1 4 0 0 0 0 0
[pid=6349/tid=6365] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6365/stat : 6365 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223111 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760856144632 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6349/tid=6366] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6366/stat : 6366 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223111 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760855092296 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6367] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6367/stat : 6367 (java) S 6347 6349 6349 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338223201 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139759846417448 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6222/tid=6229] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 35 1009 0 0 0 0 0 0 20 0 8 0 338070937 854007808 13230 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6222/tid=6234] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6234/stat : 6234 (python) S 6220 6222 2614 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338076117 854007808 13230 33554432000 4194304 5683644 140737222906944 139640341975600 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6238] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6238/stat : 6238 (python) S 6220 6222 2614 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338080878 854007808 13230 33554432000 4194304 5683644 140737222906944 139640258421296 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6222/tid=6242] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6242/stat : 6242 (python) S 6220 6222 2614 0 -1 4202560 25 1009 0 0 0 0 0 0 20 0 8 0 338085862 854007808 13230 33554432000 4194304 5683644 140737222906944 139640247931440 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6222/tid=6250] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6250/stat : 6250 (python) S 6220 6222 2614 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 8 0 338095360 854007808 13230 33554432000 4194304 5683644 140737222906944 139640237441584 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6256] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6256/stat : 6256 (python) S 6220 6222 2614 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 8 0 338113765 854007808 13230 33554432000 4194304 5683644 140737222906944 139640226951728 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6348] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6348/stat : 6348 (python) S 6220 6222 2614 0 -1 4202560 23 1009 0 0 0 0 0 0 20 0 8 0 338223103 854007808 13230 33554432000 4194304 5683644 140737222906944 139640216461872 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1528.07
Current children cumulated vsize (KiB) 15939720

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1524.7 s]
/proc/loadavg: 3.09 3.45 2.85 5/247 6367
/proc/meminfo: memFree=15137384/32873844 swapFree=7116/7116
[pid=6222] ppid=6220 vsize=833992 CPUtime=6.78 cores=0,2,4,6
/proc/6222/stat : 6222 (python) S 6220 6222 2614 0 -1 4202496 24729 1009 1 0 665 13 0 0 20 0 8 0 338070740 854007808 13230 33554432000 4194304 5683644 140737222906944 140737222901776 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6222/statm: 208498 13230 1915 364 0 143703 0
[pid=6228] ppid=6222 vsize=379240 CPUtime=0.4 cores=0,2,4,6
/proc/6228/stat : 6228 (python) S 6222 6222 2614 0 -1 4202560 1210 0 0 0 23 17 0 0 20 0 2 0 338070937 388341760 11169 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6228/statm: 94810 11169 397 364 0 30033 0
[pid=6230] ppid=6228 vsize=41496 CPUtime=229.6 cores=0,2,4,6
/proc/6230/stat : 6230 (clasp-1.3.7-x86) T 6228 6230 6230 0 -1 4202496 10921 0 0 0 22957 3 0 0 20 0 1 0 338070938 42491904 10133 33554432000 134512640 136139019 4293285232 4293284208 134883744 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6230/statm: 10374 10133 183 398 0 9974 0
[pid=6228/tid=6232] ppid=6222 vsize=379240 CPUtime=0 cores=0,2,4,6
/proc/6228/task/6232/stat : 6232 (python) S 6222 6222 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338075650 388341760 11169 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6233] ppid=6222 vsize=379252 CPUtime=0.07 cores=0,2,4,6
/proc/6233/stat : 6233 (python) S 6222 6222 2614 0 -1 4202560 1080 0 0 0 4 3 0 0 20 0 2 0 338076117 388354048 11155 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6233/statm: 94813 11155 383 364 0 30033 0
[pid=6235] ppid=6233 vsize=122252 CPUtime=47.56 cores=0,2,4,6
/proc/6235/stat : 6235 (minion) T 6233 6235 6235 0 -1 4202496 19242 0 0 0 4753 3 0 0 20 0 1 0 338076117 125186048 23245 33554432000 134512640 165565024 4288414032 4288411312 142424553 0 0 16781312 8396802 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6235/statm: 30563 23245 345 7582 0 22978 0
[pid=6233/tid=6236] ppid=6222 vsize=379252 CPUtime=0 cores=0,2,4,6
/proc/6233/task/6236/stat : 6236 (python) S 6222 6222 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338080875 388354048 11155 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6237] ppid=6222 vsize=455044 CPUtime=1.94 cores=0,2,4,6
/proc/6237/stat : 6237 (python) S 6222 6222 2614 0 -1 4202560 1183 0 0 0 111 83 0 0 20 0 2 0 338080878 465965056 11161 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6237/statm: 113761 11161 383 364 0 48978 0
[pid=6239] ppid=6237 vsize=828040 CPUtime=1053.64 cores=0,2,4,6
/proc/6239/stat : 6239 (scip-2.0.1.linu) T 6237 6239 6239 0 -1 4202496 202995 0 0 0 105025 339 0 0 20 0 1 0 338080878 847912960 204767 33554432000 4194304 13684479 140735667897744 140735667891496 7391338 0 0 16781312 2 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/6239/statm: 207010 204767 805 2317 0 204688 0
[pid=6237/tid=6240] ppid=6222 vsize=455044 CPUtime=0 cores=0,2,4,6
/proc/6237/task/6240/stat : 6240 (python) S 6222 6222 2614 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338085859 465965056 11161 33554432000 4194304 5683644 140737222906944 139640341975600 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6241] ppid=6222 vsize=530836 CPUtime=0.16 cores=0,2,4,6
/proc/6241/stat : 6241 (python) S 6222 6222 2614 0 -1 4202560 1160 0 0 0 10 6 0 0 20 0 2 0 338085862 543576064 11674 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6241/statm: 132709 11674 383 364 0 67923 0
[pid=6243] ppid=6241 vsize=573212 CPUtime=92.42 cores=0,2,4,6
/proc/6243/stat : 6243 (scip-2.0.1.linu) T 6241 6243 6243 0 -1 4202496 139907 0 0 0 9184 58 0 0 20 0 1 0 338085862 586969088 138926 33554432000 4194304 15243407 140736914138416 140736914134712 5688624 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6243/statm: 143303 138926 706 2698 0 140598 0
[pid=6241/tid=6247] ppid=6222 vsize=530836 CPUtime=0 cores=0,2,4,6
/proc/6241/task/6247/stat : 6247 (python) S 6222 6222 2614 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338090841 543576064 11674 33554432000 4194304 5683644 140737222906944 139640258421296 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6249] ppid=6222 vsize=606628 CPUtime=0.07 cores=0,2,4,6
/proc/6249/stat : 6249 (python) S 6222 6222 2614 0 -1 4202560 1119 0 0 0 4 3 0 0 20 0 2 0 338095360 621187072 11681 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6249/statm: 151657 11681 383 364 0 86868 0
[pid=6251] ppid=6249 vsize=65684 CPUtime=46.6 cores=0,2,4,6
/proc/6251/stat : 6251 (wbo1.4b-fixed) T 6249 6251 6251 0 -1 4202496 16912 0 0 0 4643 17 0 0 20 0 1 0 338095360 67260416 15813 33554432000 134512640 135752315 4294604960 4294602960 134611201 0 0 16781312 3 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6251/statm: 16421 15813 146 303 0 16116 0
[pid=6249/tid=6253] ppid=6222 vsize=606628 CPUtime=0 cores=0,2,4,6
/proc/6249/task/6253/stat : 6253 (python) S 6222 6222 2614 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338100022 621187072 11681 33554432000 4194304 5683644 140737222906944 139640247931440 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6255] ppid=6222 vsize=682420 CPUtime=0.07 cores=0,2,4,6
/proc/6255/stat : 6255 (python) S 6222 6222 2614 0 -1 4202560 1124 0 0 0 4 3 0 0 20 0 2 0 338113765 698798080 11687 33554432000 4194304 5683644 140737222906944 140737222898592 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6255/statm: 170605 11687 383 364 0 105813 0
[pid=6257] ppid=6255 vsize=58440 CPUtime=46.6 cores=0,2,4,6
/proc/6257/stat : 6257 (wbo1.4a) T 6255 6257 6257 0 -1 4202496 15074 0 0 0 4641 19 0 0 20 0 1 0 338113765 59842560 13971 33554432000 134512640 135751879 4286938464 4286936640 134616414 0 0 16781312 3 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6257/statm: 14610 13971 143 303 0 14305 0
[pid=6255/tid=6258] ppid=6222 vsize=682420 CPUtime=0 cores=0,2,4,6
/proc/6255/task/6258/stat : 6258 (python) S 6222 6222 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338118427 698798080 11687 33554432000 4194304 5683644 140737222906944 139640237441584 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6347] ppid=6222 vsize=758212 CPUtime=0 cores=0,2,4,6
/proc/6347/stat : 6347 (python) S 6222 6222 2614 0 -1 4202560 844 0 0 0 0 0 0 0 20 0 1 0 338223103 776409088 11644 33554432000 4194304 5683644 140737222906944 140737222898032 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 6 0 0 0 0 0
/proc/6347/statm: 189553 11644 335 364 0 124758 0
[pid=6349] ppid=6347 vsize=9624972 CPUtime=2.16 cores=0,2,4,6
/proc/6349/stat : 6349 (java) S 6347 6349 6349 0 -1 4202496 14779 0 1 0 210 6 0 0 20 0 19 0 338223103 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 140734949597056 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6349/statm: 2406243 44447 2258 9 0 2396937 0
[pid=6349/tid=6350] ppid=6347 vsize=9624972 CPUtime=1.01 cores=0,2,4,6
/proc/6349/task/6350/stat : 6350 (java) R 6347 6349 6349 0 -1 4202560 2504 0 1 0 98 3 0 0 20 0 19 0 338223105 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139761038873080 139760869958695 0 4 0 16800975 0 0 0 -1 0 0 0 0 0 0
[pid=6349/tid=6351] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6351/stat : 6351 (java) S 6347 6349 6349 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139761000372648 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6352] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6352/stat : 6352 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760999320104 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6349/tid=6353] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6353/stat : 6353 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760998267048 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6349/tid=6354] ppid=6347 vsize=9624972 CPUtime=0.01 cores=0,2,4,6
/proc/6349/task/6354/stat : 6354 (java) S 6347 6349 6349 0 -1 4202560 11 0 0 0 1 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760997214504 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6349/tid=6355] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6355/stat : 6355 (java) S 6347 6349 6349 0 -1 4202560 4 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760996162472 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6349/tid=6356] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6356/stat : 6356 (java) S 6347 6349 6349 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760995109928 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6349/tid=6357] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6357/stat : 6357 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760994056872 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6358] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6358/stat : 6358 (java) S 6347 6349 6349 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338223106 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760993004328 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6359] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6359/stat : 6359 (java) S 6347 6349 6349 0 -1 4202560 73 0 0 0 0 0 0 0 20 0 19 0 338223107 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760862460264 240568547241 0 0 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6360] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6360/stat : 6360 (java) S 6347 6349 6349 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338223108 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760861406584 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6361] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6361/stat : 6361 (java) S 6347 6349 6349 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 19 0 338223108 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760860353416 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6362] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6362/stat : 6362 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223110 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760859302160 240568555296 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6363] ppid=6347 vsize=9624972 CPUtime=0.5 cores=0,2,4,6
/proc/6349/task/6363/stat : 6363 (java) R 6347 6349 6349 0 -1 4202560 3931 0 0 0 50 0 0 0 20 0 19 0 338223110 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760858249608 240568550550 0 4 0 16800975 0 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6364] ppid=6347 vsize=9624972 CPUtime=0.52 cores=0,2,4,6
/proc/6349/task/6364/stat : 6364 (java) R 6347 6349 6349 0 -1 4202560 6825 0 0 0 50 2 0 0 20 0 19 0 338223111 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760857174664 139761040690262 0 4 0 16800975 0 0 0 -1 4 0 0 0 0 0
[pid=6349/tid=6365] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6365/stat : 6365 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223111 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760856144632 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6349/tid=6366] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6366/stat : 6366 (java) S 6347 6349 6349 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338223111 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139760855092296 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6349/tid=6367] ppid=6347 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6349/task/6367/stat : 6367 (java) S 6347 6349 6349 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338223201 9855971328 44447 33554432000 1073741824 1073778376 140734949605920 139759846417448 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6222/tid=6229] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6229/stat : 6229 (python) S 6220 6222 2614 0 -1 4202560 35 1009 0 0 0 0 0 0 20 0 8 0 338070937 854007808 13230 33554432000 4194304 5683644 140737222906944 139640361243184 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6222/tid=6234] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6234/stat : 6234 (python) S 6220 6222 2614 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338076117 854007808 13230 33554432000 4194304 5683644 140737222906944 139640341975600 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6238] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6238/stat : 6238 (python) S 6220 6222 2614 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338080878 854007808 13230 33554432000 4194304 5683644 140737222906944 139640258421296 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6222/tid=6242] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6242/stat : 6242 (python) S 6220 6222 2614 0 -1 4202560 25 1009 0 0 0 0 0 0 20 0 8 0 338085862 854007808 13230 33554432000 4194304 5683644 140737222906944 139640247931440 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6222/tid=6250] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6250/stat : 6250 (python) S 6220 6222 2614 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 8 0 338095360 854007808 13230 33554432000 4194304 5683644 140737222906944 139640237441584 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6256] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6256/stat : 6256 (python) S 6220 6222 2614 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 8 0 338113765 854007808 13230 33554432000 4194304 5683644 140737222906944 139640226951728 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6222/tid=6348] ppid=6220 vsize=833992 CPUtime=0 cores=0,2,4,6
/proc/6222/task/6348/stat : 6348 (python) S 6220 6222 2614 0 -1 4202560 23 1009 0 0 0 0 0 0 20 0 8 0 338223103 854007808 13230 33554432000 4194304 5683644 140737222906944 139640216461872 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1528.07
Current children cumulated vsize (KiB) 15939720

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6222 and gives
#  childrusage.ru_utime.tv_sec=6
#  childrusage.ru_utime.tv_usec=660987
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=139978
# CPU time returned by wait4() is 6.80096
# while last known CPU time is 1528.07
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1524.76
CPU time (s): 1528.07
CPU user time (s): 1522.34
CPU system time (s): 5.73
CPU usage (%): 100.217
Max. virtual memory (cumulated for all children) (KiB): 15939720

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 6.66099
system time used= 0.139978
maximum resident set size= 52920
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 25738
page faults= 1
swaps= 0
block input operations= 0
block output operations= 9944
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1040
involuntary context switches= 33

runsolver used 5.75012 second user time and 16.2245 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 12:30:58
IDJOB=3739931
IDBENCH=75548
IDSOLVER=2327
FILE ID=node146/3739931-1338719458
RUNJOBID= node146-1338713700-2630
PBS_JOBID= 14636739
Free space on /tmp= 70800 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB10/normalized-PB10/DEC-SMALLINT-LIN/oliveras/j90/normalized-j9045_1-sat.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3739931-1338719458/watcher-3739931-1338719458 -o /tmp/evaluation-result-3739931-1338719458/solver-3739931-1338719458 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3739931-1338719458.opb -seed 1996976953 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d4d1ca36c8af6075d3c9b6c16eca4a6b
RANDOM SEED=1996976953

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        29865272 kB
Buffers:          230064 kB
Cached:           612504 kB
SwapCached:         1792 kB
Active:          1924680 kB
Inactive:         291548 kB
Active(anon):    1374456 kB
Inactive(anon):     1748 kB
Active(file):     550224 kB
Inactive(file):   289800 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            107728 kB
Writeback:             0 kB
AnonPages:       1379680 kB
Mapped:            24880 kB
Shmem:                84 kB
Slab:             644432 kB
SReclaimable:      81636 kB
SUnreclaim:       562796 kB
KernelStack:        1632 kB
PageTables:        11352 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2016600 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:     38912 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 62580 MiB
End job on node146 at 2012-06-03 12:56:25