Trace number 3317460

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 3186.76 1300.06

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v600-c2556-S1677156340-036.UNKNOWN.cnf
MD5SUM99f3efd3bfc8a6f99ffbd289ed4a5837
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses2556
Sum of the clauses size7668
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32556
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.47	c INFO: solving HOME/instance-3317460-1304269765.cnf
0.31/0.47	c INFO: computing task features
0.31/0.48	c INFO: computed variable-clause graph statistics
0.31/0.48	c INFO: computed clause balance statistics
0.40/0.52	c INFO: computed variable balance statistics
0.40/0.56	c INFO: computed small-clause counts
0.40/0.56	c INFO: computed Horn-clause counts
0.40/0.56	c INFO: computed variable graph statistics
0.40/0.56	c INFO: collected features for HOME/instance-3317460-1304269765.cnf in 0.09 s
0.40/0.59	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.58)
0.50/0.62	c INFO: running march_hi@0 for 194 with 4799 remaining (b = 0.17)
0.50/0.66	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.07)
0.50/0.69	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.27)
485.12/195.15	c INFO: running march_hi@0 for 485 with 4605 remaining (b = 0.28)
488.71/196.96	c INFO: running march_hi@0 for 485 with 4603 remaining (b = 0.25)
924.49/391.43	c INFO: running march_hi@0 for 291 with 4409 remaining (b = 0.16)
1277.55/534.50	c INFO: running march_hi@97 for 97 with 4265 remaining (b = 0.06)
1772.72/732.77	c INFO: running march_hi@196 for 194 with 4067 remaining (b = 0.13)
1919.02/791.41	c INFO: running march_hi@389 for 97 with 4009 remaining (b = 0.08)
2163.85/889.63	c INFO: running march_hi@486 for 97 with 3910 remaining (b = 0.06)
2410.36/988.55	c INFO: running march_hi@583 for 97 with 3811 remaining (b = 0.07)
2463.80/1009.98	c INFO: running march_hi@98 for 97 with 3790 remaining (b = 0.06)
2639.76/1080.53	c INFO: running march_hi@194 for 194 with 3719 remaining (b = 0.13)
2683.37/1098.05	c INFO: running march_hi@0 for 485 with 3702 remaining (b = 0.22)
2739.96/1120.75	c INFO: running march_hi@489 for 97 with 3679 remaining (b = 0.05)
2942.57/1202.04	c INFO: running march_hi@584 for 97 with 3598 remaining (b = 0.05)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317460-1304269765/watcher-3317460-1304269765 -o /tmp/evaluation-result-3317460-1304269765/solver-3317460-1304269765 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317460-1304269765.cnf -seed 1529422657 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.57 8.26 7.58 9/259 7285
/proc/meminfo: memFree=17179272/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=18880 CPUtime=0.01 cores=1,3,5,7
/proc/7285/stat : 7285 (python) R 7283 7285 6070 0 -1 4202496 1249 0 0 0 1 0 0 0 25 0 1 0 538579114 19333120 1104 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634513408 0 2147483391 16781312 2 0 0 0 17 1 0 0 0
/proc/7285/statm: 4720 1104 421 364 0 689 0

[startup+0.124484 s]
/proc/loadavg: 8.57 8.26 7.58 9/259 7285
/proc/meminfo: memFree=17179272/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=111020 CPUtime=0.07 cores=1,3,5,7
/proc/7285/stat : 7285 (python) R 7283 7285 6070 0 -1 4202496 3488 0 0 0 5 2 0 0 25 0 1 0 538579114 113684480 3137 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634513408 0 2147483391 16781312 2 0 0 0 17 7 0 0 0
/proc/7285/statm: 27755 3137 881 364 0 2291 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 111020

[startup+0.200465 s]
/proc/loadavg: 8.57 8.26 7.58 9/259 7285
/proc/meminfo: memFree=17179272/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=126796 CPUtime=0.15 cores=1,3,5,7
/proc/7285/stat : 7285 (python) R 7283 7285 6070 0 -1 4202496 5469 935 1 0 11 4 0 0 25 0 1 0 538579114 129839104 4005 33554432000 4194304 5683644 140736164929056 18446744073709551615 4510807 0 0 16781312 2 0 0 0 17 7 0 0 0
/proc/7285/statm: 31699 4005 1008 364 0 2994 0
Current children cumulated CPU time (s) 0.15
Current children cumulated vsize (KiB) 126796

[startup+0.300449 s]
/proc/loadavg: 8.57 8.26 7.58 9/259 7285
/proc/meminfo: memFree=17179272/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=163648 CPUtime=0.22 cores=1,3,5,7
/proc/7285/stat : 7285 (python) R 7283 7285 6070 0 -1 4202496 6648 935 1 0 15 7 0 0 25 0 1 0 538579114 167575552 4750 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634176039 0 0 16781312 2 0 0 0 17 7 0 0 0
/proc/7285/statm: 40912 4750 1306 364 0 3453 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 163648

[startup+0.700377 s]
/proc/loadavg: 8.57 8.26 7.58 9/259 7285
/proc/meminfo: memFree=17179272/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=320300 CPUtime=0.58 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 12347 935 1 0 48 10 0 0 25 0 5 0 538579114 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 80075 7272 1881 364 0 15984 0
Current children cumulated CPU time (s) 0.58
Current children cumulated vsize (KiB) 320300

[startup+1.50123 s]
/proc/loadavg: 8.57 8.26 7.58 12/272 7302
/proc/meminfo: memFree=17140976/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=320300 CPUtime=0.58 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 12347 935 1 0 48 10 0 0 25 0 5 0 538579114 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 80075 7272 1881 364 0 15984 0
[pid=7291] ppid=7285 vsize=278652 CPUtime=0 cores=1,3,5,7
/proc/7291/stat : 7291 (python) S 7285 7285 6070 0 -1 4202560 733 0 0 0 0 0 0 0 25 0 1 0 538579174 285339648 5570 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535439 0 0 16781312 514 386984666562822143 0 0 17 7 0 0 0
/proc/7291/statm: 69663 5570 354 364 0 5581 0
[pid=7293] ppid=7291 vsize=1017932 CPUtime=0.42 cores=1,3,5,7
/proc/7293/stat : 7293 (gnovelty+2) R 7291 7293 7293 0 -1 4202496 445 0 0 0 42 0 0 0 25 0 1 0 538579176 1042362368 185 33554432000 134512640 135663704 4287087840 18446744073709551615 134525187 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/7293/statm: 254483 185 113 282 0 254199 0
[pid=7294] ppid=7285 vsize=289376 CPUtime=0 cores=1,3,5,7
/proc/7294/stat : 7294 (python) S 7285 7285 6070 0 -1 4202560 763 0 0 0 0 0 0 0 17 0 1 0 538579177 296321024 5702 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 4103465360085188872 0 0 17 7 0 0 0
/proc/7294/statm: 72344 5702 348 364 0 8259 0
[pid=7296] ppid=7294 vsize=2496 CPUtime=0.39 cores=1,3,5,7
/proc/7296/stat : 7296 (march_hi) R 7294 7296 7296 0 -1 4202496 675 0 0 0 39 0 0 0 25 0 1 0 538579179 2555904 412 33554432000 4194304 4955518 140736515668048 18446744073709551615 4235344 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7296/statm: 624 412 104 186 0 437 0
[pid=7297] ppid=7285 vsize=299260 CPUtime=0 cores=1,3,5,7
/proc/7297/stat : 7297 (python) S 7285 7285 6070 0 -1 4202560 743 0 0 0 0 0 0 0 25 0 1 0 538579180 306442240 5610 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 5641832760436850687 0 0 17 7 0 0 0
/proc/7297/statm: 74815 5610 348 364 0 10727 0
[pid=7299] ppid=7297 vsize=1017932 CPUtime=0.37 cores=1,3,5,7
/proc/7299/stat : 7299 (gnovelty+2) R 7297 7299 7299 0 -1 4202496 448 0 0 0 37 0 0 0 25 0 1 0 538579182 1042362368 185 33554432000 134512640 135663704 4289840032 18446744073709551615 134545646 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/7299/statm: 254483 185 113 282 0 254199 0
[pid=7300] ppid=7285 vsize=310056 CPUtime=0 cores=1,3,5,7
/proc/7300/stat : 7300 (python) S 7285 7285 6070 0 -1 4202560 765 0 0 0 0 0 0 0 22 0 1 0 538579183 317497344 5735 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 17793774908593897734 0 0 17 7 0 0 0
/proc/7300/statm: 77514 5735 348 364 0 13423 0
[pid=7302] ppid=7300 vsize=2496 CPUtime=0.79 cores=1,3,5,7
/proc/7302/stat : 7302 (march_hi) R 7300 7302 7302 0 -1 4202496 697 0 0 0 79 0 0 0 25 0 1 0 538579183 2555904 436 33554432000 4194304 4955518 140736230621072 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/7302/statm: 624 436 104 186 0 437 0
[pid=7285/tid=7292] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7292/stat : 7292 (python) S 7283 7285 6070 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538579174 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7295] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7295/stat : 7295 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 5 0 538579177 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7298] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7298/stat : 7298 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 5 0 538579180 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7301] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7301/stat : 7301 (python) S 7283 7285 6070 0 -1 4202560 28 935 0 0 0 0 0 0 25 0 5 0 538579183 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2.55
Current children cumulated vsize (KiB) 3538500

[startup+3.10095 s]
/proc/loadavg: 8.57 8.26 7.58 12/272 7302
/proc/meminfo: memFree=17139240/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=320300 CPUtime=0.58 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 12347 935 1 0 48 10 0 0 25 0 5 0 538579114 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 80075 7272 1881 364 0 15984 0
[pid=7291] ppid=7285 vsize=278652 CPUtime=0 cores=1,3,5,7
/proc/7291/stat : 7291 (python) S 7285 7285 6070 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 538579174 285339648 5570 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535439 0 0 16781312 514 386984666562789640 0 0 17 7 0 0 0
/proc/7291/statm: 69663 5570 354 364 0 5581 0
[pid=7293] ppid=7291 vsize=1017932 CPUtime=1.22 cores=1,3,5,7
/proc/7293/stat : 7293 (gnovelty+2) R 7291 7293 7293 0 -1 4202496 445 0 0 0 122 0 0 0 25 0 1 0 538579176 1042362368 185 33554432000 134512640 135663704 4287087840 18446744073709551615 134524843 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/7293/statm: 254483 185 113 282 0 254199 0
[pid=7294] ppid=7285 vsize=289376 CPUtime=0 cores=1,3,5,7
/proc/7294/stat : 7294 (python) S 7285 7285 6070 0 -1 4202560 763 0 0 0 0 0 0 0 15 0 1 0 538579177 296321024 5702 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 4103465360085188872 0 0 17 7 0 0 0
/proc/7294/statm: 72344 5702 348 364 0 8259 0
[pid=7296] ppid=7294 vsize=2700 CPUtime=1.19 cores=1,3,5,7
/proc/7296/stat : 7296 (march_hi) R 7294 7296 7296 0 -1 4202496 747 0 0 0 119 0 0 0 25 0 1 0 538579179 2764800 484 33554432000 4194304 4955518 140736515668048 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7296/statm: 675 484 104 186 0 488 0
[pid=7297] ppid=7285 vsize=299260 CPUtime=0 cores=1,3,5,7
/proc/7297/stat : 7297 (python) S 7285 7285 6070 0 -1 4202560 743 0 0 0 0 0 0 0 18 0 1 0 538579180 306442240 5610 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 5641832760436818181 0 0 17 7 0 0 0
/proc/7297/statm: 74815 5610 348 364 0 10727 0
[pid=7299] ppid=7297 vsize=1017932 CPUtime=1.17 cores=1,3,5,7
/proc/7299/stat : 7299 (gnovelty+2) R 7297 7299 7299 0 -1 4202496 448 0 0 0 117 0 0 0 25 0 1 0 538579182 1042362368 185 33554432000 134512640 135663704 4289840032 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/7299/statm: 254483 185 113 282 0 254199 0
[pid=7300] ppid=7285 vsize=310056 CPUtime=0 cores=1,3,5,7
/proc/7300/stat : 7300 (python) S 7285 7285 6070 0 -1 4202560 766 0 0 0 0 0 0 0 15 0 1 0 538579183 317497344 5735 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 17793774908593897734 0 0 17 7 0 0 0
/proc/7300/statm: 77514 5735 348 364 0 13423 0
[pid=7302] ppid=7300 vsize=2952 CPUtime=2.38 cores=1,3,5,7
/proc/7302/stat : 7302 (march_hi) R 7300 7302 7302 0 -1 4202496 790 0 0 0 238 0 0 0 25 0 1 0 538579183 3022848 529 33554432000 4194304 4955518 140736230621072 18446744073709551615 4238928 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/7302/statm: 738 529 104 186 0 551 0
[pid=7285/tid=7292] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7292/stat : 7292 (python) S 7283 7285 6070 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538579174 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7295] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7295/stat : 7295 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 5 0 538579177 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7298] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7298/stat : 7298 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 5 0 538579180 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7301] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7301/stat : 7301 (python) S 7283 7285 6070 0 -1 4202560 28 935 0 0 0 0 0 0 25 0 5 0 538579183 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.54
Current children cumulated vsize (KiB) 3539160

[startup+6.30139 s]
/proc/loadavg: 8.77 8.30 7.60 12/272 7302
/proc/meminfo: memFree=17134856/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=320300 CPUtime=0.58 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 12347 935 1 0 48 10 0 0 25 0 5 0 538579114 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 80075 7272 1881 364 0 15984 0
[pid=7291] ppid=7285 vsize=278652 CPUtime=0 cores=1,3,5,7
/proc/7291/stat : 7291 (python) S 7285 7285 6070 0 -1 4202560 890 0 0 0 0 0 0 0 18 0 1 0 538579174 285339648 5593 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535439 0 0 16781312 514 386984666562789640 0 0 17 7 0 0 0
/proc/7291/statm: 69663 5593 377 364 0 5581 0
[pid=7293] ppid=7291 vsize=1017932 CPUtime=2.82 cores=1,3,5,7
/proc/7293/stat : 7293 (gnovelty+2) R 7291 7293 7293 0 -1 4202496 445 0 0 0 282 0 0 0 25 0 1 0 538579176 1042362368 185 33554432000 134512640 135663704 4287087840 18446744073709551615 134524821 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/7293/statm: 254483 185 113 282 0 254199 0
[pid=7294] ppid=7285 vsize=289376 CPUtime=0 cores=1,3,5,7
/proc/7294/stat : 7294 (python) S 7285 7285 6070 0 -1 4202560 909 0 0 0 0 0 0 0 15 0 1 0 538579177 296321024 5725 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 4103465360085188872 0 0 17 7 0 0 0
/proc/7294/statm: 72344 5725 371 364 0 8259 0
[pid=7296] ppid=7294 vsize=2952 CPUtime=2.79 cores=1,3,5,7
/proc/7296/stat : 7296 (march_hi) R 7294 7296 7296 0 -1 4202496 811 0 0 0 279 0 0 0 25 0 1 0 538579179 3022848 548 33554432000 4194304 4955518 140736515668048 18446744073709551615 4227671 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7296/statm: 738 548 104 186 0 551 0
[pid=7297] ppid=7285 vsize=299260 CPUtime=0 cores=1,3,5,7
/proc/7297/stat : 7297 (python) S 7285 7285 6070 0 -1 4202560 897 0 0 0 0 0 0 0 18 0 1 0 538579180 306442240 5633 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 5641832760436818181 0 0 17 7 0 0 0
/proc/7297/statm: 74815 5633 371 364 0 10727 0
[pid=7299] ppid=7297 vsize=1017932 CPUtime=2.77 cores=1,3,5,7
/proc/7299/stat : 7299 (gnovelty+2) R 7297 7299 7299 0 -1 4202496 448 0 0 0 277 0 0 0 25 0 1 0 538579182 1042362368 185 33554432000 134512640 135663704 4289840032 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/7299/statm: 254483 185 113 282 0 254199 0
[pid=7300] ppid=7285 vsize=310056 CPUtime=0.01 cores=1,3,5,7
/proc/7300/stat : 7300 (python) S 7285 7285 6070 0 -1 4202560 933 0 0 0 0 1 0 0 15 0 1 0 538579183 317497344 5758 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 17793774908593897734 0 0 17 7 0 0 0
/proc/7300/statm: 77514 5758 371 364 0 13423 0
[pid=7302] ppid=7300 vsize=3656 CPUtime=5.51 cores=1,3,5,7
/proc/7302/stat : 7302 (march_hi) R 7300 7302 7302 0 -1 4202496 931 0 0 0 551 0 0 0 25 0 1 0 538579183 3743744 670 33554432000 4194304 4955518 140736230621072 18446744073709551615 4224615 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/7302/statm: 914 670 104 186 0 727 0
[pid=7285/tid=7292] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7292/stat : 7292 (python) S 7283 7285 6070 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538579174 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7295] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7295/stat : 7295 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 5 0 538579177 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7298] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7298/stat : 7298 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 5 0 538579180 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7301] ppid=7283 vsize=320300 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7301/stat : 7301 (python) S 7283 7285 6070 0 -1 4202560 28 935 0 0 0 0 0 0 25 0 5 0 538579183 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 14.48
Current children cumulated vsize (KiB) 3540116

[startup+12.7013 s]
/proc/loadavg: 8.95 8.35 7.62 12/272 7302
/proc/meminfo: memFree=17127928/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=320300 CPUtime=0.58 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 12347 935 1 0 48 10 0 0 25 0 5 0 538579114 327987200 7272 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 80075 7272 1881 364 0 15984 0
[pid=7291] ppid=7285 vsize=278652 CPUtime=0.01 cores=1,3,5,7
/proc/7291/stat : 7291 (python) S 7285 7285 6070 0 -1 4202560 894 0 0 0 1 0 0 0 18 0 1 0 538579174 285339648 5593 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535439 0 0 16781312 514 11459426844030238984 0 0 17 7 0 0 0
/proc/7291/statm: 69663 5593 377 364 0 5581 0
[pid=7293] ppid=7291 vsize=1017932 CPUtime=6.02 cores=1,3,5,7
/proc/7293/stat : 7293 (gnovelty+2) R 7291 7293 7293 0 -1 4202496 445 0 0 0 602 0 0 0 25 0 1 0 538579176 1042362368 185 33554432000 134512640 135663704 4287087840 18446744073709551615 134524947 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/7293/statm: 254483 185 113 282 0 254199 0
[pid=7294] ppid=7285 vsize=289376 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

/proc/meminfo: memFree=16892856/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=351344 CPUtime=0.94 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 15736 935 1 0 83 11 0 0 18 0 8 0 538579114 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 87836 7377 1888 364 0 23736 0
[pid=7291] ppid=7285 vsize=288896 CPUtime=0.32 cores=1,3,5,7
/proc/7291/stat : 7291 (python) S 7285 7285 6070 0 -1 4202560 1107 0 0 0 17 15 0 0 19 0 2 0 538579174 295829504 5631 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7291/statm: 72224 5631 410 364 0 8142 0
[pid=7293] ppid=7291 vsize=1017932 CPUtime=98.14 cores=1,3,5,7
/proc/7293/stat : 7293 (gnovelty+2) T 7291 7293 7293 0 -1 4202496 445 0 0 0 9814 0 0 0 25 0 1 0 538579176 1042362368 185 33554432000 134512640 135663704 4287087840 18446744073709551615 134544212 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/7293/statm: 254483 185 113 282 0 254199 0
[pid=7291/tid=7449] ppid=7285 vsize=288896 CPUtime=0 cores=1,3,5,7
/proc/7291/task/7449/stat : 7449 (python) S 7285 7285 6070 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538598808 295829504 5631 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604465887697024 0 0 -1 3 0 0 0
[pid=7294] ppid=7285 vsize=289376 CPUtime=2 cores=1,3,5,7
/proc/7294/stat : 7294 (python) S 7285 7285 6070 0 -1 4202560 1154 0 0 0 106 94 0 0 15 0 2 0 538579177 296321024 5764 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7294/statm: 72344 5764 409 364 0 8259 0
[pid=7296] ppid=7294 vsize=125704 CPUtime=782.05 cores=1,3,5,7
/proc/7296/stat : 7296 (march_hi) T 7294 7296 7296 0 -1 4202496 26766 0 0 0 78200 5 0 0 25 0 1 0 538579179 128720896 26503 33554432000 4194304 4955518 140736515668048 18446744073709551615 4227747 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/7296/statm: 31426 26503 104 186 0 31239 0
[pid=7294/tid=7467] ppid=7285 vsize=289376 CPUtime=0 cores=1,3,5,7
/proc/7294/task/7467/stat : 7467 (python) S 7285 7285 6070 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 538618254 296321024 5764 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604458695085120 0 0 -1 7 0 0 0
[pid=7297] ppid=7285 vsize=299260 CPUtime=0.32 cores=1,3,5,7
/proc/7297/stat : 7297 (python) S 7285 7285 6070 0 -1 4202560 1107 0 0 0 17 15 0 0 19 0 2 0 538579180 306442240 5663 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7297/statm: 74815 5663 400 364 0 10727 0
[pid=7299] ppid=7297 vsize=1017932 CPUtime=97.2 cores=1,3,5,7
/proc/7299/stat : 7299 (gnovelty+2) T 7297 7299 7299 0 -1 4202496 448 0 0 0 9720 0 0 0 25 0 1 0 538579182 1042362368 185 33554432000 134512640 135663704 4289840032 18446744073709551615 134525535 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/7299/statm: 254483 185 113 282 0 254199 0
[pid=7297/tid=7445] ppid=7285 vsize=299260 CPUtime=0 cores=1,3,5,7
/proc/7297/task/7445/stat : 7445 (python) S 7285 7285 6070 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538598627 306442240 5663 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604467111186688 0 0 -1 7 0 0 0
[pid=7300] ppid=7285 vsize=310056 CPUtime=2.41 cores=1,3,5,7
/proc/7300/stat : 7300 (python) S 7285 7285 6070 0 -1 4202560 1181 0 0 0 125 116 0 0 15 0 2 0 538579183 317497344 5797 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7300/statm: 77514 5797 409 364 0 13423 0
[pid=7302] ppid=7300 vsize=125704 CPUtime=771.06 cores=1,3,5,7
/proc/7302/stat : 7302 (march_hi) R 7300 7302 7302 0 -1 4202496 26476 0 0 0 77101 5 0 0 25 0 1 0 538579183 128720896 26215 33554432000 4194304 4955518 140736230621072 18446744073709551615 4227671 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7302/statm: 31426 26215 104 186 0 31239 0
[pid=7300/tid=7476] ppid=7285 vsize=310056 CPUtime=0 cores=1,3,5,7
/proc/7300/task/7476/stat : 7476 (python) S 7285 7285 6070 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538632561 317497344 5797 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604454223638720 0 0 -1 7 0 0 0
[pid=7446] ppid=7285 vsize=320312 CPUtime=2.08 cores=1,3,5,7
/proc/7446/stat : 7446 (python) S 7285 7285 6070 0 -1 4202560 1165 0 0 0 111 97 0 0 15 0 2 0 538598630 327999488 5794 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/7446/statm: 80078 5794 400 364 0 15984 0
[pid=7448] ppid=7446 vsize=108500 CPUtime=632.01 cores=1,3,5,7
/proc/7448/stat : 7448 (march_hi) R 7446 7448 7448 0 -1 4202496 22176 0 0 0 63196 5 0 0 25 0 1 0 538598630 111104000 21911 33554432000 4194304 4955518 140735229409248 18446744073709551615 4209653 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/7448/statm: 27125 21911 104 186 0 26938 0
[pid=7446/tid=7595] ppid=7285 vsize=320312 CPUtime=0 cores=1,3,5,7
/proc/7446/task/7595/stat : 7595 (python) S 7285 7285 6070 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 538680110 327999488 5794 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604469817549888 0 0 -1 5 0 0 0
[pid=7450] ppid=7285 vsize=330172 CPUtime=2.05 cores=1,3,5,7
/proc/7450/stat : 7450 (python) S 7285 7285 6070 0 -1 4202560 1181 0 0 0 110 95 0 0 15 0 2 0 538598811 338096128 5706 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/7450/statm: 82543 5706 400 364 0 18446 0
[pid=7452] ppid=7450 vsize=108500 CPUtime=596.42 cores=1,3,5,7
/proc/7452/stat : 7452 (march_hi) R 7450 7452 7452 0 -1 4202496 21028 0 0 0 59639 3 0 0 25 0 1 0 538598811 111104000 20763 33554432000 4194304 4955518 140734852709360 18446744073709551615 4238976 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7452/statm: 27125 20763 104 186 0 26938 0
[pid=7450/tid=7614] ppid=7285 vsize=330172 CPUtime=0 cores=1,3,5,7
/proc/7450/task/7614/stat : 7614 (python) S 7285 7285 6070 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538687165 338096128 5706 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604466767779136 0 0 -1 3 0 0 0
[pid=7616] ppid=7285 vsize=341100 CPUtime=0.39 cores=1,3,5,7
/proc/7616/stat : 7616 (python) S 7285 7285 6070 0 -1 4202560 1021 0 0 0 22 17 0 0 15 0 1 0 538688919 349286400 5839 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 11459426844030238984 0 0 17 7 0 0 0
/proc/7616/statm: 85275 5839 371 364 0 21175 0
[pid=7618] ppid=7616 vsize=37560 CPUtime=199.37 cores=1,3,5,7
/proc/7618/stat : 7618 (march_hi) R 7616 7618 7618 0 -1 4202496 7865 0 0 0 19936 1 0 0 25 0 1 0 538688919 38461440 7600 33554432000 4194304 4955518 140734033613808 18446744073709551615 4261527 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/7618/statm: 9390 7600 104 186 0 9203 0
[pid=7285/tid=7292] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7292/stat : 7292 (python) S 7283 7285 6070 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 8 0 538579174 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7295] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7295/stat : 7295 (python) S 7283 7285 6070 0 -1 4202560 24 935 0 0 0 0 0 0 18 0 8 0 538579177 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7298] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7298/stat : 7298 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 8 0 538579180 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7301] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7301/stat : 7301 (python) S 7283 7285 6070 0 -1 4202560 38 935 0 0 0 0 0 0 18 0 8 0 538579183 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7447] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7447/stat : 7447 (python) S 7283 7285 6070 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 8 0 538598630 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7451] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7451/stat : 7451 (python) S 7283 7285 6070 0 -1 4202560 26 935 0 0 0 0 0 0 18 0 8 0 538598811 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7617] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7617/stat : 7617 (python) S 7283 7285 6070 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 8 0 538688919 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3186.76
Current children cumulated vsize (KiB) 5072348

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1300 s]
/proc/loadavg: 11.19 11.13 10.26 12/321 7636
/proc/meminfo: memFree=16892856/32951124 swapFree=67111528/67111528
[pid=7285] ppid=7283 vsize=351344 CPUtime=0.94 cores=1,3,5,7
/proc/7285/stat : 7285 (python) S 7283 7285 6070 0 -1 4202496 15736 935 1 0 83 11 0 0 18 0 8 0 538579114 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/7285/statm: 87836 7377 1888 364 0 23736 0
[pid=7291] ppid=7285 vsize=288896 CPUtime=0.32 cores=1,3,5,7
/proc/7291/stat : 7291 (python) S 7285 7285 6070 0 -1 4202560 1107 0 0 0 17 15 0 0 19 0 2 0 538579174 295829504 5631 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7291/statm: 72224 5631 410 364 0 8142 0
[pid=7293] ppid=7291 vsize=1017932 CPUtime=98.14 cores=1,3,5,7
/proc/7293/stat : 7293 (gnovelty+2) T 7291 7293 7293 0 -1 4202496 445 0 0 0 9814 0 0 0 25 0 1 0 538579176 1042362368 185 33554432000 134512640 135663704 4287087840 18446744073709551615 134544212 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/7293/statm: 254483 185 113 282 0 254199 0
[pid=7291/tid=7449] ppid=7285 vsize=288896 CPUtime=0 cores=1,3,5,7
/proc/7291/task/7449/stat : 7449 (python) S 7285 7285 6070 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538598808 295829504 5631 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604465887697024 0 0 -1 3 0 0 0
[pid=7294] ppid=7285 vsize=289376 CPUtime=2 cores=1,3,5,7
/proc/7294/stat : 7294 (python) S 7285 7285 6070 0 -1 4202560 1154 0 0 0 106 94 0 0 15 0 2 0 538579177 296321024 5764 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7294/statm: 72344 5764 409 364 0 8259 0
[pid=7296] ppid=7294 vsize=125704 CPUtime=782.05 cores=1,3,5,7
/proc/7296/stat : 7296 (march_hi) T 7294 7296 7296 0 -1 4202496 26766 0 0 0 78200 5 0 0 25 0 1 0 538579179 128720896 26503 33554432000 4194304 4955518 140736515668048 18446744073709551615 4227747 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/7296/statm: 31426 26503 104 186 0 31239 0
[pid=7294/tid=7467] ppid=7285 vsize=289376 CPUtime=0 cores=1,3,5,7
/proc/7294/task/7467/stat : 7467 (python) S 7285 7285 6070 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 538618254 296321024 5764 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604458695085120 0 0 -1 7 0 0 0
[pid=7297] ppid=7285 vsize=299260 CPUtime=0.32 cores=1,3,5,7
/proc/7297/stat : 7297 (python) S 7285 7285 6070 0 -1 4202560 1107 0 0 0 17 15 0 0 19 0 2 0 538579180 306442240 5663 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7297/statm: 74815 5663 400 364 0 10727 0
[pid=7299] ppid=7297 vsize=1017932 CPUtime=97.2 cores=1,3,5,7
/proc/7299/stat : 7299 (gnovelty+2) T 7297 7299 7299 0 -1 4202496 448 0 0 0 9720 0 0 0 25 0 1 0 538579182 1042362368 185 33554432000 134512640 135663704 4289840032 18446744073709551615 134525535 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/7299/statm: 254483 185 113 282 0 254199 0
[pid=7297/tid=7445] ppid=7285 vsize=299260 CPUtime=0 cores=1,3,5,7
/proc/7297/task/7445/stat : 7445 (python) S 7285 7285 6070 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538598627 306442240 5663 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604467111186688 0 0 -1 7 0 0 0
[pid=7300] ppid=7285 vsize=310056 CPUtime=2.41 cores=1,3,5,7
/proc/7300/stat : 7300 (python) S 7285 7285 6070 0 -1 4202560 1181 0 0 0 125 116 0 0 15 0 2 0 538579183 317497344 5797 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/7300/statm: 77514 5797 409 364 0 13423 0
[pid=7302] ppid=7300 vsize=125704 CPUtime=771.06 cores=1,3,5,7
/proc/7302/stat : 7302 (march_hi) R 7300 7302 7302 0 -1 4202496 26476 0 0 0 77101 5 0 0 25 0 1 0 538579183 128720896 26215 33554432000 4194304 4955518 140736230621072 18446744073709551615 4227671 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7302/statm: 31426 26215 104 186 0 31239 0
[pid=7300/tid=7476] ppid=7285 vsize=310056 CPUtime=0 cores=1,3,5,7
/proc/7300/task/7476/stat : 7476 (python) S 7285 7285 6070 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538632561 317497344 5797 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604454223638720 0 0 -1 7 0 0 0
[pid=7446] ppid=7285 vsize=320312 CPUtime=2.08 cores=1,3,5,7
/proc/7446/stat : 7446 (python) S 7285 7285 6070 0 -1 4202560 1165 0 0 0 111 97 0 0 15 0 2 0 538598630 327999488 5794 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/7446/statm: 80078 5794 400 364 0 15984 0
[pid=7448] ppid=7446 vsize=108500 CPUtime=632.01 cores=1,3,5,7
/proc/7448/stat : 7448 (march_hi) R 7446 7448 7448 0 -1 4202496 22176 0 0 0 63196 5 0 0 25 0 1 0 538598630 111104000 21911 33554432000 4194304 4955518 140735229409248 18446744073709551615 4209653 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/7448/statm: 27125 21911 104 186 0 26938 0
[pid=7446/tid=7595] ppid=7285 vsize=320312 CPUtime=0 cores=1,3,5,7
/proc/7446/task/7595/stat : 7595 (python) S 7285 7285 6070 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 538680110 327999488 5794 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604469817549888 0 0 -1 5 0 0 0
[pid=7450] ppid=7285 vsize=330172 CPUtime=2.05 cores=1,3,5,7
/proc/7450/stat : 7450 (python) S 7285 7285 6070 0 -1 4202560 1181 0 0 0 110 95 0 0 15 0 2 0 538598811 338096128 5706 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/7450/statm: 82543 5706 400 364 0 18446 0
[pid=7452] ppid=7450 vsize=108500 CPUtime=596.42 cores=1,3,5,7
/proc/7452/stat : 7452 (march_hi) R 7450 7452 7452 0 -1 4202496 21028 0 0 0 59639 3 0 0 25 0 1 0 538598811 111104000 20763 33554432000 4194304 4955518 140734852709360 18446744073709551615 4238976 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/7452/statm: 27125 20763 104 186 0 26938 0
[pid=7450/tid=7614] ppid=7285 vsize=330172 CPUtime=0 cores=1,3,5,7
/proc/7450/task/7614/stat : 7614 (python) S 7285 7285 6070 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538687165 338096128 5706 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 514 18446604466767779136 0 0 -1 3 0 0 0
[pid=7616] ppid=7285 vsize=341100 CPUtime=0.39 cores=1,3,5,7
/proc/7616/stat : 7616 (python) S 7285 7285 6070 0 -1 4202560 1021 0 0 0 22 17 0 0 15 0 1 0 538688919 349286400 5839 33554432000 4194304 5683644 140736164929056 18446744073709551615 216634535494 0 0 16781312 514 11459426844030238984 0 0 17 7 0 0 0
/proc/7616/statm: 85275 5839 371 364 0 21175 0
[pid=7618] ppid=7616 vsize=37560 CPUtime=199.37 cores=1,3,5,7
/proc/7618/stat : 7618 (march_hi) R 7616 7618 7618 0 -1 4202496 7865 0 0 0 19936 1 0 0 25 0 1 0 538688919 38461440 7600 33554432000 4194304 4955518 140734033613808 18446744073709551615 4261527 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/7618/statm: 9390 7600 104 186 0 9203 0
[pid=7285/tid=7292] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7292/stat : 7292 (python) S 7283 7285 6070 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 8 0 538579174 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7295] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7295/stat : 7295 (python) S 7283 7285 6070 0 -1 4202560 24 935 0 0 0 0 0 0 18 0 8 0 538579177 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7298] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7298/stat : 7298 (python) S 7283 7285 6070 0 -1 4202560 19 935 0 0 0 0 0 0 25 0 8 0 538579180 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7301] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7301/stat : 7301 (python) S 7283 7285 6070 0 -1 4202560 38 935 0 0 0 0 0 0 18 0 8 0 538579183 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7447] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7447/stat : 7447 (python) S 7283 7285 6070 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 8 0 538598630 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7451] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7451/stat : 7451 (python) S 7283 7285 6070 0 -1 4202560 26 935 0 0 0 0 0 0 18 0 8 0 538598811 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
[pid=7285/tid=7617] ppid=7283 vsize=351344 CPUtime=0 cores=1,3,5,7
/proc/7285/task/7617/stat : 7617 (python) S 7283 7285 6070 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 8 0 538688919 359776256 7377 33554432000 4194304 5683644 140736164929056 18446744073709551615 216646338993 0 0 16781312 2 18446604466156049600 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3186.76
Current children cumulated vsize (KiB) 5072348

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7285 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=834873
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=114982
# CPU time returned by wait4() is 0.949855
# while last known CPU time is 3186.76
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1300.06
CPU time (s): 3186.76
CPU user time (s): 3181.97
CPU system time (s): 4.79
CPU usage (%): 245.124
Max. virtual memory (cumulated for all children) (KiB): 5072352

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.834873
system time used= 0.114982
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 16674
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1010
involuntary context switches= 44

runsolver used 3.3025 second user time and 7.2269 second system time

The end

Launcher Data 

Begin job on node103 at 2011-05-01 19:09:25
IDJOB=3317460
IDBENCH=82925
IDSOLVER=1705
FILE ID=node103/3317460-1304269765
RUNJOBID= node103-1304267295-6089
PBS_JOBID= 13173975
Free space on /tmp= 72552 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v600-c2556-S1677156340-036.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317460-1304269765/watcher-3317460-1304269765 -o /tmp/evaluation-result-3317460-1304269765/solver-3317460-1304269765 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317460-1304269765.cnf -seed 1529422657 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 99f3efd3bfc8a6f99ffbd289ed4a5837
RANDOM SEED=1529422657

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17182280 kB
Buffers:       2158820 kB
Cached:       11549276 kB
SwapCached:          0 kB
Active:        7236060 kB
Inactive:      7922600 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17182280 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           37272 kB
Writeback:           0 kB
AnonPages:     1449976 kB
Mapped:          26044 kB
Slab:           531172 kB
PageTables:      18596 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 30079908 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72520 MiB
End job on node103 at 2011-05-01 19:31:07