Trace number 3317274

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.23 1203.06

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S396831120-092.UNKNOWN.cnf
MD5SUM1c5680fb79b6ea8965a1c94337260953
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.38/0.63	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.38/0.63	c INFO: solving HOME/instance-3317274-1304268570.cnf
0.38/0.63	c INFO: computing task features
0.38/0.65	c INFO: computed variable-clause graph statistics
0.38/0.65	c INFO: computed clause balance statistics
0.47/0.71	c INFO: computed variable balance statistics
0.47/0.75	c INFO: computed small-clause counts
0.47/0.75	c INFO: computed Horn-clause counts
0.47/0.75	c INFO: computed variable graph statistics
0.47/0.75	c INFO: collected features for HOME/instance-3317274-1304268570.cnf in 0.12 s
0.47/0.78	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.63)
0.56/0.81	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.07)
0.56/0.84	c INFO: running march_hi@0 for 485 with 4799 remaining (b = 0.34)
0.56/0.87	c INFO: running march_hi@0 for 485 with 4799 remaining (b = 0.30)
396.22/99.98	c INFO: running march_hi@0 for 485 with 4700 remaining (b = 0.27)
396.59/100.01	c INFO: running march_hi@0 for 485 with 4700 remaining (b = 0.24)
1952.33/489.69	c INFO: running march_hi@0 for 291 with 4310 remaining (b = 0.15)
1978.68/496.21	c INFO: running march_hi@99 for 97 with 4304 remaining (b = 0.06)
2351.20/589.52	c INFO: running march_hi@488 for 97 with 4211 remaining (b = 0.07)
2353.59/590.19	c INFO: running march_hi@98 for 97 with 4210 remaining (b = 0.06)
2366.38/593.35	c INFO: running march_hi@587 for 97 with 4207 remaining (b = 0.08)
2733.31/685.27	c INFO: running march_hi@0 for 485 with 4115 remaining (b = 0.22)
2738.50/686.59	c INFO: running march_hi@195 for 194 with 4114 remaining (b = 0.11)
2740.87/687.11	c INFO: running march_hi@0 for 388 with 4113 remaining (b = 0.16)
3110.15/779.63	c INFO: running march_hi@490 for 97 with 4021 remaining (b = 0.04)
3508.58/879.46	c INFO: running march_hi@288 for 194 with 3921 remaining (b = 0.09)
3516.57/881.40	c INFO: running march_hi@977 for 1164 with 3919 remaining (b = 0.40)
4295.12/1076.49	c INFO: running march_hi@288 for 194 with 3724 remaining (b = 0.10)
4295.92/1076.62	c INFO: running march_hi@388 for 97 with 3724 remaining (b = 0.05)
4680.29/1172.96	c INFO: running march_hi@96 for 291 with 3627 remaining (b = 0.13)
4692.68/1176.06	c INFO: running march_hi@0 for 388 with 3624 remaining (b = 0.15)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317274-1304268570/watcher-3317274-1304268570 -o /tmp/evaluation-result-3317274-1304268570/solver-3317274-1304268570 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317274-1304268570.cnf -seed 1647509017 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 7.98 5.99 5/231 22165
/proc/meminfo: memFree=28556912/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=15384 CPUtime=0 cores=1,3,5,7
/proc/22165/stat : 22165 (python) R 22163 22165 21419 0 -1 4202496 811 0 0 0 0 0 0 0 25 0 1 0 617087462 15753216 696 33554432000 4194304 5683644 140733307756160 18446744073709551615 4433760 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/22165/statm: 3846 697 334 364 0 333 0

[startup+0.107709 s]
/proc/loadavg: 8.04 7.98 5.99 5/231 22165
/proc/meminfo: memFree=28556912/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=94184 CPUtime=0.06 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 2772 0 0 0 5 1 0 0 25 0 1 0 617087462 96444416 2478 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458063872 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 0
/proc/22165/statm: 23546 2478 765 364 0 1678 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 94184

[startup+0.200671 s]
/proc/loadavg: 8.04 7.98 5.99 5/231 22165
/proc/meminfo: memFree=28556912/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=124240 CPUtime=0.13 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 4314 0 1 0 9 4 0 0 25 0 1 0 617087462 127221760 3881 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458065664 0 0 16781312 2 18446744071562362840 0 0 17 3 0 0 0
/proc/22165/statm: 31060 3881 997 364 0 2871 0
Current children cumulated CPU time (s) 0.13
Current children cumulated vsize (KiB) 124240

[startup+0.300668 s]
/proc/loadavg: 8.04 7.98 5.99 5/231 22165
/proc/meminfo: memFree=28556912/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=135752 CPUtime=0.2 cores=1,3,5,7
/proc/22165/stat : 22165 (python) R 22163 22165 21419 0 -1 4202496 5963 931 1 0 14 6 0 0 17 0 1 0 617087462 139010048 4152 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458061829 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/22165/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 135752

[startup+0.700588 s]
/proc/loadavg: 8.04 7.98 5.99 5/231 22165
/proc/meminfo: memFree=28556912/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=278528 CPUtime=0.47 cores=1,3,5,7
/proc/22165/stat : 22165 (python) R 22163 22165 21419 0 -1 4202496 9359 931 1 0 36 11 0 0 16 0 1 0 617087462 285212672 7030 33554432000 4194304 5683644 140733307756160 18446744073709551615 4324495 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/22165/statm: 69632 7030 1826 364 0 5556 0
Current children cumulated CPU time (s) 0.47
Current children cumulated vsize (KiB) 278528

[startup+1.50139 s]
/proc/loadavg: 8.03 7.98 6.00 9/244 22182
/proc/meminfo: memFree=28518228/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=320236 CPUtime=0.64 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 12386 931 1 0 52 12 0 0 22 0 5 0 617087462 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/22165/statm: 80059 7256 1881 364 0 15968 0
[pid=22171] ppid=22165 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/22171/stat : 22171 (python) S 22165 22165 21419 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 617087540 285237248 5571 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085903 0 0 16781312 514 8163834532991074303 0 0 17 7 0 0 0
/proc/22171/statm: 69638 5571 354 364 0 5556 0
[pid=22173] ppid=22171 vsize=1017800 CPUtime=0.7 cores=1,3,5,7
/proc/22173/stat : 22173 (gnovelty+2) R 22171 22173 22173 0 -1 4202496 441 0 0 0 70 0 0 0 21 0 1 0 617087541 1042227200 181 33554432000 134512640 135663704 4292118496 18446744073709551615 134524509 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22173/statm: 254450 181 113 282 0 254166 0
[pid=22174] ppid=22165 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/22174/stat : 22174 (python) S 22165 22165 21419 0 -1 4202560 742 0 0 0 0 0 0 0 19 0 1 0 617087543 296288256 5694 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 4453146616479612927 0 0 17 7 0 0 0
/proc/22174/statm: 72336 5694 348 364 0 8251 0
[pid=22176] ppid=22174 vsize=1017800 CPUtime=0.68 cores=1,3,5,7
/proc/22176/stat : 22176 (gnovelty+2) R 22174 22176 22176 0 -1 4202496 443 0 0 0 68 0 0 0 22 0 1 0 617087544 1042227200 180 33554432000 134512640 135663704 4293927152 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22176/statm: 254450 180 113 282 0 254166 0
[pid=22177] ppid=22165 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/22177/stat : 22177 (python) S 22165 22165 21419 0 -1 4202560 757 0 0 0 0 0 0 0 17 0 1 0 617087546 306307072 5595 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 11512304561528209671 0 0 17 3 0 0 0
/proc/22177/statm: 74782 5595 348 364 0 10694 0
[pid=22179] ppid=22177 vsize=1848 CPUtime=0.65 cores=1,3,5,7
/proc/22179/stat : 22179 (march_hi) R 22177 22179 22179 0 -1 4202496 559 0 0 0 65 0 0 0 24 0 1 0 617087546 1892352 293 33554432000 4194304 4955518 140734095901584 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22179/statm: 462 293 106 186 0 275 0
[pid=22180] ppid=22165 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/22180/stat : 22180 (python) S 22165 22165 21419 0 -1 4202560 767 0 0 0 0 0 0 0 18 0 1 0 617087549 317431808 5719 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 9464807202423275783 0 0 17 7 0 0 0
/proc/22180/statm: 77498 5719 348 364 0 13407 0
[pid=22182] ppid=22180 vsize=1848 CPUtime=0.61 cores=1,3,5,7
/proc/22182/stat : 22182 (march_hi) R 22180 22182 22182 0 -1 4202496 561 0 0 0 61 0 0 0 25 0 1 0 617087549 1892352 290 33554432000 4194304 4955518 140735798025264 18446744073709551615 4224622 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22182/statm: 462 290 106 186 0 275 0
[pid=22165/tid=22172] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22172/stat : 22172 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 18 0 5 0 617087541 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22175] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22175/stat : 22175 (python) S 22163 22165 21419 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 5 0 617087543 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 5 0 0 0
[pid=22165/tid=22178] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22178/stat : 22178 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 20 0 5 0 617087546 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22181] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22181/stat : 22181 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 22 0 5 0 617087549 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3.28
Current children cumulated vsize (KiB) 3536548

[startup+3.10109 s]
/proc/loadavg: 8.03 7.98 6.00 9/244 22182
/proc/meminfo: memFree=28515748/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=320236 CPUtime=0.64 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 12386 931 1 0 52 12 0 0 22 0 5 0 617087462 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/22165/statm: 80059 7256 1881 364 0 15968 0
[pid=22171] ppid=22165 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/22171/stat : 22171 (python) S 22165 22165 21419 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 617087540 285237248 5571 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085903 0 0 16781312 514 8163834532991041799 0 0 17 7 0 0 0
/proc/22171/statm: 69638 5571 354 364 0 5556 0
[pid=22173] ppid=22171 vsize=1017800 CPUtime=2.3 cores=1,3,5,7
/proc/22173/stat : 22173 (gnovelty+2) R 22171 22173 22173 0 -1 4202496 441 0 0 0 230 0 0 0 25 0 1 0 617087541 1042227200 181 33554432000 134512640 135663704 4292118496 18446744073709551615 134546888 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22173/statm: 254450 181 113 282 0 254166 0
[pid=22174] ppid=22165 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/22174/stat : 22174 (python) S 22165 22165 21419 0 -1 4202560 742 0 0 0 0 0 0 0 18 0 1 0 617087543 296288256 5694 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 4453146616479580421 0 0 17 7 0 0 0
/proc/22174/statm: 72336 5694 348 364 0 8251 0
[pid=22176] ppid=22174 vsize=1017800 CPUtime=2.27 cores=1,3,5,7
/proc/22176/stat : 22176 (gnovelty+2) R 22174 22176 22176 0 -1 4202496 443 0 0 0 227 0 0 0 25 0 1 0 617087544 1042227200 180 33554432000 134512640 135663704 4293927152 18446744073709551615 134524888 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22176/statm: 254450 180 113 282 0 254166 0
[pid=22177] ppid=22165 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/22177/stat : 22177 (python) S 22165 22165 21419 0 -1 4202560 758 0 0 0 0 0 0 0 15 0 1 0 617087546 306307072 5595 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 11512304561528209671 0 0 17 3 0 0 0
/proc/22177/statm: 74782 5595 348 364 0 10694 0
[pid=22179] ppid=22177 vsize=2444 CPUtime=2.25 cores=1,3,5,7
/proc/22179/stat : 22179 (march_hi) R 22177 22179 22179 0 -1 4202496 708 0 0 0 225 0 0 0 25 0 1 0 617087546 2502656 424 33554432000 4194304 4955518 140734095901584 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22179/statm: 611 424 106 186 0 424 0
[pid=22180] ppid=22165 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/22180/stat : 22180 (python) S 22165 22165 21419 0 -1 4202560 771 0 0 0 0 0 0 0 15 0 1 0 617087549 317431808 5719 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 9464807202423275783 0 0 17 7 0 0 0
/proc/22180/statm: 77498 5719 348 364 0 13407 0
[pid=22182] ppid=22180 vsize=2444 CPUtime=2.2 cores=1,3,5,7
/proc/22182/stat : 22182 (march_hi) R 22180 22182 22182 0 -1 4202496 708 0 0 0 220 0 0 0 25 0 1 0 617087549 2502656 419 33554432000 4194304 4955518 140735798025264 18446744073709551615 4208693 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22182/statm: 611 419 106 186 0 424 0
[pid=22165/tid=22172] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22172/stat : 22172 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 18 0 5 0 617087541 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22175] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22175/stat : 22175 (python) S 22163 22165 21419 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 5 0 617087543 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 5 0 0 0
[pid=22165/tid=22178] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22178/stat : 22178 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 20 0 5 0 617087546 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22181] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22181/stat : 22181 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 22 0 5 0 617087549 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 9.66
Current children cumulated vsize (KiB) 3537740

[startup+6.3005 s]
/proc/loadavg: 8.03 7.98 6.00 9/244 22182
/proc/meminfo: memFree=28513764/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=320236 CPUtime=0.64 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 12386 931 1 0 52 12 0 0 22 0 5 0 617087462 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/22165/statm: 80059 7256 1881 364 0 15968 0
[pid=22171] ppid=22165 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/22171/stat : 22171 (python) S 22165 22165 21419 0 -1 4202560 903 0 0 0 0 0 0 0 18 0 1 0 617087540 285237248 5594 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085903 0 0 16781312 514 8163834532991041799 0 0 17 7 0 0 0
/proc/22171/statm: 69638 5594 377 364 0 5556 0
[pid=22173] ppid=22171 vsize=1017800 CPUtime=5.5 cores=1,3,5,7
/proc/22173/stat : 22173 (gnovelty+2) R 22171 22173 22173 0 -1 4202496 441 0 0 0 550 0 0 0 25 0 1 0 617087541 1042227200 181 33554432000 134512640 135663704 4292118496 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22173/statm: 254450 181 113 282 0 254166 0
[pid=22174] ppid=22165 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/22174/stat : 22174 (python) S 22165 22165 21419 0 -1 4202560 899 0 0 0 0 0 0 0 18 0 1 0 617087543 296288256 5717 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 4453146616479580421 0 0 17 7 0 0 0
/proc/22174/statm: 72336 5717 371 364 0 8251 0
[pid=22176] ppid=22174 vsize=1017800 CPUtime=5.46 cores=1,3,5,7
/proc/22176/stat : 22176 (gnovelty+2) R 22174 22176 22176 0 -1 4202496 443 0 0 0 546 0 0 0 25 0 1 0 617087544 1042227200 180 33554432000 134512640 135663704 4293927152 18446744073709551615 134524526 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22176/statm: 254450 180 113 282 0 254166 0
[pid=22177] ppid=22165 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/22177/stat : 22177 (python) S 22165 22165 21419 0 -1 4202560 918 0 0 0 0 0 0 0 15 0 1 0 617087546 306307072 5618 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 11512304561528209671 0 0 17 3 0 0 0
/proc/22177/statm: 74782 5618 371 364 0 10694 0
[pid=22179] ppid=22177 vsize=3296 CPUtime=5.45 cores=1,3,5,7
/proc/22179/stat : 22179 (march_hi) R 22177 22179 22179 0 -1 4202496 885 0 0 0 545 0 0 0 25 0 1 0 617087546 3375104 601 33554432000 4194304 4955518 140734095901584 18446744073709551615 4237856 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22179/statm: 824 601 106 186 0 637 0
[pid=22180] ppid=22165 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/22180/stat : 22180 (python) S 22165 22165 21419 0 -1 4202560 913 0 0 0 0 0 0 0 15 0 1 0 617087549 317431808 5742 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 9464807202423275783 0 0 17 7 0 0 0
/proc/22180/statm: 77498 5742 371 364 0 13407 0
[pid=22182] ppid=22180 vsize=3296 CPUtime=5.35 cores=1,3,5,7
/proc/22182/stat : 22182 (march_hi) R 22180 22182 22182 0 -1 4202496 879 0 0 0 535 0 0 0 25 0 1 0 617087549 3375104 590 33554432000 4194304 4955518 140735798025264 18446744073709551615 4306973 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22182/statm: 824 590 106 186 0 637 0
[pid=22165/tid=22172] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22172/stat : 22172 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 18 0 5 0 617087541 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22175] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22175/stat : 22175 (python) S 22163 22165 21419 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 5 0 617087543 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 5 0 0 0
[pid=22165/tid=22178] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22178/stat : 22178 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 20 0 5 0 617087546 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22181] ppid=22163 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22181/stat : 22181 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 22 0 5 0 617087549 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 22.4
Current children cumulated vsize (KiB) 3539444

[startup+12.7013 s]
/proc/loadavg: 8.03 7.98 6.02 9/244 22182
/proc/meminfo: memFree=28510580/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=320236 CPUtime=0.64 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 12386 931 1 0 52 12 0 0 22 0 5 0 617087462 327921664 7256 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/22165/statm: 80059 7256 1881 364 0 15968 0
[pid=22171] ppid=22165 vsize=278552 CPUtime=0.01 cores=1,3,5,7
/proc/22171/stat : 22171 (python) S 22165 22165 21419 0 -1 4202560 904 0 0 0 0 1 0 0 18 0 1 0 617087540 285237248 5594 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085903 0 0 16781312 514 10182955416833458439 0 0 17 7 0 0 0
/proc/22171/statm: 69638 5594 377 364 0 5556 0
[pid=22173] ppid=22171 vsize=1017800 CPUtime=11.9 cores=1,3,5,7
/proc/22173/stat : 22173 (gnovelty+2) R 22171 22173 22173 0 -1 4202496 441 0 0 0 1190 0 0 0 25 0 1 0 617087541 1042227200 181 33554432000 134512640 135663704 4292118496 18446744073709551615 134544380 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22173/statm: 254450 181 113 282 0 254166 0
[pid=22174] ppid=22165 vsize=289344 CPUtime=0.01 cores=1,3,5,7

################
# More data... #
################

/proc/22295/stat : 22295 (march_hi) T 22293 22295 22295 0 -1 4202496 46422 0 0 0 107188 10 0 0 25 0 1 0 617097460 219213824 46114 33554432000 4194304 4955518 140734360925024 18446744073709551615 4224445 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22295/statm: 53519 46114 106 186 0 53332 0
[pid=22293/tid=22325] ppid=22165 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/22293/task/22325/stat : 22325 (python) S 22165 22165 21419 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 617146479 327933952 5776 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471144167616 0 0 -1 3 0 0 0
[pid=22296] ppid=22165 vsize=330116 CPUtime=1.5 cores=1,3,5,7
/proc/22296/stat : 22296 (python) S 22165 22165 21419 0 -1 4202560 1183 0 0 0 77 73 0 0 15 0 2 0 617097463 338038784 5702 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22296/statm: 82529 5702 400 364 0 18432 0
[pid=22298] ppid=22296 vsize=214076 CPUtime=905.5 cores=1,3,5,7
/proc/22298/stat : 22298 (march_hi) R 22296 22298 22298 0 -1 4202496 39860 0 0 0 90542 8 0 0 25 0 1 0 617097464 219213824 39554 33554432000 4194304 4955518 140736282553984 18446744073709551615 4210637 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22298/statm: 53519 39554 106 186 0 53332 0
[pid=22296/tid=22324] ppid=22165 vsize=330116 CPUtime=0 cores=1,3,5,7
/proc/22296/task/22324/stat : 22324 (python) S 22165 22165 21419 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 617146411 338038784 5702 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604464696454208 0 0 -1 1 0 0 0
[pid=22328] ppid=22165 vsize=340940 CPUtime=0.91 cores=1,3,5,7
/proc/22328/stat : 22328 (python) S 22165 22165 21419 0 -1 4202560 1233 0 0 0 47 44 0 0 15 0 2 0 617155989 349122560 5846 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22328/statm: 85235 5846 406 364 0 21135 0
[pid=22330] ppid=22328 vsize=96900 CPUtime=515.38 cores=1,3,5,7
/proc/22330/stat : 22330 (march_hi) R 22328 22330 22330 0 -1 4202496 23520 0 0 0 51533 5 0 0 25 0 1 0 617155989 99225600 23208 33554432000 4194304 4955518 140736658367456 18446744073709551615 4206520 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22330/statm: 24225 23208 106 186 0 24038 0
[pid=22328/tid=22369] ppid=22165 vsize=340940 CPUtime=0 cores=1,3,5,7
/proc/22328/task/22369/stat : 22369 (python) S 22165 22165 21419 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 617204755 349122560 5846 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471128007872 0 0 -1 5 0 0 0
[pid=22331] ppid=22165 vsize=351196 CPUtime=0.91 cores=1,3,5,7
/proc/22331/stat : 22331 (python) S 22165 22165 21419 0 -1 4202560 1216 0 0 0 51 40 0 0 15 0 2 0 617156173 359624704 5865 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22331/statm: 87799 5865 406 364 0 23696 0
[pid=22333] ppid=22331 vsize=96900 CPUtime=515.08 cores=1,3,5,7
/proc/22333/stat : 22333 (march_hi) R 22331 22333 22333 0 -1 4202496 23583 0 0 0 51504 4 0 0 25 0 1 0 617156173 99225600 23271 33554432000 4194304 4955518 140734602858160 18446744073709551615 4297429 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22333/statm: 24225 23271 106 186 0 24038 0
[pid=22331/tid=22365] ppid=22165 vsize=351196 CPUtime=0 cores=1,3,5,7
/proc/22331/task/22365/stat : 22365 (python) S 22165 22165 21419 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 617195108 359624704 5865 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471146458048 0 0 -1 1 0 0 0
[pid=22371] ppid=22165 vsize=361452 CPUtime=0.04 cores=1,3,5,7
/proc/22371/stat : 22371 (python) S 22165 22165 21419 0 -1 4202560 985 0 0 0 2 2 0 0 15 0 1 0 617205068 370126848 5834 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 10182955416833458437 0 0 17 7 0 0 0
/proc/22371/statm: 90363 5834 371 364 0 26257 0
[pid=22373] ppid=22371 vsize=9684 CPUtime=26.43 cores=1,3,5,7
/proc/22373/stat : 22373 (march_hi) R 22371 22373 22373 0 -1 4202496 2016 0 0 0 2643 0 0 0 25 0 1 0 617205069 9916416 1723 33554432000 4194304 4955518 140735124789712 18446744073709551615 4235624 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22373/statm: 2421 1723 106 186 0 2234 0
[pid=22165/tid=22172] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22172/stat : 22172 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 18 0 10 0 617087541 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22175] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22175/stat : 22175 (python) S 22163 22165 21419 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 10 0 617087543 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 5 0 0 0
[pid=22165/tid=22178] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22178/stat : 22178 (python) S 22163 22165 21419 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 10 0 617087546 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22181] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22181/stat : 22181 (python) S 22163 22165 21419 0 -1 4202560 30 931 0 0 0 0 0 0 18 0 10 0 617087549 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22294] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22294/stat : 22294 (python) S 22163 22165 21419 0 -1 4202560 42 931 0 0 0 0 0 0 18 0 10 0 617097460 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22297] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22297/stat : 22297 (python) S 22163 22165 21419 0 -1 4202560 37 931 0 0 0 0 0 0 18 0 10 0 617097464 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22329] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22329/stat : 22329 (python) S 22163 22165 21419 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 10 0 617155989 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22332] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22332/stat : 22332 (python) S 22163 22165 21419 0 -1 4202560 32 931 0 0 0 0 0 0 18 0 10 0 617156173 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22372] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22372/stat : 22372 (python) S 22163 22165 21419 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 10 0 617205069 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 6278644

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203 s]
/proc/loadavg: 8.11 8.11 7.51 9/289 22373
/proc/meminfo: memFree=27172560/32951124 swapFree=67111348/67111528
[pid=22165] ppid=22163 vsize=371696 CPUtime=1.13 cores=1,3,5,7
/proc/22165/stat : 22165 (python) S 22163 22165 21419 0 -1 4202496 17049 931 1 0 101 12 0 0 19 0 10 0 617087462 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/22165/statm: 92924 7359 1892 364 0 28818 0
[pid=22171] ppid=22165 vsize=288796 CPUtime=0.14 cores=1,3,5,7
/proc/22171/stat : 22171 (python) S 22165 22165 21419 0 -1 4202560 1085 0 0 0 8 6 0 0 19 0 2 0 617087540 295727104 5632 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22171/statm: 72199 5632 410 364 0 8117 0
[pid=22173] ppid=22171 vsize=1017800 CPUtime=99.15 cores=1,3,5,7
/proc/22173/stat : 22173 (gnovelty+2) T 22171 22173 22173 0 -1 4202496 441 0 0 0 9915 0 0 0 25 0 1 0 617087541 1042227200 181 33554432000 134512640 135663704 4292118496 18446744073709551615 134524509 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22173/statm: 254450 181 113 282 0 254166 0
[pid=22171/tid=22291] ppid=22165 vsize=288796 CPUtime=0 cores=1,3,5,7
/proc/22171/task/22291/stat : 22291 (python) S 22165 22165 21419 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617097456 295727104 5632 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604459108451392 0 0 -1 7 0 0 0
[pid=22174] ppid=22165 vsize=289344 CPUtime=0.15 cores=1,3,5,7
/proc/22174/stat : 22174 (python) S 22165 22165 21419 0 -1 4202560 1081 0 0 0 8 7 0 0 19 0 2 0 617087543 296288256 5747 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22174/statm: 72336 5747 400 364 0 8251 0
[pid=22176] ppid=22174 vsize=1017800 CPUtime=98.98 cores=1,3,5,7
/proc/22176/stat : 22176 (gnovelty+2) T 22174 22176 22176 0 -1 4202496 443 0 0 0 9898 0 0 0 25 0 1 0 617087544 1042227200 180 33554432000 134512640 135663704 4293927152 18446744073709551615 134547147 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/22176/statm: 254450 180 113 282 0 254166 0
[pid=22174/tid=22292] ppid=22165 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/22174/task/22292/stat : 22292 (python) S 22165 22165 21419 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 617097459 296288256 5747 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471129391424 0 0 -1 7 0 0 0
[pid=22177] ppid=22165 vsize=299128 CPUtime=1.48 cores=1,3,5,7
/proc/22177/stat : 22177 (python) S 22165 22165 21419 0 -1 4202560 1154 0 0 0 81 67 0 0 15 0 2 0 617087546 306307072 5648 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22177/statm: 74782 5648 400 364 0 10694 0
[pid=22179] ppid=22177 vsize=207400 CPUtime=878.53 cores=1,3,5,7
/proc/22179/stat : 22179 (march_hi) T 22177 22179 22179 0 -1 4202496 38794 0 0 0 87845 8 0 0 25 0 1 0 617087546 212377600 38490 33554432000 4194304 4955518 140734095901584 18446744073709551615 4421280 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/22179/statm: 51850 38490 106 186 0 51663 0
[pid=22177/tid=22318] ppid=22165 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/22177/task/22318/stat : 22318 (python) S 22165 22165 21419 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 617136429 306307072 5648 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471129485568 0 0 -1 3 0 0 0
[pid=22180] ppid=22165 vsize=309992 CPUtime=1.13 cores=1,3,5,7
/proc/22180/stat : 22180 (python) S 22165 22165 21419 0 -1 4202560 1187 0 0 0 65 48 0 0 15 0 2 0 617087549 317431808 5772 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22180/statm: 77498 5772 400 364 0 13407 0
[pid=22182] ppid=22180 vsize=141100 CPUtime=680 cores=1,3,5,7
/proc/22182/stat : 22182 (march_hi) T 22180 22182 22182 0 -1 4202496 30290 0 0 0 67994 6 0 0 25 0 1 0 617087549 144486400 29981 33554432000 4194304 4955518 140735798025264 18446744073709551615 4237810 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22182/statm: 35275 29981 106 186 0 35088 0
[pid=22180/tid=22320] ppid=22165 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/22180/task/22320/stat : 22320 (python) S 22165 22165 21419 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 617137081 317431808 5772 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471128009664 0 0 -1 7 0 0 0
[pid=22293] ppid=22165 vsize=320248 CPUtime=1.81 cores=1,3,5,7
/proc/22293/stat : 22293 (python) S 22165 22165 21419 0 -1 4202560 1184 0 0 0 99 82 0 0 15 0 2 0 617097460 327933952 5776 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469892555 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22293/statm: 80062 5776 400 364 0 15968 0
[pid=22295] ppid=22293 vsize=214076 CPUtime=1071.98 cores=1,3,5,7
/proc/22295/stat : 22295 (march_hi) T 22293 22295 22295 0 -1 4202496 46422 0 0 0 107188 10 0 0 25 0 1 0 617097460 219213824 46114 33554432000 4194304 4955518 140734360925024 18446744073709551615 4224445 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22295/statm: 53519 46114 106 186 0 53332 0
[pid=22293/tid=22325] ppid=22165 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/22293/task/22325/stat : 22325 (python) S 22165 22165 21419 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 617146479 327933952 5776 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471144167616 0 0 -1 3 0 0 0
[pid=22296] ppid=22165 vsize=330116 CPUtime=1.5 cores=1,3,5,7
/proc/22296/stat : 22296 (python) S 22165 22165 21419 0 -1 4202560 1183 0 0 0 77 73 0 0 15 0 2 0 617097463 338038784 5702 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22296/statm: 82529 5702 400 364 0 18432 0
[pid=22298] ppid=22296 vsize=214076 CPUtime=905.5 cores=1,3,5,7
/proc/22298/stat : 22298 (march_hi) R 22296 22298 22298 0 -1 4202496 39860 0 0 0 90542 8 0 0 25 0 1 0 617097464 219213824 39554 33554432000 4194304 4955518 140736282553984 18446744073709551615 4210637 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22298/statm: 53519 39554 106 186 0 53332 0
[pid=22296/tid=22324] ppid=22165 vsize=330116 CPUtime=0 cores=1,3,5,7
/proc/22296/task/22324/stat : 22324 (python) S 22165 22165 21419 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 617146411 338038784 5702 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604464696454208 0 0 -1 1 0 0 0
[pid=22328] ppid=22165 vsize=340940 CPUtime=0.91 cores=1,3,5,7
/proc/22328/stat : 22328 (python) S 22165 22165 21419 0 -1 4202560 1233 0 0 0 47 44 0 0 15 0 2 0 617155989 349122560 5846 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22328/statm: 85235 5846 406 364 0 21135 0
[pid=22330] ppid=22328 vsize=96900 CPUtime=515.38 cores=1,3,5,7
/proc/22330/stat : 22330 (march_hi) R 22328 22330 22330 0 -1 4202496 23520 0 0 0 51533 5 0 0 25 0 1 0 617155989 99225600 23208 33554432000 4194304 4955518 140736658367456 18446744073709551615 4206520 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22330/statm: 24225 23208 106 186 0 24038 0
[pid=22328/tid=22369] ppid=22165 vsize=340940 CPUtime=0 cores=1,3,5,7
/proc/22328/task/22369/stat : 22369 (python) S 22165 22165 21419 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 617204755 349122560 5846 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471128007872 0 0 -1 5 0 0 0
[pid=22331] ppid=22165 vsize=351196 CPUtime=0.91 cores=1,3,5,7
/proc/22331/stat : 22331 (python) S 22165 22165 21419 0 -1 4202560 1216 0 0 0 51 40 0 0 15 0 2 0 617156173 359624704 5865 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22331/statm: 87799 5865 406 364 0 23696 0
[pid=22333] ppid=22331 vsize=96900 CPUtime=515.08 cores=1,3,5,7
/proc/22333/stat : 22333 (march_hi) R 22331 22333 22333 0 -1 4202496 23583 0 0 0 51504 4 0 0 25 0 1 0 617156173 99225600 23271 33554432000 4194304 4955518 140734602858160 18446744073709551615 4297429 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22333/statm: 24225 23271 106 186 0 24038 0
[pid=22331/tid=22365] ppid=22165 vsize=351196 CPUtime=0 cores=1,3,5,7
/proc/22331/task/22365/stat : 22365 (python) S 22165 22165 21419 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 617195108 359624704 5865 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 514 18446604471146458048 0 0 -1 1 0 0 0
[pid=22371] ppid=22165 vsize=361452 CPUtime=0.04 cores=1,3,5,7
/proc/22371/stat : 22371 (python) S 22165 22165 21419 0 -1 4202560 985 0 0 0 2 2 0 0 15 0 1 0 617205068 370126848 5834 33554432000 4194304 5683644 140733307756160 18446744073709551615 252458085958 0 0 16781312 514 10182955416833458437 0 0 17 7 0 0 0
/proc/22371/statm: 90363 5834 371 364 0 26257 0
[pid=22373] ppid=22371 vsize=9684 CPUtime=26.43 cores=1,3,5,7
/proc/22373/stat : 22373 (march_hi) R 22371 22373 22373 0 -1 4202496 2016 0 0 0 2643 0 0 0 25 0 1 0 617205069 9916416 1723 33554432000 4194304 4955518 140735124789712 18446744073709551615 4235624 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22373/statm: 2421 1723 106 186 0 2234 0
[pid=22165/tid=22172] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22172/stat : 22172 (python) S 22163 22165 21419 0 -1 4202560 25 931 0 0 0 0 0 0 18 0 10 0 617087541 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22175] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22175/stat : 22175 (python) S 22163 22165 21419 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 10 0 617087543 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 5 0 0 0
[pid=22165/tid=22178] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22178/stat : 22178 (python) S 22163 22165 21419 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 10 0 617087546 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22181] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22181/stat : 22181 (python) S 22163 22165 21419 0 -1 4202560 30 931 0 0 0 0 0 0 18 0 10 0 617087549 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22294] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22294/stat : 22294 (python) S 22163 22165 21419 0 -1 4202560 42 931 0 0 0 0 0 0 18 0 10 0 617097460 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 3 0 0 0
[pid=22165/tid=22297] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22297/stat : 22297 (python) S 22163 22165 21419 0 -1 4202560 37 931 0 0 0 0 0 0 18 0 10 0 617097464 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22329] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22329/stat : 22329 (python) S 22163 22165 21419 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 10 0 617155989 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22332] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22332/stat : 22332 (python) S 22163 22165 21419 0 -1 4202560 32 931 0 0 0 0 0 0 18 0 10 0 617156173 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
[pid=22165/tid=22372] ppid=22163 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/22165/task/22372/stat : 22372 (python) S 22163 22165 21419 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 10 0 617205069 380616704 7359 33554432000 4194304 5683644 140733307756160 18446744073709551615 252469889457 0 0 16781312 2 18446604465643423552 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 6278644

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22165 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=21844
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=133979
# CPU time returned by wait4() is 1.15582
# while last known CPU time is 4800.23
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1203.06
CPU time (s): 4800.23
CPU user time (s): 4796.01
CPU system time (s): 4.22
CPU usage (%): 399.001
Max. virtual memory (cumulated for all children) (KiB): 6278644

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.02184
system time used= 0.133979
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18478
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2638
involuntary context switches= 47

runsolver used 2.95555 second user time and 6.07408 second system time

The end

Launcher Data 

Begin job on node111 at 2011-05-01 18:49:30
IDJOB=3317274
IDBENCH=82739
IDSOLVER=1705
FILE ID=node111/3317274-1304268570
RUNJOBID= node111-1304267292-21438
PBS_JOBID= 13173967
Free space on /tmp= 72496 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S396831120-092.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317274-1304268570/watcher-3317274-1304268570 -o /tmp/evaluation-result-3317274-1304268570/solver-3317274-1304268570 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317274-1304268570.cnf -seed 1647509017 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1c5680fb79b6ea8965a1c94337260953
RANDOM SEED=1647509017

node111.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5428.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28557192 kB
Buffers:       1050720 kB
Cached:        2494464 kB
SwapCached:          0 kB
Active:         739660 kB
Inactive:      3186664 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28557192 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:           40296 kB
Writeback:           0 kB
AnonPages:      380904 kB
Mapped:          24460 kB
Slab:           393744 kB
PageTables:      13556 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 26179080 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72464 MiB
End job on node111 at 2011-05-01 19:09:35