Trace number 3317284

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.39 1227.96

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v150-c3195-S207562876-057.UNKNOWN.cnf
MD5SUM04148c6828d9f9ccd74e3efaedcb8c52
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables150
Number of clauses3195
Sum of the clauses size15975
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 53195
Number of clauses of size over 50

Solver Data 

0.33/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.33/0.45	c INFO: solving HOME/instance-3317284-1304268623.cnf
0.33/0.45	c INFO: computing task features
0.33/0.47	c INFO: computed variable-clause graph statistics
0.33/0.47	c INFO: computed clause balance statistics
0.42/0.55	c INFO: computed variable balance statistics
0.52/0.60	c INFO: computed small-clause counts
0.52/0.60	c INFO: computed Horn-clause counts
0.52/0.60	c INFO: computed variable graph statistics
0.52/0.60	c INFO: collected features for HOME/instance-3317284-1304268623.cnf in 0.15 s
0.52/0.63	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.63)
0.52/0.66	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.08)
0.52/0.69	c INFO: running march_hi@0 for 679 with 4799 remaining (b = 0.35)
0.61/0.72	c INFO: running march_hi@0 for 679 with 4799 remaining (b = 0.31)
297.73/99.87	c INFO: running march_hi@0 for 485 with 4700 remaining (b = 0.23)
492.56/148.88	c INFO: running march_hi@0 for 485 with 4651 remaining (b = 0.21)
2251.82/589.50	c INFO: running march_hi@98 for 291 with 4210 remaining (b = 0.14)
2460.20/641.68	c INFO: running march_hi@489 for 194 with 4158 remaining (b = 0.09)
2637.09/685.93	c INFO: running march_hi@0 for 291 with 4114 remaining (b = 0.14)
2814.79/730.45	c INFO: running march_hi@98 for 291 with 4070 remaining (b = 0.14)
3221.27/832.43	c INFO: running march_hi@1170 for 194 with 3968 remaining (b = 0.11)
3427.71/884.16	c INFO: running march_hi@588 for 97 with 3916 remaining (b = 0.08)
3801.04/977.68	c INFO: running march_hi@681 for 97 with 3822 remaining (b = 0.05)
3807.83/979.31	c INFO: running march_hi@680 for 97 with 3821 remaining (b = 0.04)
3982.72/1023.20	c INFO: running march_hi@288 for 194 with 3777 remaining (b = 0.12)
4011.46/1030.36	c INFO: running march_hi@0 for 388 with 3770 remaining (b = 0.17)
4184.74/1073.72	c INFO: running march_hi@1462 for 679 with 3726 remaining (b = 0.27)
4189.13/1074.86	c INFO: running march_hi@0 for 388 with 3725 remaining (b = 0.15)
4776.02/1221.85	c INFO: running march_hi@485 for 97 with 3578 remaining (b = 0.04)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317284-1304268623/watcher-3317284-1304268623 -o /tmp/evaluation-result-3317284-1304268623/solver-3317284-1304268623 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317284-1304268623.cnf -seed 469319439 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.00 8.68 6.41 7/256 5846
/proc/meminfo: memFree=29579812/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=15772 CPUtime=0 cores=1,3,5,7
/proc/5846/stat : 5846 (python) R 5844 5846 5229 0 -1 4202496 878 0 0 0 0 0 0 0 25 0 1 0 538464669 16150528 759 33554432000 4194304 5683644 140736802156896 18446744073709551615 212435677893 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5846/statm: 3943 759 343 364 0 430 0

[startup+0.09495 s]
/proc/loadavg: 9.00 8.68 6.41 7/256 5846
/proc/meminfo: memFree=29579812/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=104220 CPUtime=0.06 cores=1,3,5,7
/proc/5846/stat : 5846 (python) R 5844 5846 5229 0 -1 4202496 3046 0 0 0 5 1 0 0 25 0 1 0 538464669 106721280 2732 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436012980 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5846/statm: 26054 2731 847 364 0 1802 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104220

[startup+0.100935 s]
/proc/loadavg: 9.00 8.68 6.41 7/256 5846
/proc/meminfo: memFree=29579812/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=104616 CPUtime=0.07 cores=1,3,5,7
/proc/5846/stat : 5846 (python) R 5844 5846 5229 0 -1 4202496 3148 0 0 0 6 1 0 0 25 0 1 0 538464669 107126784 2824 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436016896 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5846/statm: 26154 2825 847 364 0 1902 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104616

[startup+0.300899 s]
/proc/loadavg: 9.00 8.68 6.41 7/256 5846
/proc/meminfo: memFree=29579812/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=198912 CPUtime=0.23 cores=1,3,5,7
/proc/5846/stat : 5846 (python) R 5844 5846 5229 0 -1 4202496 7021 936 1 0 18 5 0 0 18 0 1 0 538464669 203685888 5052 33554432000 4194304 5683644 140736802156896 18446744073709551615 47288156243598 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/5846/statm: 49728 5052 1466 364 0 3755 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 198912

[startup+0.700789 s]
/proc/loadavg: 9.00 8.68 6.41 7/256 5846
/proc/meminfo: memFree=29579812/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=309956 CPUtime=0.61 cores=1,3,5,7
/proc/5846/stat : 5846 (python) R 5844 5846 5229 0 -1 4202496 11857 936 1 0 52 9 0 0 20 0 4 0 538464669 317394944 7242 33554432000 4194304 5683644 140736802156896 18446744073709551615 212435708007 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5846/statm: 77489 7242 1877 364 0 13401 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 309956

[startup+1.50063 s]
/proc/loadavg: 9.00 8.68 6.41 11/269 5863
/proc/meminfo: memFree=29540756/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=320212 CPUtime=0.64 cores=1,3,5,7
/proc/5846/stat : 5846 (python) S 5844 5846 5229 0 -1 4202496 12392 936 1 0 55 9 0 0 22 0 5 0 538464669 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5846/statm: 80053 7250 1881 364 0 15962 0
[pid=5852] ppid=5846 vsize=278784 CPUtime=0 cores=1,3,5,7
/proc/5852/stat : 5852 (python) S 5846 5846 5229 0 -1 4202560 737 0 0 0 0 0 0 0 19 0 1 0 538464732 285474816 5601 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037135 0 0 16781312 514 6444093893544771583 0 0 17 1 0 0 0
/proc/5852/statm: 69696 5601 354 364 0 5614 0
[pid=5854] ppid=5852 vsize=1017940 CPUtime=0.51 cores=1,3,5,7
/proc/5854/stat : 5854 (gnovelty+2) R 5852 5854 5854 0 -1 4202496 454 0 0 0 51 0 0 0 22 0 1 0 538464733 1042370560 191 33554432000 134512640 135663704 4293549600 18446744073709551615 134524888 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5854/statm: 254485 191 113 282 0 254201 0
[pid=5855] ppid=5846 vsize=289320 CPUtime=0 cores=1,3,5,7
/proc/5855/stat : 5855 (python) S 5846 5846 5229 0 -1 4202560 744 0 0 0 0 0 0 0 21 0 1 0 538464735 296263680 5688 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 1342595511030382591 0 0 17 7 0 0 0
/proc/5855/statm: 72330 5688 348 364 0 8245 0
[pid=5857] ppid=5855 vsize=1017940 CPUtime=0.82 cores=1,3,5,7
/proc/5857/stat : 5857 (gnovelty+2) R 5855 5857 5857 0 -1 4202496 454 0 0 0 82 0 0 0 25 0 1 0 538464736 1042370560 191 33554432000 134512640 135663704 4288900080 18446744073709551615 134547430 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5857/statm: 254485 191 113 282 0 254201 0
[pid=5858] ppid=5846 vsize=299104 CPUtime=0 cores=1,3,5,7
/proc/5858/stat : 5858 (python) S 5846 5846 5229 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 538464738 306282496 5589 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 2130110269064446216 0 0 17 1 0 0 0
/proc/5858/statm: 74776 5589 348 364 0 10688 0
[pid=5860] ppid=5858 vsize=2124 CPUtime=0.77 cores=1,3,5,7
/proc/5860/stat : 5860 (march_hi) R 5858 5860 5860 0 -1 4202496 634 0 0 0 77 0 0 0 22 0 1 0 538464741 2174976 359 33554432000 4194304 4955518 140734370743120 18446744073709551615 4224406 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5860/statm: 531 359 106 186 0 344 0
[pid=5861] ppid=5846 vsize=309968 CPUtime=0 cores=1,3,5,7
/proc/5861/stat : 5861 (python) S 5846 5846 5229 0 -1 4202560 769 0 0 0 0 0 0 0 15 0 1 0 538464741 317407232 5713 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 16103768362097541380 0 0 17 1 0 0 0
/proc/5861/statm: 77492 5713 348 364 0 13401 0
[pid=5863] ppid=5861 vsize=1980 CPUtime=0.34 cores=1,3,5,7
/proc/5863/stat : 5863 (march_hi) R 5861 5863 5863 0 -1 4202496 573 0 0 0 34 0 0 0 24 0 1 0 538464742 2027520 299 33554432000 4194304 4955518 140734341455104 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5863/statm: 495 299 106 186 0 308 0
[pid=5846/tid=5853] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5853/stat : 5853 (python) S 5844 5846 5229 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 5 0 538464732 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5856] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5856/stat : 5856 (python) S 5844 5846 5229 0 -1 4202560 21 936 0 0 0 0 0 0 22 0 5 0 538464735 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5859] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5859/stat : 5859 (python) S 5844 5846 5229 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 5 0 538464738 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5862] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5862/stat : 5862 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 22 0 5 0 538464741 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 3537372

[startup+3.10133 s]
/proc/loadavg: 9.24 8.73 6.44 11/269 5863
/proc/meminfo: memFree=29538524/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=320212 CPUtime=0.64 cores=1,3,5,7
/proc/5846/stat : 5846 (python) S 5844 5846 5229 0 -1 4202496 12392 936 1 0 55 9 0 0 22 0 5 0 538464669 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5846/statm: 80053 7250 1881 364 0 15962 0
[pid=5852] ppid=5846 vsize=278784 CPUtime=0 cores=1,3,5,7
/proc/5852/stat : 5852 (python) S 5846 5846 5229 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 538464732 285474816 5601 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037135 0 0 16781312 514 6444093893544739080 0 0 17 1 0 0 0
/proc/5852/statm: 69696 5601 354 364 0 5614 0
[pid=5854] ppid=5852 vsize=1017940 CPUtime=1.31 cores=1,3,5,7
/proc/5854/stat : 5854 (gnovelty+2) R 5852 5854 5854 0 -1 4202496 454 0 0 0 131 0 0 0 25 0 1 0 538464733 1042370560 191 33554432000 134512640 135663704 4293549600 18446744073709551615 134524888 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5854/statm: 254485 191 113 282 0 254201 0
[pid=5855] ppid=5846 vsize=289320 CPUtime=0 cores=1,3,5,7
/proc/5855/stat : 5855 (python) S 5846 5846 5229 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 538464735 296263680 5688 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 1342595511030350086 0 0 17 7 0 0 0
/proc/5855/statm: 72330 5688 348 364 0 8245 0
[pid=5857] ppid=5855 vsize=1017940 CPUtime=2.41 cores=1,3,5,7
/proc/5857/stat : 5857 (gnovelty+2) R 5855 5857 5857 0 -1 4202496 454 0 0 0 241 0 0 0 25 0 1 0 538464736 1042370560 191 33554432000 134512640 135663704 4288900080 18446744073709551615 134545772 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5857/statm: 254485 191 113 282 0 254201 0
[pid=5858] ppid=5846 vsize=299104 CPUtime=0 cores=1,3,5,7
/proc/5858/stat : 5858 (python) S 5846 5846 5229 0 -1 4202560 757 0 0 0 0 0 0 0 15 0 1 0 538464738 306282496 5589 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 2130110269064446216 0 0 17 1 0 0 0
/proc/5858/statm: 74776 5589 348 364 0 10688 0
[pid=5860] ppid=5858 vsize=2604 CPUtime=2.37 cores=1,3,5,7
/proc/5860/stat : 5860 (march_hi) R 5858 5860 5860 0 -1 4202496 730 0 0 0 237 0 0 0 25 0 1 0 538464741 2666496 455 33554432000 4194304 4955518 140734370743120 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5860/statm: 651 455 106 186 0 464 0
[pid=5861] ppid=5846 vsize=309968 CPUtime=0 cores=1,3,5,7
/proc/5861/stat : 5861 (python) S 5846 5846 5229 0 -1 4202560 770 0 0 0 0 0 0 0 15 0 1 0 538464741 317407232 5713 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 8650958316270256127 0 0 17 1 0 0 0
/proc/5861/statm: 77492 5713 348 364 0 13401 0
[pid=5863] ppid=5861 vsize=2224 CPUtime=1.14 cores=1,3,5,7
/proc/5863/stat : 5863 (march_hi) R 5861 5863 5863 0 -1 4202496 655 0 0 0 114 0 0 0 25 0 1 0 538464742 2277376 379 33554432000 4194304 4955518 140734341455104 18446744073709551615 4314904 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5863/statm: 556 379 106 186 0 369 0
[pid=5846/tid=5853] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5853/stat : 5853 (python) S 5844 5846 5229 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 5 0 538464732 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5856] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5856/stat : 5856 (python) S 5844 5846 5229 0 -1 4202560 21 936 0 0 0 0 0 0 22 0 5 0 538464735 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5859] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5859/stat : 5859 (python) S 5844 5846 5229 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 5 0 538464738 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5862] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5862/stat : 5862 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 22 0 5 0 538464741 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 7.87
Current children cumulated vsize (KiB) 3538096

[startup+6.30073 s]
/proc/loadavg: 9.24 8.73 6.44 11/269 5864
/proc/meminfo: memFree=29537064/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=320212 CPUtime=0.64 cores=1,3,5,7
/proc/5846/stat : 5846 (python) S 5844 5846 5229 0 -1 4202496 12392 936 1 0 55 9 0 0 22 0 5 0 538464669 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5846/statm: 80053 7250 1881 364 0 15962 0
[pid=5852] ppid=5846 vsize=278784 CPUtime=0 cores=1,3,5,7
/proc/5852/stat : 5852 (python) S 5846 5846 5229 0 -1 4202560 894 0 0 0 0 0 0 0 18 0 1 0 538464732 285474816 5624 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037135 0 0 16781312 514 6444093893544739080 0 0 17 1 0 0 0
/proc/5852/statm: 69696 5624 377 364 0 5614 0
[pid=5854] ppid=5852 vsize=1017940 CPUtime=2.91 cores=1,3,5,7
/proc/5854/stat : 5854 (gnovelty+2) R 5852 5854 5854 0 -1 4202496 454 0 0 0 291 0 0 0 25 0 1 0 538464733 1042370560 191 33554432000 134512640 135663704 4293549600 18446744073709551615 134545619 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5854/statm: 254485 191 113 282 0 254201 0
[pid=5855] ppid=5846 vsize=289320 CPUtime=0 cores=1,3,5,7
/proc/5855/stat : 5855 (python) S 5846 5846 5229 0 -1 4202560 897 0 0 0 0 0 0 0 18 0 1 0 538464735 296263680 5711 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 1342595511030350086 0 0 17 7 0 0 0
/proc/5855/statm: 72330 5711 371 364 0 8245 0
[pid=5857] ppid=5855 vsize=1017940 CPUtime=5.57 cores=1,3,5,7
/proc/5857/stat : 5857 (gnovelty+2) R 5855 5857 5857 0 -1 4202496 454 0 0 0 557 0 0 0 25 0 1 0 538464736 1042370560 191 33554432000 134512640 135663704 4288900080 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5857/statm: 254485 191 113 282 0 254201 0
[pid=5858] ppid=5846 vsize=299104 CPUtime=0 cores=1,3,5,7
/proc/5858/stat : 5858 (python) S 5846 5846 5229 0 -1 4202560 921 0 0 0 0 0 0 0 15 0 1 0 538464738 306282496 5611 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 2130110269064446216 0 0 17 1 0 0 0
/proc/5858/statm: 74776 5611 370 364 0 10688 0
[pid=5860] ppid=5858 vsize=3456 CPUtime=5.57 cores=1,3,5,7
/proc/5860/stat : 5860 (march_hi) R 5858 5860 5860 0 -1 4202496 920 0 0 0 557 0 0 0 25 0 1 0 538464741 3538944 645 33554432000 4194304 4955518 140734370743120 18446744073709551615 4297490 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5860/statm: 864 645 106 186 0 677 0
[pid=5861] ppid=5846 vsize=309968 CPUtime=0 cores=1,3,5,7
/proc/5861/stat : 5861 (python) S 5846 5846 5229 0 -1 4202560 945 0 0 0 0 0 0 0 15 0 1 0 538464741 317407232 5736 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 16103768362097541380 0 0 17 1 0 0 0
/proc/5861/statm: 77492 5736 371 364 0 13401 0
[pid=5863] ppid=5861 vsize=2604 CPUtime=2.74 cores=1,3,5,7
/proc/5863/stat : 5863 (march_hi) R 5861 5863 5863 0 -1 4202496 753 0 0 0 274 0 0 0 25 0 1 0 538464742 2666496 477 33554432000 4194304 4955518 140734341455104 18446744073709551615 4296778 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5863/statm: 651 477 106 186 0 464 0
[pid=5846/tid=5853] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5853/stat : 5853 (python) S 5844 5846 5229 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 5 0 538464732 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5856] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5856/stat : 5856 (python) S 5844 5846 5229 0 -1 4202560 21 936 0 0 0 0 0 0 22 0 5 0 538464735 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5859] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5859/stat : 5859 (python) S 5844 5846 5229 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 5 0 538464738 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5862] ppid=5844 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5862/stat : 5862 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 22 0 5 0 538464741 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 17.43
Current children cumulated vsize (KiB) 3539328

[startup+12.7005 s]
/proc/loadavg: 9.36 8.77 6.48 11/269 5864
/proc/meminfo: memFree=29534796/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=320212 CPUtime=0.64 cores=1,3,5,7
/proc/5846/stat : 5846 (python) S 5844 5846 5229 0 -1 4202496 12392 936 1 0 55 9 0 0 22 0 5 0 538464669 327897088 7250 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5846/statm: 80053 7250 1881 364 0 15962 0
[pid=5852] ppid=5846 vsize=278784 CPUtime=0.01 cores=1,3,5,7
/proc/5852/stat : 5852 (python) S 5846 5846 5229 0 -1 4202560 896 0 0 0 0 1 0 0 18 0 1 0 538464732 285474816 5624 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037135 0 0 16781312 514 3174351642040434952 0 0 17 1 0 0 0
/proc/5852/statm: 69696 5624 377 364 0 5614 0
[pid=5854] ppid=5852 vsize=1017940 CPUtime=6.11 cores=1,3,5,7
/proc/5854/stat : 5854 (gnovelty+2) R 5852 5854 5854 0 -1 4202496 454 0 0 0 611 0 0 0 25 0 1 0 538464733 1042370560 191 33554432000 134512640 135663704 4293549600 18446744073709551615 134545583 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5854/statm: 254485 191 113 282 0 254201 0
[pid=5855] ppid=5846 vsize=289320 CPUtime=0.01 cores=1,3,5,7

################
# More data... #
################

/proc/5857/stat : 5857 (gnovelty+2) T 5855 5857 5857 0 -1 4202496 454 0 0 0 9831 0 0 0 25 0 1 0 538464736 1042370560 191 33554432000 134512640 135663704 4288900080 18446744073709551615 134525361 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5857/statm: 254485 191 113 282 0 254201 0
[pid=5855/tid=5875] ppid=5846 vsize=289320 CPUtime=0 cores=1,3,5,7
/proc/5855/task/5875/stat : 5875 (python) S 5846 5846 5229 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538474653 296263680 5741 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604470405159808 0 0 -1 7 0 0 0
[pid=5858] ppid=5846 vsize=299104 CPUtime=1.79 cores=1,3,5,7
/proc/5858/stat : 5858 (python) S 5846 5846 5229 0 -1 4202560 1168 0 0 0 96 83 0 0 15 0 2 0 538464738 306282496 5642 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5858/statm: 74776 5642 400 364 0 10688 0
[pid=5860] ppid=5858 vsize=208660 CPUtime=1068.13 cores=1,3,5,7
/proc/5860/stat : 5860 (march_hi) T 5858 5860 5860 0 -1 4202496 49889 0 0 0 106802 11 0 0 25 0 1 0 538464741 213667840 49614 33554432000 4194304 4955518 140734370743120 18446744073709551615 4208667 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5860/statm: 52165 49614 106 186 0 51978 0
[pid=5858/tid=5978] ppid=5846 vsize=299104 CPUtime=0 cores=1,3,5,7
/proc/5858/task/5978/stat : 5978 (python) S 5846 5846 5229 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538533259 306282496 5642 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604467811806016 0 0 -1 1 0 0 0
[pid=5861] ppid=5846 vsize=309968 CPUtime=2.06 cores=1,3,5,7
/proc/5861/stat : 5861 (python) S 5846 5846 5229 0 -1 4202560 1184 0 0 0 115 91 0 0 15 0 2 0 538464741 317407232 5766 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5861/statm: 77492 5766 400 364 0 13401 0
[pid=5863] ppid=5861 vsize=315944 CPUtime=1175.64 cores=1,3,5,7
/proc/5863/stat : 5863 (march_hi) R 5861 5863 5863 0 -1 4202496 54819 0 0 0 117554 10 0 0 25 0 1 0 538464742 323526656 54543 33554432000 4194304 4955518 140734341455104 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5863/statm: 78986 54543 106 186 0 78799 0
[pid=5861/tid=5980] ppid=5846 vsize=309968 CPUtime=0 cores=1,3,5,7
/proc/5861/task/5980/stat : 5980 (python) S 5846 5846 5229 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538537711 317407232 5766 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604467811803328 0 0 -1 1 0 0 0
[pid=5876] ppid=5846 vsize=320224 CPUtime=1.66 cores=1,3,5,7
/proc/5876/stat : 5876 (python) S 5846 5846 5229 0 -1 4202560 1184 0 0 0 92 74 0 0 16 0 2 0 538474656 327909376 5770 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5876/statm: 80056 5770 400 364 0 15962 0
[pid=5878] ppid=5876 vsize=208660 CPUtime=971.85 cores=1,3,5,7
/proc/5878/stat : 5878 (march_hi) T 5876 5878 5878 0 -1 4202496 45415 0 0 0 97178 7 0 0 25 0 1 0 538474657 213667840 45138 33554432000 4194304 4955518 140733745120688 18446744073709551615 4285766 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5878/statm: 52165 45138 106 186 0 51978 0
[pid=5876/tid=5960] ppid=5846 vsize=320224 CPUtime=0 cores=1,3,5,7
/proc/5876/task/5960/stat : 5960 (python) S 5846 5846 5229 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538523616 327909376 5770 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604454224049152 0 0 -1 1 0 0 0
[pid=5881] ppid=5846 vsize=330480 CPUtime=1.73 cores=1,3,5,7
/proc/5881/stat : 5881 (python) S 5846 5846 5229 0 -1 4202560 1191 0 0 0 94 79 0 0 15 0 2 0 538479557 338411520 5795 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5881/statm: 82620 5795 400 364 0 18523 0
[pid=5883] ppid=5881 vsize=208660 CPUtime=1029.37 cores=1,3,5,7
/proc/5883/stat : 5883 (march_hi) R 5881 5883 5883 0 -1 4202496 48163 0 0 0 102926 11 0 0 25 0 1 0 538479557 213667840 47884 33554432000 4194304 4955518 140736376886592 18446744073709551615 4282461 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5883/statm: 52165 47884 106 186 0 51978 0
[pid=5881/tid=5975] ppid=5846 vsize=330480 CPUtime=0 cores=1,3,5,7
/proc/5881/task/5975/stat : 5975 (python) S 5846 5846 5229 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538528835 338411520 5795 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604469499594880 0 0 -1 1 0 0 0
[pid=5998] ppid=5846 vsize=340848 CPUtime=0.36 cores=1,3,5,7
/proc/5998/stat : 5998 (python) S 5846 5846 5229 0 -1 4202560 1016 0 0 0 20 16 0 0 15 0 1 0 538567705 349028352 5776 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 3174351642040434952 0 0 17 1 0 0 0
/proc/5998/statm: 85212 5776 371 364 0 21112 0
[pid=6000] ppid=5998 vsize=43172 CPUtime=196.46 cores=1,3,5,7
/proc/6000/stat : 6000 (march_hi) R 5998 6000 6000 0 -1 4202496 10570 0 0 0 19644 2 0 0 25 0 1 0 538567705 44208128 10290 33554432000 4194304 4955518 140737049824656 18446744073709551615 4208626 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/6000/statm: 10793 10290 106 186 0 10606 0
[pid=6002] ppid=5846 vsize=350572 CPUtime=0.27 cores=1,3,5,7
/proc/6002/stat : 6002 (python) S 5846 5846 5229 0 -1 4202560 1039 0 0 0 13 14 0 0 16 0 1 0 538572155 358985728 5685 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 3174351642040434952 0 0 17 1 0 0 0
/proc/6002/statm: 87643 5685 377 364 0 23540 0
[pid=6004] ppid=6002 vsize=43172 CPUtime=153.03 cores=1,3,5,7
/proc/6004/stat : 6004 (march_hi) R 6002 6004 6004 0 -1 4202496 8433 0 0 0 15301 2 0 0 25 0 1 0 538572155 44208128 8148 33554432000 4194304 4955518 140737419358176 18446744073709551615 4235624 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/6004/statm: 10793 8148 106 186 0 10606 0
[pid=5846/tid=5853] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5853/stat : 5853 (python) S 5844 5846 5229 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 9 0 538464732 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5856] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5856/stat : 5856 (python) S 5844 5846 5229 0 -1 4202560 21 936 0 0 0 0 0 0 22 0 9 0 538464735 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5859] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5859/stat : 5859 (python) S 5844 5846 5229 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 9 0 538464738 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5862] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5862/stat : 5862 (python) S 5844 5846 5229 0 -1 4202560 40 936 0 0 0 0 0 0 18 0 9 0 538464741 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5877] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5877/stat : 5877 (python) S 5844 5846 5229 0 -1 4202560 32 936 0 0 0 0 0 0 18 0 9 0 538474657 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5882] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5882/stat : 5882 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 18 0 9 0 538479557 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5999] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5999/stat : 5999 (python) S 5844 5846 5229 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 9 0 538567705 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=6003] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/6003/stat : 6003 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 9 0 538572155 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 5955144

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1227.9 s]
/proc/loadavg: 9.25 9.14 8.41 10/280 6009
/proc/meminfo: memFree=28630200/32951124 swapFree=67111416/67111528
[pid=5846] ppid=5844 vsize=361452 CPUtime=1.04 cores=1,3,5,7
/proc/5846/stat : 5846 (python) S 5844 5846 5229 0 -1 4202496 17384 936 1 0 94 10 0 0 18 0 9 0 538464669 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 0
/proc/5846/statm: 90363 7329 1890 364 0 26260 0
[pid=5852] ppid=5846 vsize=289028 CPUtime=0.23 cores=1,3,5,7
/proc/5852/stat : 5852 (python) S 5846 5846 5229 0 -1 4202560 1099 0 0 0 13 10 0 0 19 0 2 0 538464732 295964672 5662 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5852/statm: 72257 5662 410 364 0 8175 0
[pid=5854] ppid=5852 vsize=1017940 CPUtime=98.31 cores=1,3,5,7
/proc/5854/stat : 5854 (gnovelty+2) T 5852 5854 5854 0 -1 4202496 454 0 0 0 9831 0 0 0 25 0 1 0 538464733 1042370560 191 33554432000 134512640 135663704 4293549600 18446744073709551615 134524734 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5854/statm: 254485 191 113 282 0 254201 0
[pid=5852/tid=5880] ppid=5846 vsize=289028 CPUtime=0 cores=1,3,5,7
/proc/5852/task/5880/stat : 5880 (python) S 5846 5846 5229 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538479554 295964672 5662 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604467297977664 0 0 -1 1 0 0 0
[pid=5855] ppid=5846 vsize=289320 CPUtime=0.15 cores=1,3,5,7
/proc/5855/stat : 5855 (python) S 5846 5846 5229 0 -1 4202560 1084 0 0 0 7 8 0 0 19 0 2 0 538464735 296263680 5741 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5855/statm: 72330 5741 400 364 0 8245 0
[pid=5857] ppid=5855 vsize=1017940 CPUtime=98.31 cores=1,3,5,7
/proc/5857/stat : 5857 (gnovelty+2) T 5855 5857 5857 0 -1 4202496 454 0 0 0 9831 0 0 0 25 0 1 0 538464736 1042370560 191 33554432000 134512640 135663704 4288900080 18446744073709551615 134525361 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5857/statm: 254485 191 113 282 0 254201 0
[pid=5855/tid=5875] ppid=5846 vsize=289320 CPUtime=0 cores=1,3,5,7
/proc/5855/task/5875/stat : 5875 (python) S 5846 5846 5229 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538474653 296263680 5741 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604470405159808 0 0 -1 7 0 0 0
[pid=5858] ppid=5846 vsize=299104 CPUtime=1.79 cores=1,3,5,7
/proc/5858/stat : 5858 (python) S 5846 5846 5229 0 -1 4202560 1168 0 0 0 96 83 0 0 15 0 2 0 538464738 306282496 5642 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5858/statm: 74776 5642 400 364 0 10688 0
[pid=5860] ppid=5858 vsize=208660 CPUtime=1068.13 cores=1,3,5,7
/proc/5860/stat : 5860 (march_hi) T 5858 5860 5860 0 -1 4202496 49889 0 0 0 106802 11 0 0 25 0 1 0 538464741 213667840 49614 33554432000 4194304 4955518 140734370743120 18446744073709551615 4208667 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5860/statm: 52165 49614 106 186 0 51978 0
[pid=5858/tid=5978] ppid=5846 vsize=299104 CPUtime=0 cores=1,3,5,7
/proc/5858/task/5978/stat : 5978 (python) S 5846 5846 5229 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 538533259 306282496 5642 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604467811806016 0 0 -1 1 0 0 0
[pid=5861] ppid=5846 vsize=309968 CPUtime=2.06 cores=1,3,5,7
/proc/5861/stat : 5861 (python) S 5846 5846 5229 0 -1 4202560 1184 0 0 0 115 91 0 0 15 0 2 0 538464741 317407232 5766 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5861/statm: 77492 5766 400 364 0 13401 0
[pid=5863] ppid=5861 vsize=315944 CPUtime=1175.64 cores=1,3,5,7
/proc/5863/stat : 5863 (march_hi) R 5861 5863 5863 0 -1 4202496 54819 0 0 0 117554 10 0 0 25 0 1 0 538464742 323526656 54543 33554432000 4194304 4955518 140734341455104 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5863/statm: 78986 54543 106 186 0 78799 0
[pid=5861/tid=5980] ppid=5846 vsize=309968 CPUtime=0 cores=1,3,5,7
/proc/5861/task/5980/stat : 5980 (python) S 5846 5846 5229 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538537711 317407232 5766 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604467811803328 0 0 -1 1 0 0 0
[pid=5876] ppid=5846 vsize=320224 CPUtime=1.66 cores=1,3,5,7
/proc/5876/stat : 5876 (python) S 5846 5846 5229 0 -1 4202560 1184 0 0 0 92 74 0 0 16 0 2 0 538474656 327909376 5770 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447843787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5876/statm: 80056 5770 400 364 0 15962 0
[pid=5878] ppid=5876 vsize=208660 CPUtime=971.85 cores=1,3,5,7
/proc/5878/stat : 5878 (march_hi) T 5876 5878 5878 0 -1 4202496 45415 0 0 0 97178 7 0 0 25 0 1 0 538474657 213667840 45138 33554432000 4194304 4955518 140733745120688 18446744073709551615 4285766 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5878/statm: 52165 45138 106 186 0 51978 0
[pid=5876/tid=5960] ppid=5846 vsize=320224 CPUtime=0 cores=1,3,5,7
/proc/5876/task/5960/stat : 5960 (python) S 5846 5846 5229 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538523616 327909376 5770 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604454224049152 0 0 -1 1 0 0 0
[pid=5881] ppid=5846 vsize=330480 CPUtime=1.73 cores=1,3,5,7
/proc/5881/stat : 5881 (python) S 5846 5846 5229 0 -1 4202560 1191 0 0 0 94 79 0 0 15 0 2 0 538479557 338411520 5795 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5881/statm: 82620 5795 400 364 0 18523 0
[pid=5883] ppid=5881 vsize=208660 CPUtime=1029.37 cores=1,3,5,7
/proc/5883/stat : 5883 (march_hi) R 5881 5883 5883 0 -1 4202496 48163 0 0 0 102926 11 0 0 25 0 1 0 538479557 213667840 47884 33554432000 4194304 4955518 140736376886592 18446744073709551615 4282461 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5883/statm: 52165 47884 106 186 0 51978 0
[pid=5881/tid=5975] ppid=5846 vsize=330480 CPUtime=0 cores=1,3,5,7
/proc/5881/task/5975/stat : 5975 (python) S 5846 5846 5229 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538528835 338411520 5795 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 514 18446604469499594880 0 0 -1 1 0 0 0
[pid=5998] ppid=5846 vsize=340848 CPUtime=0.36 cores=1,3,5,7
/proc/5998/stat : 5998 (python) S 5846 5846 5229 0 -1 4202560 1016 0 0 0 20 16 0 0 15 0 1 0 538567705 349028352 5776 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 3174351642040434952 0 0 17 1 0 0 0
/proc/5998/statm: 85212 5776 371 364 0 21112 0
[pid=6000] ppid=5998 vsize=43172 CPUtime=196.46 cores=1,3,5,7
/proc/6000/stat : 6000 (march_hi) R 5998 6000 6000 0 -1 4202496 10570 0 0 0 19644 2 0 0 25 0 1 0 538567705 44208128 10290 33554432000 4194304 4955518 140737049824656 18446744073709551615 4208626 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/6000/statm: 10793 10290 106 186 0 10606 0
[pid=6002] ppid=5846 vsize=350572 CPUtime=0.27 cores=1,3,5,7
/proc/6002/stat : 6002 (python) S 5846 5846 5229 0 -1 4202560 1039 0 0 0 13 14 0 0 16 0 1 0 538572155 358985728 5685 33554432000 4194304 5683644 140736802156896 18446744073709551615 212436037190 0 0 16781312 514 3174351642040434952 0 0 17 1 0 0 0
/proc/6002/statm: 87643 5685 377 364 0 23540 0
[pid=6004] ppid=6002 vsize=43172 CPUtime=153.03 cores=1,3,5,7
/proc/6004/stat : 6004 (march_hi) R 6002 6004 6004 0 -1 4202496 8433 0 0 0 15301 2 0 0 25 0 1 0 538572155 44208128 8148 33554432000 4194304 4955518 140737419358176 18446744073709551615 4235624 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/6004/statm: 10793 8148 106 186 0 10606 0
[pid=5846/tid=5853] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5853/stat : 5853 (python) S 5844 5846 5229 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 9 0 538464732 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5856] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5856/stat : 5856 (python) S 5844 5846 5229 0 -1 4202560 21 936 0 0 0 0 0 0 22 0 9 0 538464735 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 7 0 0 0
[pid=5846/tid=5859] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5859/stat : 5859 (python) S 5844 5846 5229 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 9 0 538464738 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5862] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5862/stat : 5862 (python) S 5844 5846 5229 0 -1 4202560 40 936 0 0 0 0 0 0 18 0 9 0 538464741 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5877] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5877/stat : 5877 (python) S 5844 5846 5229 0 -1 4202560 32 936 0 0 0 0 0 0 18 0 9 0 538474657 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5882] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5882/stat : 5882 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 18 0 9 0 538479557 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=5999] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/5999/stat : 5999 (python) S 5844 5846 5229 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 9 0 538567705 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
[pid=5846/tid=6003] ppid=5844 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/5846/task/6003/stat : 6003 (python) S 5844 5846 5229 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 9 0 538572155 370126848 7329 33554432000 4194304 5683644 140736802156896 18446744073709551615 212447840689 0 0 16781312 2 18446604454224050944 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 5955144

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5846 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=957854
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=111982
# CPU time returned by wait4() is 1.06984
# while last known CPU time is 4800.39
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1227.96
CPU time (s): 4800.39
CPU user time (s): 4796.11
CPU system time (s): 4.28
CPU usage (%): 390.925
Max. virtual memory (cumulated for all children) (KiB): 5955144

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.957854
system time used= 0.111982
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18338
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1121
involuntary context switches= 48

runsolver used 2.88756 second user time and 6.05308 second system time

The end

Launcher Data 

Begin job on node102 at 2011-05-01 18:50:23
IDJOB=3317284
IDBENCH=82749
IDSOLVER=1705
FILE ID=node102/3317284-1304268623
RUNJOBID= node102-1304267295-5248
PBS_JOBID= 13173976
Free space on /tmp= 72692 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v150-c3195-S207562876-057.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317284-1304268623/watcher-3317284-1304268623 -o /tmp/evaluation-result-3317284-1304268623/solver-3317284-1304268623 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317284-1304268623.cnf -seed 469319439 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 04148c6828d9f9ccd74e3efaedcb8c52
RANDOM SEED=469319439

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29579844 kB
Buffers:        650252 kB
Cached:        1523900 kB
SwapCached:          0 kB
Active:        1068696 kB
Inactive:      1863156 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29579844 kB
SwapTotal:    67111528 kB
SwapFree:     67111416 kB
Dirty:           33036 kB
Writeback:           0 kB
AnonPages:      757988 kB
Mapped:          23852 kB
Slab:           359704 kB
PageTables:      19412 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 21141368 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72652 MiB
End job on node102 at 2011-05-01 19:10:53