Trace number 3385836

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (TO) 2735.09 5100.05

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v800-c3408-S91768321-029.SATISFIABLE.cnf
MD5SUMa8d280a0ad5a42e6946605bb6356e226
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.021996
Satisfiable
(Un)Satisfiability was proved
Number of variables800
Number of clauses3408
Sum of the clauses size10224
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 33408
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.36/0.83	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.83	c INFO: solving HOME/instance-3385836-1308048014.cnf
0.36/0.83	c INFO: computing task features
0.36/0.84	c INFO: computed variable-clause graph statistics
0.36/0.84	c INFO: computed clause balance statistics
0.45/0.90	c INFO: computed variable balance statistics
0.45/0.95	c INFO: computed small-clause counts
0.45/0.95	c INFO: computed Horn-clause counts
0.45/0.96	c INFO: computed variable graph statistics
0.45/0.96	c INFO: collected features for HOME/instance-3385836-1308048014.cnf in 0.13 s
0.45/0.99	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.59)
0.55/1.02	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.08)
0.55/1.05	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.25)
0.55/1.08	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.26)
0.72/1.11	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.23)
0.72/1.14	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.20)
0.72/1.18	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.18)
0.81/1.21	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.13)
0.81/1.22	Traceback (most recent call last):
0.81/1.22	  File "HOME/solve", line 33, in <module>
0.81/1.22	    plac.call(main)
0.81/1.22	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
0.81/1.22	    cmd, result = parser_from(obj).consume(arglist)
0.81/1.22	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
0.81/1.22	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
0.81/1.22	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
0.81/1.22	    (_, answer, _) = solver(input_path, budget / cores, cores)
0.81/1.22	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
0.81/1.22	    process.die()
0.81/1.22	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
0.81/1.22	    os.kill(self._process.pid, signal.SIGUSR1)
0.81/1.22	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385836-1308048014/watcher-3385836-1308048014 -o /tmp/evaluation-result-3385836-1308048014/solver-3385836-1308048014 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385836-1308048014.cnf -seed 275145946 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.00 4.63 4.69 1/170 5752
/proc/meminfo: memFree=31432852/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=15184 CPUtime=0 cores=0-7
/proc/5752/stat : 5752 (python) R 5750 5752 5450 0 -1 4202496 735 0 0 0 0 0 0 0 25 0 1 0 916398078 15548416 622 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500689920 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/5752/statm: 3796 622 322 364 0 283 0

[startup+0.053661 s]
/proc/loadavg: 1.00 4.63 4.69 1/170 5752
/proc/meminfo: memFree=31432852/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=29788 CPUtime=0.04 cores=0-7
/proc/5752/stat : 5752 (python) R 5750 5752 5450 0 -1 4202496 1457 0 0 0 3 1 0 0 25 0 1 0 916398078 30502912 1296 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500689920 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/5752/statm: 7447 1296 443 364 0 817 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 29788

[startup+0.100658 s]
/proc/loadavg: 1.00 4.63 4.69 1/170 5752
/proc/meminfo: memFree=31432852/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=101884 CPUtime=0.07 cores=0-7
/proc/5752/stat : 5752 (python) R 5750 5752 5450 0 -1 4202496 2956 0 0 0 5 2 0 0 25 0 1 0 916398078 104329216 2649 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500352709 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/5752/statm: 25470 2651 837 364 0 1770 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 101884

[startup+0.3007 s]
/proc/loadavg: 1.00 4.63 4.69 1/170 5752
/proc/meminfo: memFree=31432852/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=115432 CPUtime=0.1 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 3598 0 0 0 7 3 0 0 24 0 1 0 916398078 118202368 3237 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500689920 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/5752/statm: 28858 3237 889 364 0 2359 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 115432

[startup+0.700533 s]
/proc/loadavg: 1.00 4.63 4.69 1/170 5752
/proc/meminfo: memFree=31432852/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=226788 CPUtime=0.28 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 7648 933 1 0 20 8 0 0 17 0 1 0 916398078 232230912 5616 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500689920 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/5752/statm: 56697 5616 1566 364 0 4235 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 226788

[startup+1.50037 s]
/proc/loadavg: 1.00 4.63 4.69 3/174 5760
/proc/meminfo: memFree=31408292/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=310124 CPUtime=0.83 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 15935 4019 1 0 65 12 5 1 21 0 1 0 916398078 317566976 7299 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512522047 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/5752/statm: 77531 7299 1891 364 0 13428 0
Current children cumulated CPU time (s) 0.83
Current children cumulated vsize (KiB) 310124

[startup+3.10111 s]
/proc/loadavg: 1.00 4.63 4.69 7/183 5783
/proc/meminfo: memFree=31375856/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=310124 CPUtime=0.83 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 15935 4019 1 0 65 12 5 1 21 0 1 0 916398078 317566976 7299 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512522047 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/5752/statm: 77531 7299 1891 364 0 13428 0
[pid=5765] ppid=5752 vsize=299376 CPUtime=0 cores=0-7
/proc/5765/stat : 5765 (python) S 5752 5752 5450 0 -1 4202560 750 0 0 0 0 0 0 0 15 0 1 0 916398183 306561024 5629 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 578149606458622210 0 0 17 1 0 0 0
/proc/5765/statm: 74844 5629 348 364 0 10756 0
[pid=5767] ppid=5765 vsize=3468 CPUtime=2.04 cores=0-7
/proc/5767/stat : 5767 (march_hi) R 5765 5767 5767 0 -1 4202496 892 0 0 0 204 0 0 0 25 0 1 0 916398183 3551232 630 33554432000 4194304 4955518 140735009813200 18446744073709551615 4227831 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5767/statm: 867 630 104 186 0 680 0
[pid=5769] ppid=5752 vsize=310076 CPUtime=0 cores=0-7
/proc/5769/stat : 5769 (python) S 5752 5752 5450 0 -1 4202560 755 0 0 0 0 0 0 0 15 0 1 0 916398186 317517824 5740 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 12511263651920838916 0 0 17 4 0 0 0
/proc/5769/statm: 77519 5740 348 364 0 13428 0
[pid=5771] ppid=5769 vsize=3468 CPUtime=2.01 cores=0-7
/proc/5771/stat : 5771 (march_hi) R 5769 5771 5771 0 -1 4202496 887 0 0 0 201 0 0 0 25 0 1 0 916398186 3551232 627 33554432000 4194304 4955518 140734330987952 18446744073709551615 4238320 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5771/statm: 867 627 104 186 0 680 0
[pid=5772] ppid=5752 vsize=320332 CPUtime=0 cores=0-7
/proc/5772/stat : 5772 (python) S 5752 5752 5450 0 -1 4202560 773 0 0 0 0 0 0 0 15 0 1 0 916398189 328019968 5744 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 4260827464252489987 0 0 17 4 0 0 0
/proc/5772/statm: 80083 5744 348 364 0 15989 0
[pid=5774] ppid=5772 vsize=3316 CPUtime=1.97 cores=0-7
/proc/5774/stat : 5774 (march_hi) R 5772 5774 5774 0 -1 4202496 891 0 0 0 197 0 0 0 25 0 1 0 916398189 3395584 626 33554432000 4194304 4955518 140737189765856 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5774/statm: 829 626 104 186 0 642 0
[pid=5775] ppid=5752 vsize=330588 CPUtime=0 cores=0-7
/proc/5775/stat : 5775 (python) S 5752 5752 5450 0 -1 4202560 771 0 0 0 0 0 0 0 15 0 1 0 916398192 338522112 5748 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 13836694132879229186 0 0 17 4 0 0 0
/proc/5775/statm: 82647 5748 348 364 0 18550 0
[pid=5777] ppid=5775 vsize=3316 CPUtime=1.94 cores=0-7
/proc/5777/stat : 5777 (march_hi) R 5775 5777 5777 0 -1 4202496 889 0 0 0 194 0 0 0 25 0 1 0 916398193 3395584 624 33554432000 4194304 4955518 140733679033600 18446744073709551615 4227500 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/5777/statm: 829 624 104 186 0 642 0
[pid=5778] ppid=5752 vsize=340844 CPUtime=0 cores=0-7
/proc/5778/stat : 5778 (python) S 5752 5752 5450 0 -1 4202560 783 0 0 0 0 0 0 0 15 0 1 0 916398196 349024256 5768 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 4260809872066445570 0 0 17 0 0 0 0
/proc/5778/statm: 85211 5768 348 364 0 21111 0
[pid=5780] ppid=5778 vsize=3316 CPUtime=1.91 cores=0-7
/proc/5780/stat : 5780 (march_hi) R 5778 5780 5780 0 -1 4202496 891 0 0 0 191 0 0 0 25 0 1 0 916398196 3395584 623 33554432000 4194304 4955518 140737246910032 18446744073709551615 4224512 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/5780/statm: 829 623 104 186 0 642 0
[pid=5781] ppid=5752 vsize=351100 CPUtime=0 cores=0-7
/proc/5781/stat : 5781 (python) S 5752 5752 5450 0 -1 4202560 817 0 0 0 0 0 0 0 15 0 1 0 916398200 359526400 5778 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 13836698530925740291 0 0 17 4 0 0 0
/proc/5781/statm: 87775 5778 354 364 0 23672 0
[pid=5783] ppid=5781 vsize=3316 CPUtime=1.87 cores=0-7
/proc/5783/stat : 5783 (march_hi) R 5781 5783 5783 0 -1 4202496 885 0 0 0 187 0 0 0 25 0 1 0 916398200 3395584 613 33554432000 4194304 4955518 140733861796992 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/5783/statm: 829 613 104 186 0 642 0
Current children cumulated CPU time (s) 12.57
Current children cumulated vsize (KiB) 2282640

[startup+6.30144 s]
/proc/loadavg: 1.40 4.65 4.69 7/183 5783
/proc/meminfo: memFree=31374128/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=310124 CPUtime=0.83 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 15935 4019 1 0 65 12 5 1 21 0 1 0 916398078 317566976 7299 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512522047 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/5752/statm: 77531 7299 1891 364 0 13428 0
[pid=5765] ppid=5752 vsize=299376 CPUtime=0 cores=0-7
/proc/5765/stat : 5765 (python) S 5752 5752 5450 0 -1 4202560 912 0 0 0 0 0 0 0 15 0 1 0 916398183 306561024 5652 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 578149606458622210 0 0 17 1 0 0 0
/proc/5765/statm: 74844 5652 371 364 0 10756 0
[pid=5767] ppid=5765 vsize=4172 CPUtime=5.24 cores=0-7
/proc/5767/stat : 5767 (march_hi) R 5765 5767 5767 0 -1 4202496 1036 0 0 0 524 0 0 0 25 0 1 0 916398183 4272128 774 33554432000 4194304 4955518 140735009813200 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5767/statm: 1043 774 104 186 0 856 0
[pid=5769] ppid=5752 vsize=310076 CPUtime=0 cores=0-7
/proc/5769/stat : 5769 (python) S 5752 5752 5450 0 -1 4202560 926 0 0 0 0 0 0 0 15 0 1 0 916398186 317517824 5762 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 12511263651920838916 0 0 17 4 0 0 0
/proc/5769/statm: 77519 5762 370 364 0 13428 0
[pid=5771] ppid=5769 vsize=4172 CPUtime=5.21 cores=0-7
/proc/5771/stat : 5771 (march_hi) R 5769 5771 5771 0 -1 4202496 1032 0 0 0 521 0 0 0 25 0 1 0 916398186 4272128 772 33554432000 4194304 4955518 140734330987952 18446744073709551615 4298080 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5771/statm: 1043 772 104 186 0 856 0
[pid=5772] ppid=5752 vsize=320332 CPUtime=0 cores=0-7
/proc/5772/stat : 5772 (python) S 5752 5752 5450 0 -1 4202560 927 0 0 0 0 0 0 0 15 0 1 0 916398189 328019968 5767 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 4260827464252489987 0 0 17 4 0 0 0
/proc/5772/statm: 80083 5767 371 364 0 15989 0
[pid=5774] ppid=5772 vsize=4172 CPUtime=5.17 cores=0-7
/proc/5774/stat : 5774 (march_hi) R 5772 5774 5774 0 -1 4202496 1036 0 0 0 517 0 0 0 25 0 1 0 916398189 4272128 771 33554432000 4194304 4955518 140737189765856 18446744073709551615 4235362 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/5774/statm: 1043 771 104 186 0 856 0
[pid=5775] ppid=5752 vsize=330588 CPUtime=0 cores=0-7
/proc/5775/stat : 5775 (python) S 5752 5752 5450 0 -1 4202560 936 0 0 0 0 0 0 0 15 0 1 0 916398192 338522112 5771 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 13836694132879229186 0 0 17 4 0 0 0
/proc/5775/statm: 82647 5771 371 364 0 18550 0
[pid=5777] ppid=5775 vsize=4172 CPUtime=5.14 cores=0-7
/proc/5777/stat : 5777 (march_hi) R 5775 5777 5777 0 -1 4202496 1034 0 0 0 514 0 0 0 25 0 1 0 916398193 4272128 769 33554432000 4194304 4955518 140733679033600 18446744073709551615 4244449 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/5777/statm: 1043 769 104 186 0 856 0
[pid=5778] ppid=5752 vsize=340844 CPUtime=0 cores=0-7
/proc/5778/stat : 5778 (python) S 5752 5752 5450 0 -1 4202560 956 0 0 0 0 0 0 0 15 0 1 0 916398196 349024256 5790 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 4260809872066445570 0 0 17 0 0 0 0
/proc/5778/statm: 85211 5790 370 364 0 21111 0
[pid=5780] ppid=5778 vsize=4172 CPUtime=5.11 cores=0-7
/proc/5780/stat : 5780 (march_hi) R 5778 5780 5780 0 -1 4202496 1035 0 0 0 511 0 0 0 25 0 1 0 916398196 4272128 767 33554432000 4194304 4955518 140737246910032 18446744073709551615 4263731 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/5780/statm: 1043 767 104 186 0 856 0
[pid=5781] ppid=5752 vsize=351100 CPUtime=0 cores=0-7
/proc/5781/stat : 5781 (python) S 5752 5752 5450 0 -1 4202560 997 0 0 0 0 0 0 0 15 0 1 0 916398200 359526400 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 2793647119607496963 0 0 17 4 0 0 0
/proc/5781/statm: 87775 5801 377 364 0 23672 0
[pid=5783] ppid=5781 vsize=3832 CPUtime=5.06 cores=0-7
/proc/5783/stat : 5783 (march_hi) R 5781 5783 5783 0 -1 4202496 1018 0 0 0 506 0 0 0 25 0 1 0 916398200 3923968 746 33554432000 4194304 4955518 140733861796992 18446744073709551615 4224608 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/5783/statm: 958 746 104 186 0 771 0
Current children cumulated CPU time (s) 31.76
Current children cumulated vsize (KiB) 2287132

[startup+12.7012 s]
/proc/loadavg: 1.77 4.67 4.70 7/183 5783
/proc/meminfo: memFree=31367100/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=310124 CPUtime=0.83 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 15935 4019 1 0 65 12 5 1 21 0 1 0 916398078 317566976 7299 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512522047 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/5752/statm: 77531 7299 1891 364 0 13428 0
[pid=5765] ppid=5752 vsize=299376 CPUtime=0 cores=0-7
/proc/5765/stat : 5765 (python) S 5752 5752 5450 0 -1 4202560 917 0 0 0 0 0 0 0 15 0 1 0 916398183 306561024 5652 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 578149606458622210 0 0 17 1 0 0 0
/proc/5765/statm: 74844 5652 371 364 0 10756 0
[pid=5767] ppid=5765 vsize=5936 CPUtime=11.64 cores=0-7
/proc/5767/stat : 5767 (march_hi) R 5765 5767 5767 0 -1 4202496 1352 0 0 0 1164 0 0 0 25 0 1 0 916398183 6078464 1090 33554432000 4194304 4955518 140735009813200 18446744073709551615 4346482 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5767/statm: 1484 1090 104 186 0 1297 0
[pid=5769] ppid=5752 vsize=310076 CPUtime=0 cores=0-7
/proc/5769/stat : 5769 (python) S 5752 5752 5450 0 -1 4202560 937 0 0 0 0 0 0 0 15 0 1 0 916398186 317517824 5763 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 12511263651920838916 0 0 17 4 0 0 0
/proc/5769/statm: 77519 5763 371 364 0 13428 0
[pid=5771] ppid=5769 vsize=5936 CPUtime=11.61 cores=0-7
/proc/5771/stat : 5771 (march_hi) R 5769 5771 5771 0 -1 4202496 1348 0 0 0 1161 0 0 0 25 0 1 0 916398186 6078464 1088 33554432000 4194304 4955518 140734330987952 18446744073709551615 4227493 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/5771/statm: 1484 1088 104 186 0 1297 0
[pid=5772] ppid=5752 vsize=320332 CPUtime=0 cores=0-7
/proc/5772/stat : 5772 (python) S 5752 5752 5450 0 -1 4202560 932 0 0 0 0 0 0 0 15 0 1 0 916398189 328019968 5767 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 4260827464252489987 0 0 17 4 0 0 0
/proc/5772/statm: 80083 5767 371 364 0 15989 0
[pid=5774] ppid=5772 vsize=5936 CPUtime=11.56 cores=0-7
/proc/5774/stat : 5774 (march_hi) R 5772 5774 5774 0 -1 4202496 1352 0 0 0 1156 0 0 0 25 0 1 0 916398189 6078464 1087 33554432000 4194304 4955518 140737189765856 18446744073709551615 4298048 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/5774/statm: 1484 1087 104 186 0 1297 0
[pid=5775] ppid=5752 vsize=330588 CPUtime=0 cores=0-7
/proc/5775/stat : 5775 (python) S 5752 5752 5450 0 -1 4202560 941 0 0 0 0 0 0 0 15 0 1 0 916398192 338522112 5771 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 13836694132879229186 0 0 17 4 0 0 0
/proc/5775/statm: 82647 5771 371 364 0 18550 0
[pid=5777] ppid=5775 vsize=5936 CPUtime=11.54 cores=0-7
/proc/5777/stat : 5777 (march_hi) R 5775 5777 5777 0 -1 4202496 1350 0 0 0 1154 0 0 0 25 0 1 0 916398193 6078464 1085 33554432000 4194304 4955518 140733679033600 18446744073709551615 4263703 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/5777/statm: 1484 1085 104 186 0 1297 0
[pid=5778] ppid=5752 vsize=340844 CPUtime=0.01 cores=0-7
/proc/5778/stat : 5778 (python) S 5752 5752 5450 0 -1 4202560 962 0 0 0 1 0 0 0 15 0 1 0 916398196 349024256 5791 33554432000 4194304 5683644 140737220460144 18446744073709551615 254500712006 0 0 16781312 514 4260809872066445570 0 0 17 0 0 0 0
/proc/5778/statm: 85211 5791 371 364 0 21111 0
[pid=5780] ppid=5778 vsize=5936 CPUtime=11.5 cores=0-7
/proc/5780/stat : 5780 (march_hi) R 5778 5780 5780 0 -1 4202496 1349 0 0 0 1150 0 0 0 25 0 1 0 916398196 6078464 1081 33554432000 4194304 4955518 140737246910032 18446744073709551615 4227671 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/5780/statm: 1484 1081 104 186 0 1297 0
[pid=5781] ppid=5752 vsize=351100 CPUtime=0 cores=0-7

################
# More data... #
################

/proc/5774/stat : 5774 (march_hi) T 5772 5774 5774 0 -1 4202496 19466 0 0 0 48793 3 0 0 25 0 1 0 916398189 117424128 19201 33554432000 4194304 4955518 140737189765856 18446744073709551615 4314233 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5774/statm: 28668 19201 104 186 0 28481 0
[pid=5772/tid=5801] ppid=5752 vsize=320332 CPUtime=0 cores=0-7
/proc/5772/task/5801/stat : 5801 (python) S 5752 5752 5450 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916447020 328019968 5797 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453367916416 0 0 -1 4 0 0 0
[pid=5775] ppid=5752 vsize=330588 CPUtime=0.58 cores=0-7
/proc/5775/stat : 5775 (python) S 5752 5752 5450 0 -1 4202560 1170 0 0 0 34 24 0 0 16 0 2 0 916398192 338522112 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5775/statm: 82647 5801 400 364 0 18550 0
[pid=5777] ppid=5775 vsize=85204 CPUtime=485.29 cores=0-7
/proc/5777/stat : 5777 (march_hi) T 5775 5777 5777 0 -1 4202496 18262 0 0 0 48523 6 0 0 25 0 1 0 916398193 87248896 17997 33554432000 4194304 4955518 140733679033600 18446744073709551615 4261383 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5777/statm: 21301 17997 104 186 0 21114 0
[pid=5775/tid=5799] ppid=5752 vsize=330588 CPUtime=0 cores=0-7
/proc/5775/task/5799/stat : 5799 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916446751 338522112 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453906375808 0 0 -1 4 0 0 0
[pid=5778] ppid=5752 vsize=340844 CPUtime=0.66 cores=0-7
/proc/5778/stat : 5778 (python) S 5752 5752 5450 0 -1 4202560 1190 0 0 0 39 27 0 0 16 0 2 0 916398196 349024256 5821 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/5778/statm: 85211 5821 400 364 0 21111 0
[pid=5780] ppid=5778 vsize=85204 CPUtime=486.96 cores=0-7
/proc/5780/stat : 5780 (march_hi) T 5778 5780 5780 0 -1 4202496 19279 0 0 0 48692 4 0 0 25 0 1 0 916398196 87248896 19011 33554432000 4194304 4955518 140737246910032 18446744073709551615 4227735 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5780/statm: 21301 19011 104 186 0 21114 0
[pid=5778/tid=5800] ppid=5752 vsize=340844 CPUtime=0 cores=0-7
/proc/5778/task/5800/stat : 5800 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916446892 349024256 5821 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453864917120 0 0 -1 0 0 0 0
[pid=5781] ppid=5752 vsize=351100 CPUtime=0.47 cores=0-7
/proc/5781/stat : 5781 (python) S 5752 5752 5450 0 -1 4202560 1227 0 0 0 27 20 0 0 16 0 2 0 916398200 359526400 5831 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5781/statm: 87775 5831 406 364 0 23672 0
[pid=5783] ppid=5781 vsize=85204 CPUtime=389.39 cores=0-7
/proc/5783/stat : 5783 (march_hi) T 5781 5783 5783 0 -1 4202496 15773 0 0 0 38936 3 0 0 25 0 1 0 916398200 87248896 15501 33554432000 4194304 4955518 140733861796992 18446744073709551615 4227671 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5783/statm: 21301 15501 104 186 0 21114 0
[pid=5781/tid=5794] ppid=5752 vsize=351100 CPUtime=0 cores=0-7
/proc/5781/task/5794/stat : 5794 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916437143 359526400 5831 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453563139904 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2735.09
Current children cumulated vsize (KiB) 2832600
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/189 5955
/proc/meminfo: memFree=30960228/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=310124 CPUtime=0.83 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 15935 4019 1 0 65 12 5 1 21 0 1 0 916398078 317566976 7299 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512522047 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/5752/statm: 77531 7299 1891 364 0 13428 0
[pid=5765] ppid=5752 vsize=299376 CPUtime=0.47 cores=0-7
/proc/5765/stat : 5765 (python) S 5752 5752 5450 0 -1 4202560 1156 0 0 0 23 24 0 0 16 0 2 0 916398183 306561024 5682 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5765/statm: 74844 5682 400 364 0 10756 0
[pid=5767] ppid=5765 vsize=85204 CPUtime=392.42 cores=0-7
/proc/5767/stat : 5767 (march_hi) T 5765 5767 5767 0 -1 4202496 15952 0 0 0 39238 4 0 0 25 0 1 0 916398183 87248896 15690 33554432000 4194304 4955518 140735009813200 18446744073709551615 4290777 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5767/statm: 21301 15690 104 186 0 21114 0
[pid=5765/tid=5795] ppid=5752 vsize=299376 CPUtime=0 cores=0-7
/proc/5765/task/5795/stat : 5795 (python) S 5752 5752 5450 0 -1 4202560 44 0 0 0 0 0 0 0 16 0 2 0 916437424 306561024 5682 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453861997376 0 0 -1 2 0 0 0
[pid=5769] ppid=5752 vsize=310076 CPUtime=0.58 cores=0-7
/proc/5769/stat : 5769 (python) S 5752 5752 5450 0 -1 4202560 1166 0 0 0 34 24 0 0 16 0 2 0 916398186 317517824 5793 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5769/statm: 77519 5793 400 364 0 13428 0
[pid=5771] ppid=5769 vsize=114672 CPUtime=488.9 cores=0-7
/proc/5771/stat : 5771 (march_hi) T 5769 5771 5771 0 -1 4202496 19448 0 0 0 48887 3 0 0 25 0 1 0 916398186 117424128 19188 33554432000 4194304 4955518 140734330987952 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5771/statm: 28668 19188 104 186 0 28481 0
[pid=5769/tid=5802] ppid=5752 vsize=310076 CPUtime=0 cores=0-7
/proc/5769/task/5802/stat : 5802 (python) S 5752 5752 5450 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916447075 317517824 5793 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453563138112 0 0 -1 4 0 0 0
[pid=5772] ppid=5752 vsize=320332 CPUtime=0.58 cores=0-7
/proc/5772/stat : 5772 (python) S 5752 5752 5450 0 -1 4202560 1167 0 0 0 31 27 0 0 16 0 2 0 916398189 328019968 5797 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5772/statm: 80083 5797 400 364 0 15989 0
[pid=5774] ppid=5772 vsize=114672 CPUtime=487.96 cores=0-7
/proc/5774/stat : 5774 (march_hi) T 5772 5774 5774 0 -1 4202496 19466 0 0 0 48793 3 0 0 25 0 1 0 916398189 117424128 19201 33554432000 4194304 4955518 140737189765856 18446744073709551615 4314233 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5774/statm: 28668 19201 104 186 0 28481 0
[pid=5772/tid=5801] ppid=5752 vsize=320332 CPUtime=0 cores=0-7
/proc/5772/task/5801/stat : 5801 (python) S 5752 5752 5450 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916447020 328019968 5797 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453367916416 0 0 -1 4 0 0 0
[pid=5775] ppid=5752 vsize=330588 CPUtime=0.58 cores=0-7
/proc/5775/stat : 5775 (python) S 5752 5752 5450 0 -1 4202560 1170 0 0 0 34 24 0 0 16 0 2 0 916398192 338522112 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5775/statm: 82647 5801 400 364 0 18550 0
[pid=5777] ppid=5775 vsize=85204 CPUtime=485.29 cores=0-7
/proc/5777/stat : 5777 (march_hi) T 5775 5777 5777 0 -1 4202496 18262 0 0 0 48523 6 0 0 25 0 1 0 916398193 87248896 17997 33554432000 4194304 4955518 140733679033600 18446744073709551615 4261383 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5777/statm: 21301 17997 104 186 0 21114 0
[pid=5775/tid=5799] ppid=5752 vsize=330588 CPUtime=0 cores=0-7
/proc/5775/task/5799/stat : 5799 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916446751 338522112 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453906375808 0 0 -1 4 0 0 0
[pid=5778] ppid=5752 vsize=340844 CPUtime=0.66 cores=0-7
/proc/5778/stat : 5778 (python) S 5752 5752 5450 0 -1 4202560 1190 0 0 0 39 27 0 0 16 0 2 0 916398196 349024256 5821 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/5778/statm: 85211 5821 400 364 0 21111 0
[pid=5780] ppid=5778 vsize=85204 CPUtime=486.96 cores=0-7
/proc/5780/stat : 5780 (march_hi) T 5778 5780 5780 0 -1 4202496 19279 0 0 0 48692 4 0 0 25 0 1 0 916398196 87248896 19011 33554432000 4194304 4955518 140737246910032 18446744073709551615 4227735 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5780/statm: 21301 19011 104 186 0 21114 0
[pid=5778/tid=5800] ppid=5752 vsize=340844 CPUtime=0 cores=0-7
/proc/5778/task/5800/stat : 5800 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916446892 349024256 5821 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453864917120 0 0 -1 0 0 0 0
[pid=5781] ppid=5752 vsize=351100 CPUtime=0.47 cores=0-7
/proc/5781/stat : 5781 (python) S 5752 5752 5450 0 -1 4202560 1227 0 0 0 27 20 0 0 16 0 2 0 916398200 359526400 5831 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5781/statm: 87775 5831 406 364 0 23672 0
[pid=5783] ppid=5781 vsize=85204 CPUtime=389.39 cores=0-7
/proc/5783/stat : 5783 (march_hi) T 5781 5783 5783 0 -1 4202496 15773 0 0 0 38936 3 0 0 25 0 1 0 916398200 87248896 15501 33554432000 4194304 4955518 140733861796992 18446744073709551615 4227671 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5783/statm: 21301 15501 104 186 0 21114 0
[pid=5781/tid=5794] ppid=5752 vsize=351100 CPUtime=0 cores=0-7
/proc/5781/task/5794/stat : 5794 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916437143 359526400 5831 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453563139904 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2735.09
Current children cumulated vsize (KiB) 2832600
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.00 0.00 0.00 1/189 5956
/proc/meminfo: memFree=30960596/32950928 swapFree=67053888/67111528
[pid=5752] ppid=5750 vsize=310124 CPUtime=0.83 cores=0-7
/proc/5752/stat : 5752 (python) S 5750 5752 5450 0 -1 4202496 15935 4019 1 0 65 12 5 1 21 0 1 0 916398078 317566976 7299 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512522047 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/5752/statm: 77531 7299 1891 364 0 13428 0
[pid=5765] ppid=5752 vsize=299376 CPUtime=0.47 cores=0-7
/proc/5765/stat : 5765 (python) S 5752 5752 5450 0 -1 4202560 1156 0 0 0 23 24 0 0 16 0 2 0 916398183 306561024 5682 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5765/statm: 74844 5682 400 364 0 10756 0
[pid=5767] ppid=5765 vsize=85204 CPUtime=392.42 cores=0-7
/proc/5767/stat : 5767 (march_hi) T 5765 5767 5767 0 -1 4202496 15952 0 0 0 39238 4 0 0 25 0 1 0 916398183 87248896 15690 33554432000 4194304 4955518 140735009813200 18446744073709551615 4290777 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5767/statm: 21301 15690 104 186 0 21114 0
[pid=5765/tid=5795] ppid=5752 vsize=299376 CPUtime=0 cores=0-7
/proc/5765/task/5795/stat : 5795 (python) S 5752 5752 5450 0 -1 4202560 44 0 0 0 0 0 0 0 16 0 2 0 916437424 306561024 5682 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453861997376 0 0 -1 2 0 0 0
[pid=5769] ppid=5752 vsize=310076 CPUtime=0.58 cores=0-7
/proc/5769/stat : 5769 (python) S 5752 5752 5450 0 -1 4202560 1166 0 0 0 34 24 0 0 16 0 2 0 916398186 317517824 5793 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5769/statm: 77519 5793 400 364 0 13428 0
[pid=5771] ppid=5769 vsize=114672 CPUtime=488.9 cores=0-7
/proc/5771/stat : 5771 (march_hi) T 5769 5771 5771 0 -1 4202496 19448 0 0 0 48887 3 0 0 25 0 1 0 916398186 117424128 19188 33554432000 4194304 4955518 140734330987952 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5771/statm: 28668 19188 104 186 0 28481 0
[pid=5769/tid=5802] ppid=5752 vsize=310076 CPUtime=0 cores=0-7
/proc/5769/task/5802/stat : 5802 (python) S 5752 5752 5450 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916447075 317517824 5793 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453563138112 0 0 -1 4 0 0 0
[pid=5772] ppid=5752 vsize=320332 CPUtime=0.58 cores=0-7
/proc/5772/stat : 5772 (python) S 5752 5752 5450 0 -1 4202560 1167 0 0 0 31 27 0 0 16 0 2 0 916398189 328019968 5797 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5772/statm: 80083 5797 400 364 0 15989 0
[pid=5774] ppid=5772 vsize=114672 CPUtime=487.96 cores=0-7
/proc/5774/stat : 5774 (march_hi) T 5772 5774 5774 0 -1 4202496 19466 0 0 0 48793 3 0 0 25 0 1 0 916398189 117424128 19201 33554432000 4194304 4955518 140737189765856 18446744073709551615 4314233 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5774/statm: 28668 19201 104 186 0 28481 0
[pid=5772/tid=5801] ppid=5752 vsize=320332 CPUtime=0 cores=0-7
/proc/5772/task/5801/stat : 5801 (python) S 5752 5752 5450 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916447020 328019968 5797 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453367916416 0 0 -1 4 0 0 0
[pid=5775] ppid=5752 vsize=330588 CPUtime=0.58 cores=0-7
/proc/5775/stat : 5775 (python) S 5752 5752 5450 0 -1 4202560 1170 0 0 0 34 24 0 0 16 0 2 0 916398192 338522112 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5775/statm: 82647 5801 400 364 0 18550 0
[pid=5777] ppid=5775 vsize=85204 CPUtime=485.29 cores=0-7
/proc/5777/stat : 5777 (march_hi) T 5775 5777 5777 0 -1 4202496 18262 0 0 0 48523 6 0 0 25 0 1 0 916398193 87248896 17997 33554432000 4194304 4955518 140733679033600 18446744073709551615 4261383 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5777/statm: 21301 17997 104 186 0 21114 0
[pid=5775/tid=5799] ppid=5752 vsize=330588 CPUtime=0 cores=0-7
/proc/5775/task/5799/stat : 5799 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916446751 338522112 5801 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453906375808 0 0 -1 4 0 0 0
[pid=5778] ppid=5752 vsize=340844 CPUtime=0.66 cores=0-7
/proc/5778/stat : 5778 (python) S 5752 5752 5450 0 -1 4202560 1190 0 0 0 39 27 0 0 16 0 2 0 916398196 349024256 5821 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/5778/statm: 85211 5821 400 364 0 21111 0
[pid=5780] ppid=5778 vsize=85204 CPUtime=486.96 cores=0-7
/proc/5780/stat : 5780 (march_hi) T 5778 5780 5780 0 -1 4202496 19279 0 0 0 48692 4 0 0 25 0 1 0 916398196 87248896 19011 33554432000 4194304 4955518 140737246910032 18446744073709551615 4227735 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5780/statm: 21301 19011 104 186 0 21114 0
[pid=5778/tid=5800] ppid=5752 vsize=340844 CPUtime=0 cores=0-7
/proc/5778/task/5800/stat : 5800 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916446892 349024256 5821 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453864917120 0 0 -1 0 0 0 0
[pid=5781] ppid=5752 vsize=351100 CPUtime=0.47 cores=0-7
/proc/5781/stat : 5781 (python) S 5752 5752 5450 0 -1 4202560 1227 0 0 0 27 20 0 0 16 0 2 0 916398200 359526400 5831 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5781/statm: 87775 5831 406 364 0 23672 0
[pid=5783] ppid=5781 vsize=85204 CPUtime=389.39 cores=0-7
/proc/5783/stat : 5783 (march_hi) T 5781 5783 5783 0 -1 4202496 15773 0 0 0 38936 3 0 0 25 0 1 0 916398200 87248896 15501 33554432000 4194304 4955518 140733861796992 18446744073709551615 4227671 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5783/statm: 21301 15501 104 186 0 21114 0
[pid=5781/tid=5794] ppid=5752 vsize=351100 CPUtime=0 cores=0-7
/proc/5781/task/5794/stat : 5794 (python) S 5752 5752 5450 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 916437143 359526400 5831 33554432000 4194304 5683644 140737220460144 18446744073709551615 254512515505 0 0 16781312 514 18446604453563139904 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2735.09
Current children cumulated vsize (KiB) 2832600
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5752 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=612602
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=633751
# CPU time returned by wait4() is 4.24635
# while last known CPU time is 2735.09
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.05
CPU time (s): 2735.09
CPU user time (s): 2733.27
CPU system time (s): 1.82
CPU usage (%): 53.6287
Max. virtual memory (cumulated for all children) (KiB): 2832600

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.6126
system time used= 1.63375
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28960
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4696
involuntary context switches= 55

runsolver used 6.98594 second user time and 18.3542 second system time

The end

Launcher Data 

Begin job on node149 at 2011-06-14 12:40:14
IDJOB=3385836
IDBENCH=83101
IDSOLVER=1870
FILE ID=node149/3385836-1308048014
RUNJOBID= node149-1308048000-5464
PBS_JOBID= 13566191
Free space on /tmp= 74024 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v800-c3408-S91768321-029.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385836-1308048014/watcher-3385836-1308048014 -o /tmp/evaluation-result-3385836-1308048014/solver-3385836-1308048014 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385836-1308048014.cnf -seed 275145946 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a8d280a0ad5a42e6946605bb6356e226
RANDOM SEED=275145946

node149.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31432784 kB
Buffers:        245352 kB
Cached:        1025216 kB
SwapCached:      22948 kB
Active:         443640 kB
Inactive:       857192 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31432784 kB
SwapTotal:    67111528 kB
SwapFree:     67053888 kB
Dirty:           32684 kB
Writeback:           0 kB
AnonPages:       28760 kB
Mapped:           9144 kB
Slab:           146484 kB
PageTables:       5100 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   188432 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73992 MiB
End job on node149 at 2011-06-14 14:05:16