Trace number 3322142

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 2930.64 5100.04

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v550-c2343-S1114532273-035.SATISFIABLE.cnf
MD5SUMff7e731e0657a2728f5ff73a4fd8bb2e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.015997
Satisfiable
(Un)Satisfiability was proved
Number of variables550
Number of clauses2343
Sum of the clauses size7029
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32343
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.48/3.91	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.48/3.91	c INFO: solving HOME/instance-3322142-1308048002.cnf
0.48/3.91	c INFO: computing task features
0.48/3.91	c INFO: computed variable-clause graph statistics
0.48/3.92	c INFO: computed clause balance statistics
0.48/3.96	c INFO: computed variable balance statistics
0.48/3.99	c INFO: computed small-clause counts
0.48/3.99	c INFO: computed Horn-clause counts
0.48/4.00	c INFO: computed variable graph statistics
0.48/4.00	c INFO: collected features for HOME/instance-3322142-1308048002.cnf in 0.09 s
0.57/4.03	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.57)
0.57/4.07	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.33)
0.68/4.12	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.29)
0.68/4.17	c INFO: running march_hi@0 for 97 with 39999 remaining (b = 0.08)
0.83/4.21	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.08)
0.83/4.26	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.31)
1.54/4.31	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.28)
1.54/4.35	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.25)
1.54/4.36	Traceback (most recent call last):
1.54/4.36	  File "HOME/solve", line 33, in <module>
1.54/4.36	    plac.call(main)
1.54/4.36	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
1.54/4.36	    cmd, result = parser_from(obj).consume(arglist)
1.54/4.36	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
1.54/4.36	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
1.54/4.36	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
1.54/4.36	    (_, answer, _) = solver(input_path, budget, cores)
1.54/4.36	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
1.54/4.36	    process.die()
1.54/4.36	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
1.54/4.36	    os.kill(self._process.pid, signal.SIGUSR1)
1.54/4.36	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322142-1308048002/watcher-3322142-1308048002 -o /tmp/evaluation-result-3322142-1308048002/solver-3322142-1308048002 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322142-1308048002.cnf -seed 1384931477 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.03 1.45 2.47 1/165 17764
/proc/meminfo: memFree=28268928/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=11804 CPUtime=0 cores=0-7
/proc/17764/stat : 17764 (runsolver) S 17762 17764 17734 0 -1 4202560 42 0 0 0 0 0 0 0 21 0 1 0 916400218 12087296 50 33554432000 4194304 5436572 140733743135632 18446744073709551615 4883175 0 2147483391 4096 24578 18446744071699870519 0 0 17 2 0 0 0
/proc/17764/statm: 2951 50 31 304 0 2643 0

[startup+0.0878971 s]
/proc/loadavg: 0.03 1.45 2.47 1/165 17764
/proc/meminfo: memFree=28268928/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=15080 CPUtime=0 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 618 0 0 0 0 0 0 0 22 0 1 0 916400218 15441920 513 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 7 0 0 0
/proc/17764/statm: 3770 513 300 364 0 257 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15080

[startup+0.100852 s]
/proc/loadavg: 0.03 1.45 2.47 1/165 17764
/proc/meminfo: memFree=28268928/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=15184 CPUtime=0 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 714 0 0 0 0 0 0 0 22 0 1 0 916400218 15548416 604 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 7 0 0 0
/proc/17764/statm: 3796 604 319 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.300877 s]
/proc/loadavg: 0.03 1.45 2.47 1/165 17764
/proc/meminfo: memFree=28268928/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=15380 CPUtime=0 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 821 0 0 0 0 0 0 0 20 0 1 0 916400218 15749120 704 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 7 0 0 0
/proc/17764/statm: 3845 704 334 364 0 332 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15380

[startup+0.700791 s]
/proc/loadavg: 0.03 1.45 2.47 1/165 17764
/proc/meminfo: memFree=28268928/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=16396 CPUtime=0 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 1125 0 0 0 0 0 0 0 16 0 1 0 916400218 16789504 990 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 7 0 0 0
/proc/17764/statm: 4099 990 413 364 0 586 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16396

[startup+1.50065 s]
/proc/loadavg: 0.03 1.45 2.47 1/166 17765
/proc/meminfo: memFree=28264580/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=85620 CPUtime=0.05 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 2632 0 0 0 4 1 0 0 15 0 1 0 916400218 87674880 2348 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 7 0 0 0
/proc/17764/statm: 21405 2348 713 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85620

[startup+3.1004 s]
/proc/loadavg: 0.10 1.45 2.46 1/166 17765
/proc/meminfo: memFree=28260064/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=135748 CPUtime=0.28 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 5968 936 1 0 13 4 2 9 15 0 1 0 916400218 139005952 4151 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647142405 0 2147483391 16781312 2 18446744071699109054 0 0 17 7 0 0 0
/proc/17764/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 135748
heavy processes:

[startup+6.30083 s]
/proc/loadavg: 0.10 1.45 2.46 7/178 17795
/proc/meminfo: memFree=28209760/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=310048 CPUtime=1.14 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 15952 3978 1 0 70 16 18 10 18 0 1 0 916400218 317489152 7280 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658976575 0 0 16781312 2 18446744071562232103 0 0 17 7 0 0 0
/proc/17764/statm: 77512 7280 1891 364 0 13409 0
[pid=17773] ppid=17764 vsize=289384 CPUtime=0 cores=0-7
/proc/17773/stat : 17773 (python) S 17764 17764 17734 0 -1 4202560 758 0 0 0 0 0 0 0 15 0 1 0 916400625 296329216 5704 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 2358588785316299016 0 0 17 0 0 0 0
/proc/17773/statm: 72346 5704 348 364 0 8261 0
[pid=17775] ppid=17773 vsize=2812 CPUtime=2.2 cores=0-7
/proc/17775/stat : 17775 (march_hi) R 17773 17775 17775 0 -1 4202496 754 0 0 0 220 0 0 0 25 0 1 0 916400627 2879488 492 33554432000 4194304 4955518 140734052599808 18446744073709551615 4224608 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17775/statm: 703 492 104 186 0 516 0
[pid=17777] ppid=17764 vsize=299268 CPUtime=0 cores=0-7
/proc/17777/stat : 17777 (python) S 17764 17764 17734 0 -1 4202560 765 0 0 0 0 0 0 0 15 0 1 0 916400630 306450432 5598 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 2358584387269787912 0 0 17 0 0 0 0
/proc/17777/statm: 74817 5598 348 364 0 10729 0
[pid=17779] ppid=17777 vsize=2812 CPUtime=2.16 cores=0-7
/proc/17779/stat : 17779 (march_hi) R 17777 17779 17779 0 -1 4202496 756 0 0 0 216 0 0 0 25 0 1 0 916400632 2879488 491 33554432000 4194304 4955518 140737003989024 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17779/statm: 703 491 104 186 0 516 0
[pid=17780] ppid=17764 vsize=310000 CPUtime=0 cores=0-7
/proc/17780/stat : 17780 (python) S 17764 17764 17734 0 -1 4202560 758 0 0 0 0 0 0 0 15 0 1 0 916400635 317440000 5721 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 9618131899893383432 0 0 17 0 0 0 0
/proc/17780/statm: 77500 5721 348 364 0 13409 0
[pid=17782] ppid=17780 vsize=2812 CPUtime=2.12 cores=0-7
/proc/17782/stat : 17782 (march_hi) R 17780 17782 17782 0 -1 4202496 748 0 0 0 212 0 0 0 25 0 1 0 916400635 2879488 487 33554432000 4194304 4955518 140733772423520 18446744073709551615 4235379 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17782/statm: 703 487 104 186 0 516 0
[pid=17786] ppid=17764 vsize=330512 CPUtime=0 cores=0-7
/proc/17786/stat : 17786 (python) S 17764 17764 17734 0 -1 4202560 779 0 0 0 0 0 0 0 15 0 1 0 916400644 338444288 5729 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 11248047936908525832 0 0 17 6 0 0 0
/proc/17786/statm: 82628 5729 348 364 0 18531 0
[pid=17788] ppid=17786 vsize=2660 CPUtime=2 cores=0-7
/proc/17788/stat : 17788 (march_hi) R 17786 17788 17788 0 -1 4202496 745 0 0 0 197 3 0 0 25 0 1 0 916400646 2723840 479 33554432000 4194304 4955518 140733568069872 18446744073709551615 4263664 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/17788/statm: 665 479 104 186 0 478 0
[pid=17789] ppid=17764 vsize=340768 CPUtime=0 cores=0-7
/proc/17789/stat : 17789 (python) S 17764 17764 17734 0 -1 4202560 785 0 0 0 0 0 0 0 15 0 1 0 916400650 348946432 5749 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 1617403598985986310 0 0 17 6 0 0 0
/proc/17789/statm: 85192 5749 348 364 0 21092 0
[pid=17791] ppid=17789 vsize=2660 CPUtime=1.96 cores=0-7
/proc/17791/stat : 17791 (march_hi) R 17789 17791 17791 0 -1 4202496 725 0 0 0 196 0 0 0 25 0 1 0 916400650 2723840 461 33554432000 4194304 4955518 140735714258544 18446744073709551615 4263701 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/17791/statm: 665 461 104 186 0 478 0
[pid=17793] ppid=17764 vsize=351024 CPUtime=0 cores=0-7
/proc/17793/stat : 17793 (python) S 17764 17764 17734 0 -1 4202560 821 0 0 0 0 0 0 0 15 0 1 0 916400654 359448576 5759 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 15555441913325650180 0 0 17 7 0 0 0
/proc/17793/statm: 87756 5759 354 364 0 23653 0
[pid=17795] ppid=17793 vsize=2660 CPUtime=1.93 cores=0-7
/proc/17795/stat : 17795 (march_hi) R 17793 17795 17795 0 -1 4202496 750 0 0 0 193 0 0 0 25 0 1 0 916400654 2723840 481 33554432000 4194304 4955518 140736064733936 18446744073709551615 4238749 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/17795/statm: 665 481 104 186 0 478 0
Current children cumulated CPU time (s) 13.51
Current children cumulated vsize (KiB) 2247420

[startup+12.7008 s]
/proc/loadavg: 1.23 1.64 2.51 7/178 17795
/proc/meminfo: memFree=28203952/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=310048 CPUtime=1.14 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 15952 3978 1 0 70 16 18 10 18 0 1 0 916400218 317489152 7280 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658976575 0 0 16781312 2 18446744071562232103 0 0 17 7 0 0 0
/proc/17764/statm: 77512 7280 1891 364 0 13409 0
[pid=17773] ppid=17764 vsize=289384 CPUtime=0 cores=0-7
/proc/17773/stat : 17773 (python) S 17764 17764 17734 0 -1 4202560 900 0 0 0 0 0 0 0 15 0 1 0 916400625 296329216 5727 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 2358588785316299016 0 0 17 0 0 0 0
/proc/17773/statm: 72346 5727 371 364 0 8261 0
[pid=17775] ppid=17773 vsize=4248 CPUtime=8.6 cores=0-7
/proc/17775/stat : 17775 (march_hi) R 17773 17775 17775 0 -1 4202496 1018 0 0 0 860 0 0 0 25 0 1 0 916400627 4349952 756 33554432000 4194304 4955518 140734052599808 18446744073709551615 4289745 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17775/statm: 1062 756 104 186 0 875 0
[pid=17777] ppid=17764 vsize=299268 CPUtime=0 cores=0-7
/proc/17777/stat : 17777 (python) S 17764 17764 17734 0 -1 4202560 906 0 0 0 0 0 0 0 15 0 1 0 916400630 306450432 5621 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 2358584387269787912 0 0 17 0 0 0 0
/proc/17777/statm: 74817 5621 371 364 0 10729 0
[pid=17779] ppid=17777 vsize=4248 CPUtime=8.56 cores=0-7
/proc/17779/stat : 17779 (march_hi) R 17777 17779 17779 0 -1 4202496 1020 0 0 0 856 0 0 0 25 0 1 0 916400632 4349952 755 33554432000 4194304 4955518 140737003989024 18446744073709551615 4209664 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17779/statm: 1062 755 104 186 0 875 0
[pid=17780] ppid=17764 vsize=310000 CPUtime=0 cores=0-7
/proc/17780/stat : 17780 (python) S 17764 17764 17734 0 -1 4202560 931 0 0 0 0 0 0 0 15 0 1 0 916400635 317440000 5744 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 9618131899893383432 0 0 17 0 0 0 0
/proc/17780/statm: 77500 5744 371 364 0 13409 0
[pid=17782] ppid=17780 vsize=4248 CPUtime=8.52 cores=0-7
/proc/17782/stat : 17782 (march_hi) R 17780 17782 17782 0 -1 4202496 1010 0 0 0 852 0 0 0 25 0 1 0 916400635 4349952 749 33554432000 4194304 4955518 140733772423520 18446744073709551615 4328300 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17782/statm: 1062 749 104 186 0 875 0
[pid=17786] ppid=17764 vsize=330512 CPUtime=0.01 cores=0-7
/proc/17786/stat : 17786 (python) S 17764 17764 17734 0 -1 4202560 950 0 0 0 1 0 0 0 15 0 1 0 916400644 338444288 5752 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 11248047936908525832 0 0 17 6 0 0 0
/proc/17786/statm: 82628 5752 371 364 0 18531 0
[pid=17788] ppid=17786 vsize=4116 CPUtime=8.4 cores=0-7
/proc/17788/stat : 17788 (march_hi) R 17786 17788 17788 0 -1 4202496 990 0 0 0 836 4 0 0 25 0 1 0 916400646 4214784 724 33554432000 4194304 4955518 140733568069872 18446744073709551615 4227902 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/17788/statm: 1029 724 104 186 0 842 0
[pid=17789] ppid=17764 vsize=340768 CPUtime=0.01 cores=0-7
/proc/17789/stat : 17789 (python) S 17764 17764 17734 0 -1 4202560 953 0 0 0 1 0 0 0 15 0 1 0 916400650 348946432 5772 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 1617403598985986310 0 0 17 6 0 0 0
/proc/17789/statm: 85192 5772 371 364 0 21092 0
[pid=17791] ppid=17789 vsize=3604 CPUtime=8.35 cores=0-7
/proc/17791/stat : 17791 (march_hi) R 17789 17791 17791 0 -1 4202496 900 0 0 0 831 4 0 0 25 0 1 0 916400650 3690496 636 33554432000 4194304 4955518 140735714258544 18446744073709551615 4290851 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/17791/statm: 901 636 104 186 0 714 0
[pid=17793] ppid=17764 vsize=351024 CPUtime=0 cores=0-7
/proc/17793/stat : 17793 (python) S 17764 17764 17734 0 -1 4202560 976 0 0 0 0 0 0 0 15 0 1 0 916400654 359448576 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 15555441913325650180 0 0 17 7 0 0 0
/proc/17793/statm: 87756 5782 377 364 0 23653 0
[pid=17795] ppid=17793 vsize=4248 CPUtime=8.33 cores=0-7
/proc/17795/stat : 17795 (march_hi) R 17793 17795 17795 0 -1 4202496 1015 0 0 0 833 0 0 0 25 0 1 0 916400654 4349952 746 33554432000 4194304 4955518 140736064733936 18446744073709551615 4286685 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/17795/statm: 1062 746 104 186 0 875 0
Current children cumulated CPU time (s) 51.92
Current children cumulated vsize (KiB) 2255716

[startup+25.5006 s]
/proc/loadavg: 1.96 1.79 2.55 7/178 17795
/proc/meminfo: memFree=28194276/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=310048 CPUtime=1.14 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 15952 3978 1 0 70 16 18 10 18 0 1 0 916400218 317489152 7280 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658976575 0 0 16781312 2 18446744071562232103 0 0 17 7 0 0 0
/proc/17764/statm: 77512 7280 1891 364 0 13409 0
[pid=17773] ppid=17764 vsize=289384 CPUtime=0.03 cores=0-7
/proc/17773/stat : 17773 (python) S 17764 17764 17734 0 -1 4202560 910 0 0 0 1 2 0 0 15 0 1 0 916400625 296329216 5727 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 2358588785316299016 0 0 17 0 0 0 0
/proc/17773/statm: 72346 5727 371 364 0 8261 0
[pid=17775] ppid=17773 vsize=7168 CPUtime=21.4 cores=0-7
/proc/17775/stat : 17775 (march_hi) R 17773 17775 17775 0 -1 4202496 1576 0 0 0 2140 0 0 0 25 0 1 0 916400627 7340032 1314 33554432000 4194304 4955518 140734052599808 18446744073709551615 4224642 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/17775/statm: 1792 1314 104 186 0 1605 0
[pid=17777] ppid=17764 vsize=299268 CPUtime=0.03 cores=0-7
/proc/17777/stat : 17777 (python) S 17764 17764 17734 0 -1 4202560 916 0 0 0 2 1 0 0 15 0 1 0 916400630 306450432 5621 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 2358584387269787912 0 0 17 0 0 0 0
/proc/17777/statm: 74817 5621 371 364 0 10729 0
[pid=17779] ppid=17777 vsize=7168 CPUtime=21.36 cores=0-7
/proc/17779/stat : 17779 (march_hi) R 17777 17779 17779 0 -1 4202496 1578 0 0 0 2136 0 0 0 25 0 1 0 916400632 7340032 1313 33554432000 4194304 4955518 140737003989024 18446744073709551615 4227747 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/17779/statm: 1792 1313 104 186 0 1605 0
[pid=17780] ppid=17764 vsize=310000 CPUtime=0.02 cores=0-7
/proc/17780/stat : 17780 (python) S 17764 17764 17734 0 -1 4202560 938 0 0 0 1 1 0 0 15 0 1 0 916400635 317440000 5744 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 9618131899893383432 0 0 17 0 0 0 0
/proc/17780/statm: 77500 5744 371 364 0 13409 0
[pid=17782] ppid=17780 vsize=7168 CPUtime=21.31 cores=0-7
/proc/17782/stat : 17782 (march_hi) R 17780 17782 17782 0 -1 4202496 1561 0 0 0 2131 0 0 0 25 0 1 0 916400635 7340032 1300 33554432000 4194304 4955518 140733772423520 18446744073709551615 4306973 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/17782/statm: 1792 1300 104 186 0 1605 0
[pid=17786] ppid=17764 vsize=330512 CPUtime=0.02 cores=0-7
/proc/17786/stat : 17786 (python) S 17764 17764 17734 0 -1 4202560 959 0 0 0 2 0 0 0 15 0 1 0 916400644 338444288 5752 33554432000 4194304 5683644 140733939074032 18446744073709551615 210647166534 0 0 16781312 514 11248047936908525832 0 0 17 6 0 0 0
/proc/17786/statm: 82628 5752 371 364 0 18531 0

################
# More data... #
################

/proc/17782/stat : 17782 (march_hi) T 17780 17782 17782 0 -1 4202496 4375 0 0 0 10027 0 0 0 25 0 1 0 916400635 18804736 4114 33554432000 4194304 4955518 140733772423520 18446744073709551615 4227677 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/17782/statm: 4591 4114 104 186 0 4404 0
[pid=17780/tid=17798] ppid=17764 vsize=310000 CPUtime=0 cores=0-7
/proc/17780/task/17798/stat : 17798 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916410663 317440000 5774 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471139076160 0 0 -1 5 0 0 0
[pid=17786] ppid=17764 vsize=330512 CPUtime=0.84 cores=0-7
/proc/17786/stat : 17786 (python) S 17764 17764 17734 0 -1 4202560 1177 0 0 0 45 39 0 0 16 0 2 0 916400644 338444288 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17786/statm: 82628 5782 400 364 0 18531 0
[pid=17788] ppid=17786 vsize=123192 CPUtime=681.73 cores=0-7
/proc/17788/stat : 17788 (march_hi) T 17786 17788 17788 0 -1 4202496 22057 0 0 0 68161 12 0 0 25 0 1 0 916400646 126148608 21791 33554432000 4194304 4955518 140733568069872 18446744073709551615 4289578 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/17788/statm: 30798 21791 104 186 0 30611 0
[pid=17786/tid=17818] ppid=17764 vsize=330512 CPUtime=0 cores=0-7
/proc/17786/task/17818/stat : 17818 (python) S 17764 17764 17734 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916468843 338444288 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001496640 0 0 -1 6 0 0 0
[pid=17789] ppid=17764 vsize=340768 CPUtime=0.84 cores=0-7
/proc/17789/stat : 17789 (python) S 17764 17764 17734 0 -1 4202560 1197 0 0 0 48 36 0 0 16 0 2 0 916400650 348946432 5802 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17789/statm: 85192 5802 400 364 0 21092 0
[pid=17791] ppid=17789 vsize=123192 CPUtime=680.29 cores=0-7
/proc/17791/stat : 17791 (march_hi) T 17789 17791 17791 0 -1 4202496 22738 0 0 0 68005 24 0 0 25 0 1 0 916400650 126148608 22474 33554432000 4194304 4955518 140735714258544 18446744073709551615 4224512 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17791/statm: 30798 22474 104 186 0 30611 0
[pid=17789/tid=17817] ppid=17764 vsize=340768 CPUtime=0 cores=0-7
/proc/17789/task/17817/stat : 17817 (python) S 17764 17764 17734 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916468700 348946432 5802 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001493952 0 0 -1 3 0 0 0
[pid=17793] ppid=17764 vsize=351024 CPUtime=0.84 cores=0-7
/proc/17793/stat : 17793 (python) S 17764 17764 17734 0 -1 4202560 1216 0 0 0 46 38 0 0 16 0 2 0 916400654 359448576 5812 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/17793/statm: 87756 5812 406 364 0 23653 0
[pid=17795] ppid=17793 vsize=123192 CPUtime=682.2 cores=0-7
/proc/17795/stat : 17795 (march_hi) T 17793 17795 17795 0 -1 4202496 23107 0 0 0 68216 4 0 0 25 0 1 0 916400654 126148608 22838 33554432000 4194304 4955518 140736064733936 18446744073709551615 4227811 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/17795/statm: 30798 22838 104 186 0 30611 0
[pid=17793/tid=17819] ppid=17764 vsize=351024 CPUtime=0 cores=0-7
/proc/17793/task/17819/stat : 17819 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916468874 359448576 5812 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001494848 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2930.64
Current children cumulated vsize (KiB) 2774320
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.01 0.00 0.00 1/184 17965
/proc/meminfo: memFree=27826480/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=310048 CPUtime=1.14 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 15952 3978 1 0 70 16 18 10 18 0 1 0 916400218 317489152 7280 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658976575 0 0 16781312 2 18446744071562232103 0 0 17 7 0 0 0
/proc/17764/statm: 77512 7280 1891 364 0 13409 0
[pid=17773] ppid=17764 vsize=289384 CPUtime=0.5 cores=0-7
/proc/17773/stat : 17773 (python) S 17764 17764 17734 0 -1 4202560 1132 0 0 0 27 23 0 0 16 0 2 0 916400625 296329216 5757 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/17773/statm: 72346 5757 400 364 0 8261 0
[pid=17775] ppid=17773 vsize=77688 CPUtime=390.76 cores=0-7
/proc/17775/stat : 17775 (march_hi) T 17773 17775 17775 0 -1 4202496 14021 0 0 0 39074 2 0 0 25 0 1 0 916400627 79552512 13759 33554432000 4194304 4955518 140734052599808 18446744073709551615 4227559 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/17775/statm: 19422 13759 104 186 0 19235 0
[pid=17773/tid=17808] ppid=17764 vsize=289384 CPUtime=0 cores=0-7
/proc/17773/task/17808/stat : 17808 (python) S 17764 17764 17734 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 916439704 296329216 5757 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604470973492160 0 0 -1 3 0 0 0
[pid=17777] ppid=17764 vsize=299268 CPUtime=0.48 cores=0-7
/proc/17777/stat : 17777 (python) S 17764 17764 17734 0 -1 4202560 1149 0 0 0 25 23 0 0 16 0 2 0 916400630 306450432 5651 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/17777/statm: 74817 5651 400 364 0 10729 0
[pid=17779] ppid=17777 vsize=77688 CPUtime=390.62 cores=0-7
/proc/17779/stat : 17779 (march_hi) T 17777 17779 17779 0 -1 4202496 14023 0 0 0 39060 2 0 0 25 0 1 0 916400632 79552512 13758 33554432000 4194304 4955518 140737003989024 18446744073709551615 4209793 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/17779/statm: 19422 13758 104 186 0 19235 0
[pid=17777/tid=17807] ppid=17764 vsize=299268 CPUtime=0 cores=0-7
/proc/17777/task/17807/stat : 17807 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439695 306450432 5651 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471139077056 0 0 -1 5 0 0 0
[pid=17780] ppid=17764 vsize=310000 CPUtime=0.13 cores=0-7
/proc/17780/stat : 17780 (python) S 17764 17764 17734 0 -1 4202560 1121 0 0 0 8 5 0 0 16 0 2 0 916400635 317440000 5774 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/17780/statm: 77500 5774 400 364 0 13409 0
[pid=17782] ppid=17780 vsize=18364 CPUtime=100.27 cores=0-7
/proc/17782/stat : 17782 (march_hi) T 17780 17782 17782 0 -1 4202496 4375 0 0 0 10027 0 0 0 25 0 1 0 916400635 18804736 4114 33554432000 4194304 4955518 140733772423520 18446744073709551615 4227677 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/17782/statm: 4591 4114 104 186 0 4404 0
[pid=17780/tid=17798] ppid=17764 vsize=310000 CPUtime=0 cores=0-7
/proc/17780/task/17798/stat : 17798 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916410663 317440000 5774 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471139076160 0 0 -1 5 0 0 0
[pid=17786] ppid=17764 vsize=330512 CPUtime=0.84 cores=0-7
/proc/17786/stat : 17786 (python) S 17764 17764 17734 0 -1 4202560 1177 0 0 0 45 39 0 0 16 0 2 0 916400644 338444288 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17786/statm: 82628 5782 400 364 0 18531 0
[pid=17788] ppid=17786 vsize=123192 CPUtime=681.73 cores=0-7
/proc/17788/stat : 17788 (march_hi) T 17786 17788 17788 0 -1 4202496 22057 0 0 0 68161 12 0 0 25 0 1 0 916400646 126148608 21791 33554432000 4194304 4955518 140733568069872 18446744073709551615 4289578 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/17788/statm: 30798 21791 104 186 0 30611 0
[pid=17786/tid=17818] ppid=17764 vsize=330512 CPUtime=0 cores=0-7
/proc/17786/task/17818/stat : 17818 (python) S 17764 17764 17734 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916468843 338444288 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001496640 0 0 -1 6 0 0 0
[pid=17789] ppid=17764 vsize=340768 CPUtime=0.84 cores=0-7
/proc/17789/stat : 17789 (python) S 17764 17764 17734 0 -1 4202560 1197 0 0 0 48 36 0 0 16 0 2 0 916400650 348946432 5802 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17789/statm: 85192 5802 400 364 0 21092 0
[pid=17791] ppid=17789 vsize=123192 CPUtime=680.29 cores=0-7
/proc/17791/stat : 17791 (march_hi) T 17789 17791 17791 0 -1 4202496 22738 0 0 0 68005 24 0 0 25 0 1 0 916400650 126148608 22474 33554432000 4194304 4955518 140735714258544 18446744073709551615 4224512 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17791/statm: 30798 22474 104 186 0 30611 0
[pid=17789/tid=17817] ppid=17764 vsize=340768 CPUtime=0 cores=0-7
/proc/17789/task/17817/stat : 17817 (python) S 17764 17764 17734 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916468700 348946432 5802 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001493952 0 0 -1 3 0 0 0
[pid=17793] ppid=17764 vsize=351024 CPUtime=0.84 cores=0-7
/proc/17793/stat : 17793 (python) S 17764 17764 17734 0 -1 4202560 1216 0 0 0 46 38 0 0 16 0 2 0 916400654 359448576 5812 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/17793/statm: 87756 5812 406 364 0 23653 0
[pid=17795] ppid=17793 vsize=123192 CPUtime=682.2 cores=0-7
/proc/17795/stat : 17795 (march_hi) T 17793 17795 17795 0 -1 4202496 23107 0 0 0 68216 4 0 0 25 0 1 0 916400654 126148608 22838 33554432000 4194304 4955518 140736064733936 18446744073709551615 4227811 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/17795/statm: 30798 22838 104 186 0 30611 0
[pid=17793/tid=17819] ppid=17764 vsize=351024 CPUtime=0 cores=0-7
/proc/17793/task/17819/stat : 17819 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916468874 359448576 5812 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001494848 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2930.64
Current children cumulated vsize (KiB) 2774320
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.01 0.00 0.00 1/184 17965
/proc/meminfo: memFree=27826472/32951132 swapFree=67061148/67111528
[pid=17764] ppid=17762 vsize=310048 CPUtime=1.14 cores=0-7
/proc/17764/stat : 17764 (python) S 17762 17764 17734 0 -1 4202496 15952 3978 1 0 70 16 18 10 18 0 1 0 916400218 317489152 7280 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658976575 0 0 16781312 2 18446744071562232103 0 0 17 7 0 0 0
/proc/17764/statm: 77512 7280 1891 364 0 13409 0
[pid=17773] ppid=17764 vsize=289384 CPUtime=0.5 cores=0-7
/proc/17773/stat : 17773 (python) S 17764 17764 17734 0 -1 4202560 1132 0 0 0 27 23 0 0 16 0 2 0 916400625 296329216 5757 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/17773/statm: 72346 5757 400 364 0 8261 0
[pid=17775] ppid=17773 vsize=77688 CPUtime=390.76 cores=0-7
/proc/17775/stat : 17775 (march_hi) T 17773 17775 17775 0 -1 4202496 14021 0 0 0 39074 2 0 0 25 0 1 0 916400627 79552512 13759 33554432000 4194304 4955518 140734052599808 18446744073709551615 4227559 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/17775/statm: 19422 13759 104 186 0 19235 0
[pid=17773/tid=17808] ppid=17764 vsize=289384 CPUtime=0 cores=0-7
/proc/17773/task/17808/stat : 17808 (python) S 17764 17764 17734 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 916439704 296329216 5757 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604470973492160 0 0 -1 3 0 0 0
[pid=17777] ppid=17764 vsize=299268 CPUtime=0.48 cores=0-7
/proc/17777/stat : 17777 (python) S 17764 17764 17734 0 -1 4202560 1149 0 0 0 25 23 0 0 16 0 2 0 916400630 306450432 5651 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/17777/statm: 74817 5651 400 364 0 10729 0
[pid=17779] ppid=17777 vsize=77688 CPUtime=390.62 cores=0-7
/proc/17779/stat : 17779 (march_hi) T 17777 17779 17779 0 -1 4202496 14023 0 0 0 39060 2 0 0 25 0 1 0 916400632 79552512 13758 33554432000 4194304 4955518 140737003989024 18446744073709551615 4209793 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/17779/statm: 19422 13758 104 186 0 19235 0
[pid=17777/tid=17807] ppid=17764 vsize=299268 CPUtime=0 cores=0-7
/proc/17777/task/17807/stat : 17807 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439695 306450432 5651 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471139077056 0 0 -1 5 0 0 0
[pid=17780] ppid=17764 vsize=310000 CPUtime=0.13 cores=0-7
/proc/17780/stat : 17780 (python) S 17764 17764 17734 0 -1 4202560 1121 0 0 0 8 5 0 0 16 0 2 0 916400635 317440000 5774 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/17780/statm: 77500 5774 400 364 0 13409 0
[pid=17782] ppid=17780 vsize=18364 CPUtime=100.27 cores=0-7
/proc/17782/stat : 17782 (march_hi) T 17780 17782 17782 0 -1 4202496 4375 0 0 0 10027 0 0 0 25 0 1 0 916400635 18804736 4114 33554432000 4194304 4955518 140733772423520 18446744073709551615 4227677 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/17782/statm: 4591 4114 104 186 0 4404 0
[pid=17780/tid=17798] ppid=17764 vsize=310000 CPUtime=0 cores=0-7
/proc/17780/task/17798/stat : 17798 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916410663 317440000 5774 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471139076160 0 0 -1 5 0 0 0
[pid=17786] ppid=17764 vsize=330512 CPUtime=0.84 cores=0-7
/proc/17786/stat : 17786 (python) S 17764 17764 17734 0 -1 4202560 1177 0 0 0 45 39 0 0 16 0 2 0 916400644 338444288 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17786/statm: 82628 5782 400 364 0 18531 0
[pid=17788] ppid=17786 vsize=123192 CPUtime=681.73 cores=0-7
/proc/17788/stat : 17788 (march_hi) T 17786 17788 17788 0 -1 4202496 22057 0 0 0 68161 12 0 0 25 0 1 0 916400646 126148608 21791 33554432000 4194304 4955518 140733568069872 18446744073709551615 4289578 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/17788/statm: 30798 21791 104 186 0 30611 0
[pid=17786/tid=17818] ppid=17764 vsize=330512 CPUtime=0 cores=0-7
/proc/17786/task/17818/stat : 17818 (python) S 17764 17764 17734 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 916468843 338444288 5782 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001496640 0 0 -1 6 0 0 0
[pid=17789] ppid=17764 vsize=340768 CPUtime=0.84 cores=0-7
/proc/17789/stat : 17789 (python) S 17764 17764 17734 0 -1 4202560 1197 0 0 0 48 36 0 0 16 0 2 0 916400650 348946432 5802 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/17789/statm: 85192 5802 400 364 0 21092 0
[pid=17791] ppid=17789 vsize=123192 CPUtime=680.29 cores=0-7
/proc/17791/stat : 17791 (march_hi) T 17789 17791 17791 0 -1 4202496 22738 0 0 0 68005 24 0 0 25 0 1 0 916400650 126148608 22474 33554432000 4194304 4955518 140735714258544 18446744073709551615 4224512 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/17791/statm: 30798 22474 104 186 0 30611 0
[pid=17789/tid=17817] ppid=17764 vsize=340768 CPUtime=0 cores=0-7
/proc/17789/task/17817/stat : 17817 (python) S 17764 17764 17734 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916468700 348946432 5802 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001493952 0 0 -1 3 0 0 0
[pid=17793] ppid=17764 vsize=351024 CPUtime=0.84 cores=0-7
/proc/17793/stat : 17793 (python) S 17764 17764 17734 0 -1 4202560 1216 0 0 0 46 38 0 0 16 0 2 0 916400654 359448576 5812 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/17793/statm: 87756 5812 406 364 0 23653 0
[pid=17795] ppid=17793 vsize=123192 CPUtime=682.2 cores=0-7
/proc/17795/stat : 17795 (march_hi) T 17793 17795 17795 0 -1 4202496 23107 0 0 0 68216 4 0 0 25 0 1 0 916400654 126148608 22838 33554432000 4194304 4955518 140736064733936 18446744073709551615 4227811 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/17795/statm: 30798 22838 104 186 0 30611 0
[pid=17793/tid=17819] ppid=17764 vsize=351024 CPUtime=0 cores=0-7
/proc/17793/task/17819/stat : 17819 (python) S 17764 17764 17734 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916468874 359448576 5812 33554432000 4194304 5683644 140733939074032 18446744073709551615 210658970033 0 0 16781312 514 18446604471001494848 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2930.64
Current children cumulated vsize (KiB) 2774320
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 17764 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=919556
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=937705
# CPU time returned by wait4() is 4.85726
# while last known CPU time is 2930.64
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.04
CPU time (s): 2930.64
CPU user time (s): 2928.3
CPU system time (s): 2.34
CPU usage (%): 57.463
Max. virtual memory (cumulated for all children) (KiB): 2915612

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.91956
system time used= 1.93771
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28858
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7338
involuntary context switches= 46

runsolver used 6.63799 second user time and 16.7285 second system time

The end

Launcher Data 

Begin job on node129 at 2011-06-14 12:40:02
IDJOB=3322142
IDBENCH=82920
IDSOLVER=1712
FILE ID=node129/3322142-1308048002
RUNJOBID= node129-1308048001-17748
PBS_JOBID= 13566203
Free space on /tmp= 74028 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v550-c2343-S1114532273-035.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322142-1308048002/watcher-3322142-1308048002 -o /tmp/evaluation-result-3322142-1308048002/solver-3322142-1308048002 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322142-1308048002.cnf -seed 1384931477 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ff7e731e0657a2728f5ff73a4fd8bb2e
RANDOM SEED=1384931477

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      28270208 kB
Buffers:        465172 kB
Cached:        3907172 kB
SwapCached:      13132 kB
Active:        1327824 kB
Inactive:      3061956 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      28270208 kB
SwapTotal:    67111528 kB
SwapFree:     67061148 kB
Dirty:           31448 kB
Writeback:           0 kB
AnonPages:       14448 kB
Mapped:           8576 kB
Slab:           229296 kB
PageTables:       3776 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   154224 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73996 MiB
End job on node129 at 2011-06-14 14:05:08