Trace number 3322160

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 1572.05 5100.02

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v600-c2556-S727005963-093.SATISFIABLE.cnf
MD5SUM91e4bda9499071fffccca1352d82691f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.028995
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses2556
Sum of the clauses size7668
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32556
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.38/3.38	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.38/3.38	c INFO: solving HOME/instance-3322160-1308048003.cnf
0.38/3.38	c INFO: computing task features
0.38/3.39	c INFO: computed variable-clause graph statistics
0.38/3.39	c INFO: computed clause balance statistics
0.44/3.44	c INFO: computed variable balance statistics
0.44/3.47	c INFO: computed small-clause counts
0.44/3.48	c INFO: computed Horn-clause counts
0.44/3.48	c INFO: computed variable graph statistics
0.44/3.48	c INFO: collected features for HOME/instance-3322160-1308048003.cnf in 0.09 s
0.54/3.52	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.58)
0.54/3.55	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.06)
0.54/3.59	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.43)
0.69/3.63	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.39)
0.69/3.67	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.36)
0.87/3.71	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.25)
0.87/3.75	c INFO: running kcnfs-2006@0 for 582 with 39999 remaining (b = 0.27)
0.87/3.79	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.22)
0.87/3.79	Traceback (most recent call last):
0.87/3.79	  File "HOME/solve", line 33, in <module>
0.87/3.79	    plac.call(main)
0.87/3.79	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
0.87/3.79	    cmd, result = parser_from(obj).consume(arglist)
0.87/3.79	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
0.87/3.79	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
0.87/3.79	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
0.87/3.79	    (_, answer, _) = solver(input_path, budget, cores)
0.87/3.79	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
0.87/3.79	    process.die()
0.87/3.79	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
0.87/3.79	    os.kill(self._process.pid, signal.SIGUSR1)
0.87/3.79	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322160-1308048003/watcher-3322160-1308048003 -o /tmp/evaluation-result-3322160-1308048003/solver-3322160-1308048003 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322160-1308048003.cnf -seed 14014297 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.58 2.66 2.87 1/170 11061
/proc/meminfo: memFree=31286032/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=11804 CPUtime=0 cores=0-7
/proc/11061/stat : 11061 (runsolver) S 11059 11061 11031 0 -1 4202560 42 0 0 0 0 0 0 0 22 0 1 0 916400440 12087296 51 33554432000 4194304 5436572 140736312963008 18446744073709551615 4883175 0 2147483391 4096 24578 18446744071699960631 0 0 17 3 0 0 0
/proc/11061/statm: 2951 51 31 304 0 2643 0

[startup+0.0693491 s]
/proc/loadavg: 0.58 2.66 2.87 1/170 11061
/proc/meminfo: memFree=31286032/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=15184 CPUtime=0 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 713 0 0 0 0 0 0 0 23 0 1 0 916400440 15548416 603 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/11061/statm: 3796 603 319 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.100343 s]
/proc/loadavg: 0.58 2.66 2.87 1/170 11061
/proc/meminfo: memFree=31286032/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=15184 CPUtime=0 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 725 0 0 0 0 0 0 0 23 0 1 0 916400440 15548416 614 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/11061/statm: 3796 614 321 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.300308 s]
/proc/loadavg: 0.58 2.66 2.87 1/170 11061
/proc/meminfo: memFree=31286032/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=15776 CPUtime=0 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 904 0 0 0 0 0 0 0 21 0 1 0 916400440 16154624 783 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/11061/statm: 3944 783 346 364 0 431 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15776

[startup+0.700237 s]
/proc/loadavg: 0.58 2.66 2.87 1/170 11061
/proc/meminfo: memFree=31286032/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=25288 CPUtime=0.02 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 1345 0 0 0 2 0 0 0 17 0 1 0 916400440 25894912 1190 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/11061/statm: 6322 1190 438 364 0 723 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 25288

[startup+1.50017 s]
/proc/loadavg: 0.54 2.62 2.85 1/171 11062
/proc/meminfo: memFree=31281224/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=93964 CPUtime=0.06 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 2695 0 0 0 5 1 0 0 15 0 1 0 916400440 96219136 2407 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/11061/statm: 23491 2407 764 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 93964

[startup+3.10082 s]
/proc/loadavg: 0.54 2.62 2.85 1/171 11062
/proc/meminfo: memFree=31277008/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=151216 CPUtime=0.25 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 6313 935 1 0 17 8 0 0 15 0 1 0 916400440 154845184 4441 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/11061/statm: 37804 4441 1220 364 0 3215 0
Current children cumulated CPU time (s) 0.25
Current children cumulated vsize (KiB) 151216
heavy processes:

[startup+6.30125 s]
/proc/loadavg: 0.54 2.62 2.85 5/179 11092
/proc/meminfo: memFree=31239004/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 25 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=299268 CPUtime=0 cores=0-7
/proc/11074/stat : 11074 (python) S 11061 11061 11031 0 -1 4202560 755 0 0 0 0 0 0 0 15 0 1 0 916400800 306450432 5613 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 4124188955245510916 0 0 17 2 0 0 0
/proc/11074/statm: 74817 5613 348 364 0 10729 0
[pid=11076] ppid=11074 vsize=2916 CPUtime=2.69 cores=0-7
/proc/11076/stat : 11076 (march_hi) R 11074 11076 11076 0 -1 4202496 802 0 0 0 269 0 0 0 25 0 1 0 916400800 2985984 537 33554432000 4194304 4955518 140735326042784 18446744073709551615 4227811 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11076/statm: 729 537 104 186 0 542 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 782 0 0 0 0 0 0 0 15 0 1 0 916400812 338509824 5744 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 3838742542535328004 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5744 348 364 0 18547 0
[pid=11086] ppid=11084 vsize=2916 CPUtime=2.58 cores=0-7
/proc/11086/stat : 11086 (march_hi) R 11084 11086 11086 0 -1 4202496 797 0 0 0 258 0 0 0 25 0 1 0 916400812 2985984 531 33554432000 4194304 4955518 140735922591360 18446744073709551615 4261360 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11086/statm: 729 531 104 186 0 542 0
[pid=11087] ppid=11061 vsize=340832 CPUtime=0 cores=0-7
/proc/11087/stat : 11087 (python) S 11061 11061 11031 0 -1 4202560 771 0 0 0 0 0 0 0 18 0 1 0 916400816 349011968 5748 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 4038927875825434881 0 0 17 0 0 0 0
/proc/11087/statm: 85208 5748 348 364 0 21108 0
[pid=11089] ppid=11087 vsize=182580 CPUtime=2.54 cores=0-7
/proc/11089/stat : 11089 (kcnfs-2006) R 11087 11089 11089 0 -1 4202496 628 0 0 0 254 0 0 0 25 0 1 0 916400816 186961920 365 33554432000 134512640 135163436 4293538336 18446744073709551615 134559348 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/11089/statm: 45645 365 77 159 0 45485 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 806 0 0 0 0 0 0 0 15 0 1 0 916400820 359514112 5758 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 15835681038966685954 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5758 354 364 0 23669 0
[pid=11092] ppid=11090 vsize=2916 CPUtime=2.49 cores=0-7
/proc/11092/stat : 11092 (march_hi) R 11090 11092 11092 0 -1 4202496 796 0 0 0 249 0 0 0 25 0 1 0 916400820 2985984 524 33554432000 4194304 4955518 140734149606144 18446744073709551615 4227522 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/11092/statm: 729 524 104 186 0 542 0
Current children cumulated CPU time (s) 11.56
Current children cumulated vsize (KiB) 1823204

[startup+12.7011 s]
/proc/loadavg: 1.07 2.67 2.86 5/179 11092
/proc/meminfo: memFree=31236284/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 25 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=299268 CPUtime=0 cores=0-7
/proc/11074/stat : 11074 (python) S 11061 11061 11031 0 -1 4202560 896 0 0 0 0 0 0 0 15 0 1 0 916400800 306450432 5636 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 4124188955245510916 0 0 17 2 0 0 0
/proc/11074/statm: 74817 5636 371 364 0 10729 0
[pid=11076] ppid=11074 vsize=4400 CPUtime=9.09 cores=0-7
/proc/11076/stat : 11076 (march_hi) R 11074 11076 11076 0 -1 4202496 1069 0 0 0 909 0 0 0 25 0 1 0 916400800 4505600 804 33554432000 4194304 4955518 140735326042784 18446744073709551615 4261466 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11076/statm: 1100 804 104 186 0 913 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 928 0 0 0 0 0 0 0 15 0 1 0 916400812 338509824 5767 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 3838742542535328004 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5767 371 364 0 18547 0
[pid=11086] ppid=11084 vsize=4400 CPUtime=8.98 cores=0-7
/proc/11086/stat : 11086 (march_hi) R 11084 11086 11086 0 -1 4202496 1066 0 0 0 898 0 0 0 25 0 1 0 916400812 4505600 800 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209643 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11086/statm: 1100 800 104 186 0 913 0
[pid=11087] ppid=11061 vsize=340832 CPUtime=0 cores=0-7
/proc/11087/stat : 11087 (python) S 11061 11061 11031 0 -1 4202560 930 0 0 0 0 0 0 0 18 0 1 0 916400816 349011968 5771 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 4038927875825434881 0 0 17 0 0 0 0
/proc/11087/statm: 85208 5771 371 364 0 21108 0
[pid=11089] ppid=11087 vsize=182580 CPUtime=8.94 cores=0-7
/proc/11089/stat : 11089 (kcnfs-2006) R 11087 11089 11089 0 -1 4202496 628 0 0 0 894 0 0 0 25 0 1 0 916400816 186961920 365 33554432000 134512640 135163436 4293538336 18446744073709551615 134590446 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/11089/statm: 45645 365 77 159 0 45485 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 965 0 0 0 0 0 0 0 15 0 1 0 916400820 359514112 5781 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 15835681038966685954 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5781 377 364 0 23669 0
[pid=11092] ppid=11090 vsize=4400 CPUtime=8.89 cores=0-7
/proc/11092/stat : 11092 (march_hi) R 11090 11092 11092 0 -1 4202496 1058 0 0 0 889 0 0 0 25 0 1 0 916400820 4505600 786 33554432000 4194304 4955518 140734149606144 18446744073709551615 4314216 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/11092/statm: 1100 786 104 186 0 913 0
Current children cumulated CPU time (s) 37.16
Current children cumulated vsize (KiB) 1827656

[startup+25.5009 s]
/proc/loadavg: 1.52 2.71 2.88 5/179 11093
/proc/meminfo: memFree=31232660/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 25 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=299268 CPUtime=0.03 cores=0-7
/proc/11074/stat : 11074 (python) S 11061 11061 11031 0 -1 4202560 906 0 0 0 2 1 0 0 15 0 1 0 916400800 306450432 5636 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 4124188955245510916 0 0 17 2 0 0 0
/proc/11074/statm: 74817 5636 371 364 0 10729 0
[pid=11076] ppid=11074 vsize=7128 CPUtime=21.89 cores=0-7
/proc/11076/stat : 11076 (march_hi) R 11074 11076 11076 0 -1 4202496 1572 0 0 0 2189 0 0 0 25 0 1 0 916400800 7299072 1307 33554432000 4194304 4955518 140735326042784 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11076/statm: 1782 1307 104 186 0 1595 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0.03 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 937 0 0 0 2 1 0 0 15 0 1 0 916400812 338509824 5767 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 3838742542535328004 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5767 371 364 0 18547 0
[pid=11086] ppid=11084 vsize=7128 CPUtime=21.78 cores=0-7
/proc/11086/stat : 11086 (march_hi) R 11084 11086 11086 0 -1 4202496 1570 0 0 0 2178 0 0 0 25 0 1 0 916400812 7299072 1304 33554432000 4194304 4955518 140735922591360 18446744073709551615 4238345 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/11086/statm: 1782 1304 104 186 0 1595 0
[pid=11087] ppid=11061 vsize=340832 CPUtime=0.01 cores=0-7
/proc/11087/stat : 11087 (python) S 11061 11061 11031 0 -1 4202560 936 0 0 0 0 1 0 0 18 0 1 0 916400816 349011968 5771 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 17678000727940628737 0 0 17 0 0 0 0
/proc/11087/statm: 85208 5771 371 364 0 21108 0
[pid=11089] ppid=11087 vsize=182580 CPUtime=21.74 cores=0-7
/proc/11089/stat : 11089 (kcnfs-2006) R 11087 11089 11089 0 -1 4202496 628 0 0 0 2174 0 0 0 25 0 1 0 916400816 186961920 365 33554432000 134512640 135163436 4293538336 18446744073709551615 134585921 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/11089/statm: 45645 365 77 159 0 45485 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0.03 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 972 0 0 0 2 1 0 0 15 0 1 0 916400820 359514112 5781 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 15835681038966685954 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5781 377 364 0 23669 0
[pid=11092] ppid=11090 vsize=7128 CPUtime=21.68 cores=0-7
/proc/11092/stat : 11092 (march_hi) R 11090 11092 11092 0 -1 4202496 1556 0 0 0 2168 0 0 0 25 0 1 0 916400820 7299072 1284 33554432000 4194304 4955518 140734149606144 18446744073709551615 4244416 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/11092/statm: 1782 1284 104 186 0 1595 0
Current children cumulated CPU time (s) 88.45
Current children cumulated vsize (KiB) 1835840

[startup+51.1003 s]
/proc/loadavg: 2.36 2.81 2.91 5/179 11093
/proc/meminfo: memFree=31222252/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 25 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 2 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=299268 CPUtime=0.06 cores=0-7
/proc/11074/stat : 11074 (python) S 11061 11061 11031 0 -1 4202560 915 0 0 0 4 2 0 0 15 0 1 0 916400800 306450432 5636 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 17678000727940628740 0 0 17 2 0 0 0
/proc/11074/statm: 74817 5636 371 364 0 10729 0
[pid=11076] ppid=11074 vsize=12520 CPUtime=47.49 cores=0-7
/proc/11076/stat : 11076 (march_hi) R 11074 11076 11076 0 -1 4202496 2523 0 0 0 4749 0 0 0 25 0 1 0 916400800 12820480 2258 33554432000 4194304 4955518 140735326042784 18446744073709551615 4227667 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/11076/statm: 3130 2258 104 186 0 2943 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0.06 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 945 0 0 0 4 2 0 0 15 0 1 0 916400812 338509824 5767 33554432000 4194304 5683644 140736073751264 18446744073709551615 248089718342 0 0 16781312 514 3838742542535328004 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5767 371 364 0 18547 0
[pid=11086] ppid=11084 vsize=12520 CPUtime=47.37 cores=0-7
/proc/11086/stat : 11086 (march_hi) R 11084 11086 11086 0 -1 4202496 2522 0 0 0 4737 0 0 0 25 0 1 0 916400812 12820480 2256 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209776 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/11086/statm: 3130 2256 104 186 0 2943 0

################
# More data... #
################

[pid=11086] ppid=11084 vsize=79120 CPUtime=390.88 cores=0-7
/proc/11086/stat : 11086 (march_hi) T 11084 11086 11086 0 -1 4202496 14087 0 0 0 39086 2 0 0 25 0 1 0 916400812 81018880 13821 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209711 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/11086/statm: 19780 13821 104 186 0 19593 0
[pid=11084/tid=11104] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/task/11104/stat : 11104 (python) S 11061 11061 11031 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439901 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453840265344 0 0 -1 0 0 0 0
[pid=11087] ppid=11061 vsize=0 CPUtime=103.57 cores=0-7
/proc/11087/stat : 11087 (python) Z 11061 11061 11031 0 -1 4202588 1261 640 0 0 6 5 10346 0 15 0 1 0 916400816 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/11087/statm: 0 0 0 0 0 0 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0.59 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 1197 0 0 0 35 24 0 0 16 0 2 0 916400820 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5811 406 364 0 23669 0
[pid=11092] ppid=11090 vsize=79120 CPUtime=485.87 cores=0-7
/proc/11092/stat : 11092 (march_hi) T 11090 11092 11092 0 -1 4202496 17123 0 0 0 48583 4 0 0 25 0 1 0 916400820 81018880 16851 33554432000 4194304 4955518 140734149606144 18446744073709551615 4227807 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/11092/statm: 19780 16851 104 186 0 19593 0
[pid=11090/tid=11108] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/task/11108/stat : 11108 (python) S 11061 11061 11031 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916449409 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453846935360 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1572.05
Current children cumulated vsize (KiB) 1150016
heavy processes:

[startup+4962.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 11256
/proc/meminfo: memFree=31129340/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 18 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=0 CPUtime=589.39 cores=0-7
/proc/11074/stat : 11074 (python) Z 11061 11061 11031 0 -1 4202588 1206 20455 0 0 40 32 58862 5 15 0 1 0 916400800 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/11074/statm: 0 0 0 0 0 0 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0.49 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 1163 0 0 0 30 19 0 0 16 0 2 0 916400812 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5797 400 364 0 18547 0
[pid=11086] ppid=11084 vsize=79120 CPUtime=390.88 cores=0-7
/proc/11086/stat : 11086 (march_hi) T 11084 11086 11086 0 -1 4202496 14087 0 0 0 39086 2 0 0 25 0 1 0 916400812 81018880 13821 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209711 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/11086/statm: 19780 13821 104 186 0 19593 0
[pid=11084/tid=11104] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/task/11104/stat : 11104 (python) S 11061 11061 11031 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439901 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453840265344 0 0 -1 0 0 0 0
[pid=11087] ppid=11061 vsize=0 CPUtime=103.57 cores=0-7
/proc/11087/stat : 11087 (python) Z 11061 11061 11031 0 -1 4202588 1261 640 0 0 6 5 10346 0 15 0 1 0 916400816 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/11087/statm: 0 0 0 0 0 0 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0.59 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 1197 0 0 0 35 24 0 0 16 0 2 0 916400820 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5811 406 364 0 23669 0
[pid=11092] ppid=11090 vsize=79120 CPUtime=485.87 cores=0-7
/proc/11092/stat : 11092 (march_hi) T 11090 11092 11092 0 -1 4202496 17123 0 0 0 48583 4 0 0 25 0 1 0 916400820 81018880 16851 33554432000 4194304 4955518 140734149606144 18446744073709551615 4227807 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/11092/statm: 19780 16851 104 186 0 19593 0
[pid=11090/tid=11108] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/task/11108/stat : 11108 (python) S 11061 11061 11031 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916449409 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453846935360 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1572.05
Current children cumulated vsize (KiB) 1150016
heavy processes:

[startup+5022.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 11258
/proc/meminfo: memFree=31129288/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 18 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=0 CPUtime=589.39 cores=0-7
/proc/11074/stat : 11074 (python) Z 11061 11061 11031 0 -1 4202588 1206 20455 0 0 40 32 58862 5 15 0 1 0 916400800 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/11074/statm: 0 0 0 0 0 0 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0.49 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 1163 0 0 0 30 19 0 0 16 0 2 0 916400812 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5797 400 364 0 18547 0
[pid=11086] ppid=11084 vsize=79120 CPUtime=390.88 cores=0-7
/proc/11086/stat : 11086 (march_hi) T 11084 11086 11086 0 -1 4202496 14087 0 0 0 39086 2 0 0 25 0 1 0 916400812 81018880 13821 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209711 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/11086/statm: 19780 13821 104 186 0 19593 0
[pid=11084/tid=11104] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/task/11104/stat : 11104 (python) S 11061 11061 11031 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439901 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453840265344 0 0 -1 0 0 0 0
[pid=11087] ppid=11061 vsize=0 CPUtime=103.57 cores=0-7
/proc/11087/stat : 11087 (python) Z 11061 11061 11031 0 -1 4202588 1261 640 0 0 6 5 10346 0 15 0 1 0 916400816 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/11087/statm: 0 0 0 0 0 0 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0.59 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 1197 0 0 0 35 24 0 0 16 0 2 0 916400820 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5811 406 364 0 23669 0
[pid=11092] ppid=11090 vsize=79120 CPUtime=485.87 cores=0-7
/proc/11092/stat : 11092 (march_hi) T 11090 11092 11092 0 -1 4202496 17123 0 0 0 48583 4 0 0 25 0 1 0 916400820 81018880 16851 33554432000 4194304 4955518 140734149606144 18446744073709551615 4227807 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/11092/statm: 19780 16851 104 186 0 19593 0
[pid=11090/tid=11108] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/task/11108/stat : 11108 (python) S 11061 11061 11031 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916449409 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453846935360 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1572.05
Current children cumulated vsize (KiB) 1150016
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 11262
/proc/meminfo: memFree=31129344/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 18 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=0 CPUtime=589.39 cores=0-7
/proc/11074/stat : 11074 (python) Z 11061 11061 11031 0 -1 4202588 1206 20455 0 0 40 32 58862 5 15 0 1 0 916400800 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/11074/statm: 0 0 0 0 0 0 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0.49 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 1163 0 0 0 30 19 0 0 16 0 2 0 916400812 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5797 400 364 0 18547 0
[pid=11086] ppid=11084 vsize=79120 CPUtime=390.88 cores=0-7
/proc/11086/stat : 11086 (march_hi) T 11084 11086 11086 0 -1 4202496 14087 0 0 0 39086 2 0 0 25 0 1 0 916400812 81018880 13821 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209711 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/11086/statm: 19780 13821 104 186 0 19593 0
[pid=11084/tid=11104] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/task/11104/stat : 11104 (python) S 11061 11061 11031 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439901 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453840265344 0 0 -1 0 0 0 0
[pid=11087] ppid=11061 vsize=0 CPUtime=103.57 cores=0-7
/proc/11087/stat : 11087 (python) Z 11061 11061 11031 0 -1 4202588 1261 640 0 0 6 5 10346 0 15 0 1 0 916400816 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/11087/statm: 0 0 0 0 0 0 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0.59 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 1197 0 0 0 35 24 0 0 16 0 2 0 916400820 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5811 406 364 0 23669 0
[pid=11092] ppid=11090 vsize=79120 CPUtime=485.87 cores=0-7
/proc/11092/stat : 11092 (march_hi) T 11090 11092 11092 0 -1 4202496 17123 0 0 0 48583 4 0 0 25 0 1 0 916400820 81018880 16851 33554432000 4194304 4955518 140734149606144 18446744073709551615 4227807 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/11092/statm: 19780 16851 104 186 0 19593 0
[pid=11090/tid=11108] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/task/11108/stat : 11108 (python) S 11061 11061 11031 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916449409 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453846935360 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1572.05
Current children cumulated vsize (KiB) 1150016
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.00 0.00 0.00 1/179 11263
/proc/meminfo: memFree=31129336/32950928 swapFree=67055160/67111528
[pid=11061] ppid=11059 vsize=310112 CPUtime=1.26 cores=0-7
/proc/11061/stat : 11061 (python) S 11059 11061 11031 0 -1 4202496 15902 6893 1 0 69 14 41 2 18 0 1 0 916400440 317554688 7279 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101528383 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/11061/statm: 77528 7279 1891 364 0 13425 0
[pid=11074] ppid=11061 vsize=0 CPUtime=589.39 cores=0-7
/proc/11074/stat : 11074 (python) Z 11061 11061 11031 0 -1 4202588 1206 20455 0 0 40 32 58862 5 15 0 1 0 916400800 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/11074/statm: 0 0 0 0 0 0 0
[pid=11084] ppid=11061 vsize=330576 CPUtime=0.49 cores=0-7
/proc/11084/stat : 11084 (python) S 11061 11061 11031 0 -1 4202560 1163 0 0 0 30 19 0 0 16 0 2 0 916400812 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/11084/statm: 82644 5797 400 364 0 18547 0
[pid=11086] ppid=11084 vsize=79120 CPUtime=390.88 cores=0-7
/proc/11086/stat : 11086 (march_hi) T 11084 11086 11086 0 -1 4202496 14087 0 0 0 39086 2 0 0 25 0 1 0 916400812 81018880 13821 33554432000 4194304 4955518 140735922591360 18446744073709551615 4209711 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/11086/statm: 19780 13821 104 186 0 19593 0
[pid=11084/tid=11104] ppid=11061 vsize=330576 CPUtime=0 cores=0-7
/proc/11084/task/11104/stat : 11104 (python) S 11061 11061 11031 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 916439901 338509824 5797 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453840265344 0 0 -1 0 0 0 0
[pid=11087] ppid=11061 vsize=0 CPUtime=103.57 cores=0-7
/proc/11087/stat : 11087 (python) Z 11061 11061 11031 0 -1 4202588 1261 640 0 0 6 5 10346 0 15 0 1 0 916400816 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 0 0 0 0
/proc/11087/statm: 0 0 0 0 0 0 0
[pid=11090] ppid=11061 vsize=351088 CPUtime=0.59 cores=0-7
/proc/11090/stat : 11090 (python) S 11061 11061 11031 0 -1 4202560 1197 0 0 0 35 24 0 0 16 0 2 0 916400820 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/11090/statm: 87772 5811 406 364 0 23669 0
[pid=11092] ppid=11090 vsize=79120 CPUtime=485.87 cores=0-7
/proc/11092/stat : 11092 (march_hi) T 11090 11092 11092 0 -1 4202496 17123 0 0 0 48583 4 0 0 25 0 1 0 916400820 81018880 16851 33554432000 4194304 4955518 140734149606144 18446744073709551615 4227807 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/11092/statm: 19780 16851 104 186 0 19593 0
[pid=11090/tid=11108] ppid=11061 vsize=351088 CPUtime=0 cores=0-7
/proc/11090/task/11108/stat : 11108 (python) S 11061 11061 11031 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 916449409 359514112 5811 33554432000 4194304 5683644 140736073751264 18446744073709551615 248101521841 0 0 16781312 514 18446604453846935360 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1572.05
Current children cumulated vsize (KiB) 1150016
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 11061 and gives
#  childrusage.ru_utime.tv_sec=694
#  childrusage.ru_utime.tv_usec=346443
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=44841
# CPU time returned by wait4() is 695.391
# while last known CPU time is 1572.05
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.02
CPU time (s): 1572.05
CPU user time (s): 1570.98
CPU system time (s): 1.07
CPU usage (%): 30.8244
Max. virtual memory (cumulated for all children) (KiB): 1888020

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 694.346
system time used= 1.04484
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 50649
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5323
involuntary context switches= 1312

runsolver used 6.60499 second user time and 16.8354 second system time

The end

Launcher Data 

Begin job on node123 at 2011-06-14 12:40:03
IDJOB=3322160
IDBENCH=82938
IDSOLVER=1712
FILE ID=node123/3322160-1308048003
RUNJOBID= node123-1308048003-11045
PBS_JOBID= 13566206
Free space on /tmp= 74116 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v600-c2556-S727005963-093.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322160-1308048003/watcher-3322160-1308048003 -o /tmp/evaluation-result-3322160-1308048003/solver-3322160-1308048003 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322160-1308048003.cnf -seed 14014297 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 91e4bda9499071fffccca1352d82691f
RANDOM SEED=14014297

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31286436 kB
Buffers:        204696 kB
Cached:        1215116 kB
SwapCached:      10116 kB
Active:         359376 kB
Inactive:      1078000 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31286436 kB
SwapTotal:    67111528 kB
SwapFree:     67055160 kB
Dirty:           31960 kB
Writeback:           0 kB
AnonPages:       15416 kB
Mapped:          12304 kB
Slab:           162080 kB
PageTables:       4236 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   187456 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74088 MiB
End job on node123 at 2011-06-14 14:05:08