Trace number 3322031

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 3239.81 5100.02

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v650-c2769-S1894319588-015.SATISFIABLE.cnf
MD5SUMc06f05996bba25e7e26c6579a49532fc
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.011998
Satisfiable
(Un)Satisfiability was proved
Number of variables650
Number of clauses2769
Sum of the clauses size8307
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32769
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.46	c INFO: solving HOME/instance-3322031-1308045988.cnf
0.31/0.46	c INFO: computing task features
0.31/0.47	c INFO: computed variable-clause graph statistics
0.31/0.48	c INFO: computed clause balance statistics
0.40/0.52	c INFO: computed variable balance statistics
0.40/0.56	c INFO: computed small-clause counts
0.40/0.56	c INFO: computed Horn-clause counts
0.40/0.57	c INFO: computed variable graph statistics
0.40/0.57	c INFO: collected features for HOME/instance-3322031-1308045988.cnf in 0.10 s
0.50/0.60	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.58)
0.50/0.64	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.07)
0.50/0.68	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.41)
0.60/0.72	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.38)
0.60/0.76	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.34)
0.79/0.80	c INFO: running march_hi@0 for 679 with 39999 remaining (b = 0.30)
0.79/0.84	c INFO: running kcnfs-2006@0 for 582 with 39999 remaining (b = 0.26)
0.79/0.88	c INFO: running kcnfs-2006@0 for 582 with 39999 remaining (b = 0.23)
0.79/0.88	Traceback (most recent call last):
0.79/0.88	  File "HOME/solve", line 33, in <module>
0.79/0.88	    plac.call(main)
0.79/0.88	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
0.79/0.88	    cmd, result = parser_from(obj).consume(arglist)
0.79/0.88	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
0.79/0.88	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
0.79/0.88	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
0.79/0.88	    (_, answer, _) = solver(input_path, budget, cores)
0.79/0.88	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
0.79/0.88	    process.die()
0.79/0.88	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
0.79/0.88	    os.kill(self._process.pid, signal.SIGUSR1)
0.79/0.88	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /var/tmp/evaluation-result-3322031-1308045988/watcher-3322031-1308045988 -o /var/tmp/evaluation-result-3322031-1308045988/solver-3322031-1308045988 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322031-1308045988.cnf -seed 768940577 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.08 1.88 2.31 1/170 26236
/proc/meminfo: memFree=15565576/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=15184 CPUtime=0 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 735 0 0 0 0 0 0 0 22 0 1 0 897014408 15548416 622 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338381824 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/26236/statm: 3796 622 322 364 0 283 0

[startup+0.058381 s]
/proc/loadavg: 1.08 1.88 2.31 1/170 26236
/proc/meminfo: memFree=15565576/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=36504 CPUtime=0.03 cores=0-7
/proc/26236/stat : 26236 (python) R 26234 26236 25962 0 -1 4202496 1621 0 0 0 3 0 0 0 23 0 1 0 897014408 37380096 1450 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338381824 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/26236/statm: 9126 1450 452 364 0 943 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 36504

[startup+0.100373 s]
/proc/loadavg: 1.08 1.88 2.31 1/170 26236
/proc/meminfo: memFree=15565576/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=104224 CPUtime=0.06 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 3061 0 0 0 5 1 0 0 23 0 1 0 897014408 106725376 2742 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338381824 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/26236/statm: 26056 2742 847 364 0 1804 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104224

[startup+0.300338 s]
/proc/loadavg: 1.08 1.88 2.31 1/170 26236
/proc/meminfo: memFree=15565576/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=170752 CPUtime=0.23 cores=0-7
/proc/26236/stat : 26236 (python) R 26234 26236 25962 0 -1 4202496 6736 934 1 0 18 5 0 0 17 0 1 0 897014408 174850048 4808 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338381824 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/26236/statm: 42688 4808 1336 364 0 3453 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 170752

[startup+0.700174 s]
/proc/loadavg: 1.08 1.88 2.31 1/170 26236
/proc/meminfo: memFree=15565576/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=310016 CPUtime=0.6 cores=0-7
/proc/26236/stat : 26236 (python) R 26234 26236 25962 0 -1 4202496 11916 934 1 0 51 9 0 0 18 0 4 0 897014408 317456384 7258 33554432000 4194304 5683644 140736437348368 18446744073709551615 270341792437 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/26236/statm: 77504 7258 1878 364 0 13416 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 310016

[startup+1.50104 s]
/proc/loadavg: 1.08 1.88 2.31 7/183 26267
/proc/meminfo: memFree=15512672/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1.05 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15657 4013 1 0 69 10 25 1 25 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26248] ppid=26236 vsize=299640 CPUtime=0 cores=0-7
/proc/26248/stat : 26248 (python) S 26236 26236 25962 0 -1 4202560 762 0 0 0 0 0 0 0 15 0 1 0 897014477 306831360 5707 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 8127607549001171202 0 0 17 6 0 0 0
/proc/26248/statm: 74910 5707 348 364 0 10822 0
[pid=26250] ppid=26248 vsize=2680 CPUtime=0.8 cores=0-7
/proc/26250/stat : 26250 (march_hi) R 26248 26250 26250 0 -1 4202496 749 0 0 0 80 0 0 0 25 0 1 0 897014477 2744320 482 33554432000 4194304 4955518 140736858503104 18446744073709551615 4261424 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26250/statm: 670 482 104 186 0 483 0
[pid=26252] ppid=26236 vsize=310028 CPUtime=0 cores=0-7
/proc/26252/stat : 26252 (python) S 26236 26236 25962 0 -1 4202560 773 0 0 0 0 0 0 0 15 0 1 0 897014481 317468672 5728 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 1252893504145686788 0 0 17 6 0 0 0
/proc/26252/statm: 77507 5728 348 364 0 13416 0
[pid=26254] ppid=26252 vsize=2680 CPUtime=0.76 cores=0-7
/proc/26254/stat : 26254 (march_hi) R 26252 26254 26254 0 -1 4202496 745 0 0 0 76 0 0 0 22 0 1 0 897014481 2744320 481 33554432000 4194304 4955518 140737474625344 18446744073709551615 4227741 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26254/statm: 670 481 104 186 0 483 0
[pid=26255] ppid=26236 vsize=320284 CPUtime=0 cores=0-7
/proc/26255/stat : 26255 (python) S 26236 26236 25962 0 -1 4202560 772 0 0 0 0 0 0 0 15 0 1 0 897014485 327970816 5749 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 13384350651165671683 0 0 17 0 0 0 0
/proc/26255/statm: 80071 5749 348 364 0 15977 0
[pid=26257] ppid=26255 vsize=2680 CPUtime=0.72 cores=0-7
/proc/26257/stat : 26257 (march_hi) R 26255 26257 26257 0 -1 4202496 747 0 0 0 72 0 0 0 25 0 1 0 897014486 2744320 479 33554432000 4194304 4955518 140733542369968 18446744073709551615 4263664 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26257/statm: 670 479 104 186 0 483 0
[pid=26259] ppid=26236 vsize=330540 CPUtime=0 cores=0-7
/proc/26259/stat : 26259 (python) S 26236 26236 25962 0 -1 4202560 773 0 0 0 0 0 0 0 19 0 1 0 897014489 338472960 5753 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4458163413159215362 0 0 17 5 0 0 0
/proc/26259/statm: 82635 5753 348 364 0 18538 0
[pid=26261] ppid=26259 vsize=2680 CPUtime=0.68 cores=0-7
/proc/26261/stat : 26261 (march_hi) R 26259 26261 26261 0 -1 4202496 739 0 0 0 68 0 0 0 25 0 1 0 897014489 2744320 477 33554432000 4194304 4955518 140736621665264 18446744073709551615 4224679 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26261/statm: 670 477 104 186 0 483 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 736 0 0 0 0 0 0 0 25 0 1 0 897014493 416083968 5930 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4613779492861608196 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5930 348 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=0.64 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) R 26262 26264 26264 0 -1 4202496 640 0 0 0 64 0 0 0 25 0 1 0 897014493 186957824 385 33554432000 134512640 135163436 4287288464 18446744073709551615 134569431 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26264/statm: 45644 385 77 159 0 45484 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 774 0 0 0 0 0 0 0 25 0 1 0 897014497 426586112 5897 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 16289216391284752641 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5897 354 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=0.6 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) R 26265 26267 26267 0 -1 4202496 650 0 0 0 60 0 0 0 25 0 1 0 897014497 186966016 388 33554432000 134512640 135163436 4292525376 18446744073709551615 134596442 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/26267/statm: 45646 388 77 159 0 45486 0
Current children cumulated CPU time (s) 5.25
Current children cumulated vsize (KiB) 2834904

[startup+3.10073 s]
/proc/loadavg: 1.08 1.88 2.31 7/183 26267
/proc/meminfo: memFree=15511804/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1.05 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15657 4013 1 0 69 10 25 1 25 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26248] ppid=26236 vsize=299640 CPUtime=0 cores=0-7
/proc/26248/stat : 26248 (python) S 26236 26236 25962 0 -1 4202560 764 0 0 0 0 0 0 0 15 0 1 0 897014477 306831360 5707 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 8127607549001171202 0 0 17 6 0 0 0
/proc/26248/statm: 74910 5707 348 364 0 10822 0
[pid=26250] ppid=26248 vsize=3164 CPUtime=2.4 cores=0-7
/proc/26250/stat : 26250 (march_hi) R 26248 26250 26250 0 -1 4202496 817 0 0 0 240 0 0 0 25 0 1 0 897014477 3239936 550 33554432000 4194304 4955518 140736858503104 18446744073709551615 4227647 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26250/statm: 791 550 104 186 0 604 0
[pid=26252] ppid=26236 vsize=310028 CPUtime=0 cores=0-7
/proc/26252/stat : 26252 (python) S 26236 26236 25962 0 -1 4202560 775 0 0 0 0 0 0 0 15 0 1 0 897014481 317468672 5728 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 1252893504145686788 0 0 17 6 0 0 0
/proc/26252/statm: 77507 5728 348 364 0 13416 0
[pid=26254] ppid=26252 vsize=3164 CPUtime=2.36 cores=0-7
/proc/26254/stat : 26254 (march_hi) R 26252 26254 26254 0 -1 4202496 813 0 0 0 236 0 0 0 25 0 1 0 897014481 3239936 549 33554432000 4194304 4955518 140737474625344 18446744073709551615 4227884 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26254/statm: 791 549 104 186 0 604 0
[pid=26255] ppid=26236 vsize=320284 CPUtime=0 cores=0-7
/proc/26255/stat : 26255 (python) S 26236 26236 25962 0 -1 4202560 775 0 0 0 0 0 0 0 15 0 1 0 897014485 327970816 5749 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 13384350651165671683 0 0 17 0 0 0 0
/proc/26255/statm: 80071 5749 348 364 0 15977 0
[pid=26257] ppid=26255 vsize=3164 CPUtime=2.32 cores=0-7
/proc/26257/stat : 26257 (march_hi) R 26255 26257 26257 0 -1 4202496 814 0 0 0 232 0 0 0 25 0 1 0 897014486 3239936 546 33554432000 4194304 4955518 140733542369968 18446744073709551615 4227695 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26257/statm: 791 546 104 186 0 604 0
[pid=26259] ppid=26236 vsize=330540 CPUtime=0 cores=0-7
/proc/26259/stat : 26259 (python) S 26236 26236 25962 0 -1 4202560 776 0 0 0 0 0 0 0 15 0 1 0 897014489 338472960 5753 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4458163413159215362 0 0 17 5 0 0 0
/proc/26259/statm: 82635 5753 348 364 0 18538 0
[pid=26261] ppid=26259 vsize=3164 CPUtime=2.28 cores=0-7
/proc/26261/stat : 26261 (march_hi) R 26259 26261 26261 0 -1 4202496 807 0 0 0 228 0 0 0 25 0 1 0 897014489 3239936 545 33554432000 4194304 4955518 140736621665264 18446744073709551615 4306854 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26261/statm: 791 545 104 186 0 604 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 897014493 416083968 5930 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4613779492861608196 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5930 348 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=2.24 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) R 26262 26264 26264 0 -1 4202496 640 0 0 0 224 0 0 0 25 0 1 0 897014493 186957824 385 33554432000 134512640 135163436 4287288464 18446744073709551615 134675333 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26264/statm: 45644 385 77 159 0 45484 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 774 0 0 0 0 0 0 0 18 0 1 0 897014497 426586112 5897 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 16289216391284752641 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5897 354 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=2.2 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) R 26265 26267 26267 0 -1 4202496 650 0 0 0 220 0 0 0 25 0 1 0 897014497 186966016 388 33554432000 134512640 135163436 4292525376 18446744073709551615 134666464 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/26267/statm: 45646 388 77 159 0 45486 0
Current children cumulated CPU time (s) 14.85
Current children cumulated vsize (KiB) 2836840

[startup+6.30105 s]
/proc/loadavg: 1.47 1.95 2.33 7/183 26267
/proc/meminfo: memFree=15509324/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1.05 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15657 4013 1 0 69 10 25 1 25 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 4 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26248] ppid=26236 vsize=299640 CPUtime=0 cores=0-7
/proc/26248/stat : 26248 (python) S 26236 26236 25962 0 -1 4202560 907 0 0 0 0 0 0 0 15 0 1 0 897014477 306831360 5730 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 8127607549001171202 0 0 17 6 0 0 0
/proc/26248/statm: 74910 5730 371 364 0 10822 0
[pid=26250] ppid=26248 vsize=3876 CPUtime=5.6 cores=0-7
/proc/26250/stat : 26250 (march_hi) R 26248 26250 26250 0 -1 4202496 969 0 0 0 560 0 0 0 25 0 1 0 897014477 3969024 702 33554432000 4194304 4955518 140736858503104 18446744073709551615 4238478 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26250/statm: 969 702 104 186 0 782 0
[pid=26252] ppid=26236 vsize=310028 CPUtime=0 cores=0-7
/proc/26252/stat : 26252 (python) S 26236 26236 25962 0 -1 4202560 928 0 0 0 0 0 0 0 15 0 1 0 897014481 317468672 5751 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 1252893504145686788 0 0 17 6 0 0 0
/proc/26252/statm: 77507 5751 371 364 0 13416 0
[pid=26254] ppid=26252 vsize=3876 CPUtime=5.56 cores=0-7
/proc/26254/stat : 26254 (march_hi) R 26252 26254 26254 0 -1 4202496 966 0 0 0 556 0 0 0 25 0 1 0 897014481 3969024 702 33554432000 4194304 4955518 140737474625344 18446744073709551615 4298055 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26254/statm: 969 702 104 186 0 782 0
[pid=26255] ppid=26236 vsize=320284 CPUtime=0 cores=0-7
/proc/26255/stat : 26255 (python) S 26236 26236 25962 0 -1 4202560 924 0 0 0 0 0 0 0 15 0 1 0 897014485 327970816 5772 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 13384350651165671683 0 0 17 0 0 0 0
/proc/26255/statm: 80071 5772 371 364 0 15977 0
[pid=26257] ppid=26255 vsize=3876 CPUtime=5.51 cores=0-7
/proc/26257/stat : 26257 (march_hi) R 26255 26257 26257 0 -1 4202496 965 0 0 0 551 0 0 0 25 0 1 0 897014486 3969024 697 33554432000 4194304 4955518 140733542369968 18446744073709551615 4413869 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26257/statm: 969 697 104 186 0 782 0
[pid=26259] ppid=26236 vsize=330540 CPUtime=0 cores=0-7
/proc/26259/stat : 26259 (python) S 26236 26236 25962 0 -1 4202560 924 0 0 0 0 0 0 0 15 0 1 0 897014489 338472960 5776 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4458163413159215362 0 0 17 5 0 0 0
/proc/26259/statm: 82635 5776 371 364 0 18538 0
[pid=26261] ppid=26259 vsize=3876 CPUtime=5.48 cores=0-7
/proc/26261/stat : 26261 (march_hi) R 26259 26261 26261 0 -1 4202496 959 0 0 0 548 0 0 0 25 0 1 0 897014489 3969024 697 33554432000 4194304 4955518 140736621665264 18446744073709551615 4314020 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26261/statm: 969 697 104 186 0 782 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 810 0 0 0 0 0 0 0 18 0 1 0 897014493 416083968 5953 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4606961696135676164 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5953 371 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=5.44 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) R 26262 26264 26264 0 -1 4202496 640 0 0 0 544 0 0 0 25 0 1 0 897014493 186957824 385 33554432000 134512640 135163436 4287288464 18446744073709551615 134564031 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/26264/statm: 45644 385 77 159 0 45484 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 851 0 0 0 0 0 0 0 18 0 1 0 897014497 426586112 5919 33554432000 4194304 5683644 140736437348368 18446744073709551615 270338403910 0 0 16781312 514 4325455933099442433 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5919 376 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=5.4 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) R 26265 26267 26267 0 -1 4202496 650 0 0 0 540 0 0 0 25 0 1 0 897014497 186966016 388 33554432000 134512640 135163436 4292525376 18446744073709551615 134572057 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/26267/statm: 45646 388 77 159 0 45486 0
Current children cumulated CPU time (s) 34.04
Current children cumulated vsize (KiB) 2839688

[startup+12.7008 s]

################
# More data... #
################

Current children cumulated vsize (KiB) 1563692
heavy processes:

[startup+4902.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 26419
/proc/meminfo: memFree=15536800/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1555.65 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15660 64297 1 0 69 10 155389 97 18 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26252] ppid=26236 vsize=0 CPUtime=510.03 cores=0-7
/proc/26252/stat : 26252 (python) Z 26236 26236 25962 0 -1 4202588 1250 18848 0 0 35 27 50936 5 15 0 1 0 897014481 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/26252/statm: 0 0 0 0 0 0 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0.68 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 1021 0 0 0 32 36 0 0 19 0 2 0 897014493 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5983 400 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=586.18 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) T 26262 26264 26264 0 -1 4202496 641 0 0 0 58618 0 0 0 25 0 1 0 897014493 186957824 386 33554432000 134512640 135163436 4287288464 18446744073709551615 134569023 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26264/statm: 45644 386 77 159 0 45484 0
[pid=26262/tid=26288] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/task/26288/stat : 26288 (python) S 26236 26236 25962 0 -1 4202560 100 0 0 0 0 0 0 0 19 0 2 0 897073151 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453985335104 0 0 -1 1 0 0 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0.68 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 1043 0 0 0 38 30 0 0 18 0 2 0 897014497 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5950 406 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=586.59 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) T 26265 26267 26267 0 -1 4202496 651 0 0 0 58659 0 0 0 25 0 1 0 897014497 186966016 389 33554432000 134512640 135163436 4292525376 18446744073709551615 134675114 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26267/statm: 45646 389 77 159 0 45486 0
[pid=26265/tid=26289] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/task/26289/stat : 26289 (python) S 26236 26236 25962 0 -1 4202560 82 0 0 0 0 0 0 0 18 0 2 0 897073156 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453919791936 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3239.81
Current children cumulated vsize (KiB) 1563692
heavy processes:

[startup+4962.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 26421
/proc/meminfo: memFree=15536780/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1555.65 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15660 64297 1 0 69 10 155389 97 18 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26252] ppid=26236 vsize=0 CPUtime=510.03 cores=0-7
/proc/26252/stat : 26252 (python) Z 26236 26236 25962 0 -1 4202588 1250 18848 0 0 35 27 50936 5 15 0 1 0 897014481 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/26252/statm: 0 0 0 0 0 0 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0.68 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 1021 0 0 0 32 36 0 0 19 0 2 0 897014493 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5983 400 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=586.18 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) T 26262 26264 26264 0 -1 4202496 641 0 0 0 58618 0 0 0 25 0 1 0 897014493 186957824 386 33554432000 134512640 135163436 4287288464 18446744073709551615 134569023 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26264/statm: 45644 386 77 159 0 45484 0
[pid=26262/tid=26288] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/task/26288/stat : 26288 (python) S 26236 26236 25962 0 -1 4202560 100 0 0 0 0 0 0 0 19 0 2 0 897073151 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453985335104 0 0 -1 1 0 0 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0.68 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 1043 0 0 0 38 30 0 0 18 0 2 0 897014497 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5950 406 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=586.59 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) T 26265 26267 26267 0 -1 4202496 651 0 0 0 58659 0 0 0 25 0 1 0 897014497 186966016 389 33554432000 134512640 135163436 4292525376 18446744073709551615 134675114 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26267/statm: 45646 389 77 159 0 45486 0
[pid=26265/tid=26289] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/task/26289/stat : 26289 (python) S 26236 26236 25962 0 -1 4202560 82 0 0 0 0 0 0 0 18 0 2 0 897073156 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453919791936 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3239.81
Current children cumulated vsize (KiB) 1563692
heavy processes:

[startup+5022.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 26422
/proc/meminfo: memFree=15536864/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1555.65 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15660 64297 1 0 69 10 155389 97 18 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26252] ppid=26236 vsize=0 CPUtime=510.03 cores=0-7
/proc/26252/stat : 26252 (python) Z 26236 26236 25962 0 -1 4202588 1250 18848 0 0 35 27 50936 5 15 0 1 0 897014481 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/26252/statm: 0 0 0 0 0 0 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0.68 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 1021 0 0 0 32 36 0 0 19 0 2 0 897014493 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5983 400 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=586.18 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) T 26262 26264 26264 0 -1 4202496 641 0 0 0 58618 0 0 0 25 0 1 0 897014493 186957824 386 33554432000 134512640 135163436 4287288464 18446744073709551615 134569023 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26264/statm: 45644 386 77 159 0 45484 0
[pid=26262/tid=26288] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/task/26288/stat : 26288 (python) S 26236 26236 25962 0 -1 4202560 100 0 0 0 0 0 0 0 19 0 2 0 897073151 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453985335104 0 0 -1 1 0 0 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0.68 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 1043 0 0 0 38 30 0 0 18 0 2 0 897014497 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5950 406 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=586.59 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) T 26265 26267 26267 0 -1 4202496 651 0 0 0 58659 0 0 0 25 0 1 0 897014497 186966016 389 33554432000 134512640 135163436 4292525376 18446744073709551615 134675114 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26267/statm: 45646 389 77 159 0 45486 0
[pid=26265/tid=26289] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/task/26289/stat : 26289 (python) S 26236 26236 25962 0 -1 4202560 82 0 0 0 0 0 0 0 18 0 2 0 897073156 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453919791936 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3239.81
Current children cumulated vsize (KiB) 1563692
heavy processes:

[startup+5082.3 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 26424
/proc/meminfo: memFree=15536844/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1555.65 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15660 64297 1 0 69 10 155389 97 18 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26252] ppid=26236 vsize=0 CPUtime=510.03 cores=0-7
/proc/26252/stat : 26252 (python) Z 26236 26236 25962 0 -1 4202588 1250 18848 0 0 35 27 50936 5 15 0 1 0 897014481 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/26252/statm: 0 0 0 0 0 0 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0.68 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 1021 0 0 0 32 36 0 0 19 0 2 0 897014493 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5983 400 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=586.18 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) T 26262 26264 26264 0 -1 4202496 641 0 0 0 58618 0 0 0 25 0 1 0 897014493 186957824 386 33554432000 134512640 135163436 4287288464 18446744073709551615 134569023 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26264/statm: 45644 386 77 159 0 45484 0
[pid=26262/tid=26288] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/task/26288/stat : 26288 (python) S 26236 26236 25962 0 -1 4202560 100 0 0 0 0 0 0 0 19 0 2 0 897073151 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453985335104 0 0 -1 1 0 0 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0.68 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 1043 0 0 0 38 30 0 0 18 0 2 0 897014497 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5950 406 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=586.59 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) T 26265 26267 26267 0 -1 4202496 651 0 0 0 58659 0 0 0 25 0 1 0 897014497 186966016 389 33554432000 134512640 135163436 4292525376 18446744073709551615 134675114 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26267/statm: 45646 389 77 159 0 45486 0
[pid=26265/tid=26289] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/task/26289/stat : 26289 (python) S 26236 26236 25962 0 -1 4202560 82 0 0 0 0 0 0 0 18 0 2 0 897073156 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453919791936 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3239.81
Current children cumulated vsize (KiB) 1563692
heavy processes:



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 26424
/proc/meminfo: memFree=15536704/32951124 swapFree=67054460/67111528
[pid=26236] ppid=26234 vsize=375612 CPUtime=1555.65 cores=0-7
/proc/26236/stat : 26236 (python) S 26234 26236 25962 0 -1 4202496 15660 64297 1 0 69 10 155389 97 18 0 1 0 897014408 384626688 7418 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350213951 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/26236/statm: 93903 7418 1891 364 0 29800 0
[pid=26252] ppid=26236 vsize=0 CPUtime=510.03 cores=0-7
/proc/26252/stat : 26252 (python) Z 26236 26236 25962 0 -1 4202588 1250 18848 0 0 35 27 50936 5 15 0 1 0 897014481 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/26252/statm: 0 0 0 0 0 0 0
[pid=26262] ppid=26236 vsize=406332 CPUtime=0.68 cores=0-7
/proc/26262/stat : 26262 (python) S 26236 26236 25962 0 -1 4202560 1021 0 0 0 32 36 0 0 19 0 2 0 897014493 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26262/statm: 101583 5983 400 364 0 37483 0
[pid=26264] ppid=26262 vsize=182576 CPUtime=586.18 cores=0-7
/proc/26264/stat : 26264 (kcnfs-2006) T 26262 26264 26264 0 -1 4202496 641 0 0 0 58618 0 0 0 25 0 1 0 897014493 186957824 386 33554432000 134512640 135163436 4287288464 18446744073709551615 134569023 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26264/statm: 45644 386 77 159 0 45484 0
[pid=26262/tid=26288] ppid=26236 vsize=406332 CPUtime=0 cores=0-7
/proc/26262/task/26288/stat : 26288 (python) S 26236 26236 25962 0 -1 4202560 100 0 0 0 0 0 0 0 19 0 2 0 897073151 416083968 5983 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453985335104 0 0 -1 1 0 0 0
[pid=26265] ppid=26236 vsize=416588 CPUtime=0.68 cores=0-7
/proc/26265/stat : 26265 (python) S 26236 26236 25962 0 -1 4202560 1043 0 0 0 38 30 0 0 18 0 2 0 897014497 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350210507 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26265/statm: 104147 5950 406 364 0 40044 0
[pid=26267] ppid=26265 vsize=182584 CPUtime=586.59 cores=0-7
/proc/26267/stat : 26267 (kcnfs-2006) T 26265 26267 26267 0 -1 4202496 651 0 0 0 58659 0 0 0 25 0 1 0 897014497 186966016 389 33554432000 134512640 135163436 4292525376 18446744073709551615 134675114 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26267/statm: 45646 389 77 159 0 45486 0
[pid=26265/tid=26289] ppid=26236 vsize=416588 CPUtime=0 cores=0-7
/proc/26265/task/26289/stat : 26289 (python) S 26236 26236 25962 0 -1 4202560 82 0 0 0 0 0 0 0 18 0 2 0 897073156 426586112 5950 33554432000 4194304 5683644 140736437348368 18446744073709551615 270350207409 0 0 16781312 514 18446604453919791936 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3239.81
Current children cumulated vsize (KiB) 1563692
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 26236 and gives
#  childrusage.ru_utime.tv_sec=2065
#  childrusage.ru_utime.tv_usec=16069
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=85682
# CPU time returned by wait4() is 2067.1
# while last known CPU time is 3239.81
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.02
CPU time (s): 3239.81
CPU user time (s): 3237.76
CPU system time (s): 2.05
CPU usage (%): 63.5254
Max. virtual memory (cumulated for all children) (KiB): 3145964

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2065.02
system time used= 2.08568
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 104068
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5356
involuntary context switches= 7115

runsolver used 6.53301 second user time and 16.7505 second system time

The end

Launcher Data 

Begin job on node143 at 2011-06-14 12:06:28
IDJOB=3322031
IDBENCH=82809
IDSOLVER=1712
FILE ID=node143/3322031-1308045988
RUNJOBID= node143-1308045214-25976
PBS_JOBID= 13565699
Free space on /var/tmp= 2916 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v650-c2769-S1894319588-015.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /var/tmp/evaluation-result-3322031-1308045988/watcher-3322031-1308045988 -o /var/tmp/evaluation-result-3322031-1308045988/solver-3322031-1308045988 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322031-1308045988.cnf -seed 768940577 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c06f05996bba25e7e26c6579a49532fc
RANDOM SEED=768940577

node143.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15566128 kB
Buffers:        123776 kB
Cached:       16688172 kB
SwapCached:       9672 kB
Active:         550908 kB
Inactive:     16279220 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15566128 kB
SwapTotal:    67111528 kB
SwapFree:     67054460 kB
Dirty:           32232 kB
Writeback:           0 kB
AnonPages:       15852 kB
Mapped:          12684 kB
Slab:           487568 kB
PageTables:       4008 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183968 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /var/tmp at the end= 2884 MiB
End job on node143 at 2011-06-14 13:31:30