Trace number 3385612

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (exit code) 288.766 49.1114

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v400-c1704-S1050579435-065.SATISFIABLE.cnf
MD5SUMa9aff6c7ce1658eb3edc84aa7f38478c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.007998
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses1704
Sum of the clauses size5112
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 31704
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.32/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.46	c INFO: solving HOME/instance-3385612-1305570863.cnf
0.32/0.46	c INFO: computing task features
0.32/0.47	c INFO: computed variable-clause graph statistics
0.32/0.47	c INFO: computed clause balance statistics
0.41/0.50	c INFO: computed variable balance statistics
0.41/0.53	c INFO: computed small-clause counts
0.41/0.53	c INFO: computed Horn-clause counts
0.41/0.53	c INFO: computed variable graph statistics
0.41/0.53	c INFO: collected features for HOME/instance-3385612-1305570863.cnf in 0.06 s
0.41/0.56	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.56)
0.41/0.59	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.39)
0.52/0.62	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.34)
0.52/0.65	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.34)
0.52/0.68	c INFO: running march_hi@0 for 97 with 4999 remaining (b = 0.11)
0.68/0.71	c INFO: running TNM@0 for 97 with 4999 remaining (b = 0.07)
0.68/0.74	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.24)
0.68/0.78	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.21)
0.68/0.78	Traceback (most recent call last):
0.68/0.78	  File "HOME/solve", line 33, in <module>
0.68/0.78	    plac.call(main)
0.68/0.78	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
0.68/0.78	    cmd, result = parser_from(obj).consume(arglist)
0.68/0.78	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
0.68/0.78	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
0.68/0.78	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
0.68/0.78	    (_, answer, _) = solver(input_path, budget / cores, cores)
0.68/0.78	  File "HOME/borg/build/lib/borg/bilevel.py", line 226, in __call__
0.68/0.78	    process.die()
0.68/0.78	  File "HOME/borg/build/lib/borg/solvers.py", line 185, in die
0.68/0.78	    os.kill(self._process.pid, signal.SIGUSR1)
0.68/0.78	OSError: [Errno 3] No such process

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385612-1305570863/watcher-3385612-1305570863 -o /tmp/evaluation-result-3385612-1305570863/solver-3385612-1305570863 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385612-1305570863.cnf -seed 455596738 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.85 8.46 8.48 1/449 23313
/proc/meminfo: memFree=17864548/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=16240 CPUtime=0.01 cores=0-7
/proc/23313/stat : 23313 (python) R 23311 23313 22174 0 -1 4202496 1049 0 0 0 1 0 0 0 17 0 1 0 649501966 16629760 921 33554432000 4194304 5683644 140735208512544 18446744073709551615 5293775 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/23313/statm: 4060 921 381 364 0 547 0

[startup+0.0872739 s]
/proc/loadavg: 7.85 8.46 8.48 1/449 23313
/proc/meminfo: memFree=17864548/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=94188 CPUtime=0.05 cores=0-7
/proc/23313/stat : 23313 (python) R 23311 23313 22174 0 -1 4202496 2769 0 0 0 4 1 0 0 17 0 1 0 649501966 96448512 2477 33554432000 4194304 5683644 140735208512544 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/23313/statm: 23547 2478 765 364 0 1679 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 94188

[startup+0.100276 s]
/proc/loadavg: 7.85 8.46 8.48 1/449 23313
/proc/meminfo: memFree=17864548/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=104228 CPUtime=0.07 cores=0-7
/proc/23313/stat : 23313 (python) R 23311 23313 22174 0 -1 4202496 3058 0 0 0 5 2 0 0 17 0 1 0 649501966 106729472 2741 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726403072 0 2147483391 16781312 2 0 0 0 17 6 0 0 0
/proc/23313/statm: 26057 2741 847 364 0 1805 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104228

[startup+0.300235 s]
/proc/loadavg: 7.85 8.46 8.48 1/449 23313
/proc/meminfo: memFree=17864548/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=170756 CPUtime=0.23 cores=0-7
/proc/23313/stat : 23313 (python) R 23311 23313 22174 0 -1 4202496 6740 938 1 0 16 7 0 0 17 0 1 0 649501966 174854144 4809 33554432000 4194304 5683644 140735208512544 18446744073709551615 264721481351 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/23313/statm: 42689 4809 1336 364 0 3454 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 170756

[startup+0.70018 s]
/proc/loadavg: 7.85 8.46 8.48 1/449 23313
/proc/meminfo: memFree=17864548/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=395908 CPUtime=0.68 cores=0-7
/proc/23313/stat : 23313 (python) R 23311 23313 22174 0 -1 4202496 13300 2427 1 0 51 11 6 0 18 0 6 0 649501966 405409792 7412 33554432000 4194304 5683644 140735208512544 18446744073709551615 4620004 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/23313/statm: 98977 7412 1882 364 0 34883 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 395908

[startup+1.501 s]
/proc/loadavg: 7.85 8.46 8.48 7/462 23344
/proc/meminfo: memFree=17816944/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=0.84 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15419 3911 1 0 59 12 12 1 22 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23322] ppid=23313 vsize=289360 CPUtime=0 cores=0-7
/proc/23322/stat : 23322 (python) S 23313 23313 22174 0 -1 4202560 752 0 0 0 0 0 0 0 15 0 1 0 649502025 296304640 5697 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617489553504305410 0 0 17 1 0 0 0
/proc/23322/statm: 72340 5697 348 364 0 8255 0
[pid=23324] ppid=23322 vsize=2188 CPUtime=0.9 cores=0-7
/proc/23324/stat : 23324 (march_hi) R 23322 23324 23324 0 -1 4202496 626 0 0 0 90 0 0 0 18 0 1 0 649502025 2240512 361 33554432000 4194304 4955518 140733677059392 18446744073709551615 4209624 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23324/statm: 547 361 104 186 0 360 0
[pid=23325] ppid=23313 vsize=365152 CPUtime=0 cores=0-7
/proc/23325/stat : 23325 (python) S 23313 23313 22174 0 -1 4202560 710 0 0 0 0 0 0 0 15 0 1 0 649502028 373915648 5874 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617480757411283202 0 0 17 1 0 0 0
/proc/23325/statm: 91288 5874 348 364 0 27200 0
[pid=23327] ppid=23325 vsize=2188 CPUtime=0.87 cores=0-7
/proc/23327/stat : 23327 (march_hi) R 23325 23327 23327 0 -1 4202496 613 0 0 0 87 0 0 0 25 0 1 0 649502028 2240512 360 33554432000 4194304 4955518 140734032183072 18446744073709551615 4209701 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23327/statm: 547 360 104 186 0 360 0
[pid=23329] ppid=23313 vsize=375408 CPUtime=0 cores=0-7
/proc/23329/stat : 23329 (python) S 23313 23313 22174 0 -1 4202560 723 0 0 0 0 0 0 0 15 0 1 0 649502031 384417792 5835 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 14380820447233016066 0 0 17 7 0 0 0
/proc/23329/statm: 93852 5835 348 364 0 29761 0
[pid=23331] ppid=23329 vsize=2120 CPUtime=0.84 cores=0-7
/proc/23331/stat : 23331 (march_hi) R 23329 23331 23331 0 -1 4202496 615 0 0 0 84 0 0 0 23 0 1 0 649502031 2170880 359 33554432000 4194304 4955518 140734606082528 18446744073709551615 4209675 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23331/statm: 530 359 104 186 0 343 0
[pid=23332] ppid=23313 vsize=385664 CPUtime=0 cores=0-7
/proc/23332/stat : 23332 (python) S 23313 23313 22174 0 -1 4202560 728 0 0 0 0 0 0 0 15 0 1 0 649502034 394919936 5857 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3371071868730573057 0 0 17 0 0 0 0
/proc/23332/statm: 96416 5857 348 364 0 32322 0
[pid=23334] ppid=23332 vsize=2120 CPUtime=0.8 cores=0-7
/proc/23334/stat : 23334 (march_hi) R 23332 23334 23334 0 -1 4202496 616 0 0 0 80 0 0 0 22 0 1 0 649502035 2170880 358 33554432000 4194304 4955518 140734308731104 18446744073709551615 4263648 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23334/statm: 530 358 104 186 0 343 0
[pid=23338] ppid=23313 vsize=406176 CPUtime=0 cores=0-7
/proc/23338/stat : 23338 (python) S 23313 23313 22174 0 -1 4202560 748 0 0 0 0 0 0 0 15 0 1 0 649502041 415924224 5882 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3800840177702633730 0 0 17 6 0 0 0
/proc/23338/statm: 101544 5882 348 364 0 37444 0
[pid=23340] ppid=23338 vsize=2120 CPUtime=0.73 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 615 0 0 0 73 0 0 0 25 0 1 0 649502041 2170880 355 33554432000 4194304 4955518 140735587326960 18446744073709551615 4238413 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23340/statm: 530 355 104 186 0 343 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 786 0 0 0 0 0 0 0 17 0 1 0 649502044 426426368 5892 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5892 354 364 0 40005 0
[pid=23344] ppid=23342 vsize=2120 CPUtime=0.7 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 619 0 0 0 70 0 0 0 25 0 1 0 649502044 2170880 354 33554432000 4194304 4955518 140733277793936 18446744073709551615 4227500 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23344/statm: 530 354 104 186 0 343 0
Current children cumulated CPU time (s) 5.68
Current children cumulated vsize (KiB) 2626504

[startup+3.1007 s]
/proc/loadavg: 7.70 8.42 8.47 7/462 23344
/proc/meminfo: memFree=17814464/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=0.84 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15419 3911 1 0 59 12 12 1 22 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23322] ppid=23313 vsize=289360 CPUtime=0 cores=0-7
/proc/23322/stat : 23322 (python) S 23313 23313 22174 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 649502025 296304640 5697 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617489553504305410 0 0 17 1 0 0 0
/proc/23322/statm: 72340 5697 348 364 0 8255 0
[pid=23324] ppid=23322 vsize=2628 CPUtime=2.5 cores=0-7
/proc/23324/stat : 23324 (march_hi) R 23322 23324 23324 0 -1 4202496 703 0 0 0 250 0 0 0 22 0 1 0 649502025 2691072 438 33554432000 4194304 4955518 140733677059392 18446744073709551615 4298080 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23324/statm: 657 438 104 186 0 470 0
[pid=23325] ppid=23313 vsize=365152 CPUtime=0 cores=0-7
/proc/23325/stat : 23325 (python) S 23313 23313 22174 0 -1 4202560 713 0 0 0 0 0 0 0 15 0 1 0 649502028 373915648 5874 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617480757411283202 0 0 17 1 0 0 0
/proc/23325/statm: 91288 5874 348 364 0 27200 0
[pid=23327] ppid=23325 vsize=2628 CPUtime=2.46 cores=0-7
/proc/23327/stat : 23327 (march_hi) R 23325 23327 23327 0 -1 4202496 690 0 0 0 246 0 0 0 25 0 1 0 649502028 2691072 437 33554432000 4194304 4955518 140734032183072 18446744073709551615 4227651 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23327/statm: 657 437 104 186 0 470 0
[pid=23329] ppid=23313 vsize=375408 CPUtime=0 cores=0-7
/proc/23329/stat : 23329 (python) S 23313 23313 22174 0 -1 4202560 725 0 0 0 0 0 0 0 15 0 1 0 649502031 384417792 5835 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 14380820447233016066 0 0 17 7 0 0 0
/proc/23329/statm: 93852 5835 348 364 0 29761 0
[pid=23331] ppid=23329 vsize=2628 CPUtime=2.44 cores=0-7
/proc/23331/stat : 23331 (march_hi) R 23329 23331 23331 0 -1 4202496 693 0 0 0 244 0 0 0 25 0 1 0 649502031 2691072 437 33554432000 4194304 4955518 140734606082528 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23331/statm: 657 437 104 186 0 470 0
[pid=23332] ppid=23313 vsize=385664 CPUtime=0 cores=0-7
/proc/23332/stat : 23332 (python) S 23313 23313 22174 0 -1 4202560 729 0 0 0 0 0 0 0 15 0 1 0 649502034 394919936 5857 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3371071868730573057 0 0 17 0 0 0 0
/proc/23332/statm: 96416 5857 348 364 0 32322 0
[pid=23334] ppid=23332 vsize=2628 CPUtime=2.4 cores=0-7
/proc/23334/stat : 23334 (march_hi) R 23332 23334 23334 0 -1 4202496 694 0 0 0 240 0 0 0 25 0 1 0 649502035 2691072 436 33554432000 4194304 4955518 140734308731104 18446744073709551615 4227747 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23334/statm: 657 436 104 186 0 470 0
[pid=23338] ppid=23313 vsize=406176 CPUtime=0 cores=0-7
/proc/23338/stat : 23338 (python) S 23313 23313 22174 0 -1 4202560 751 0 0 0 0 0 0 0 15 0 1 0 649502041 415924224 5882 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3800840177702633730 0 0 17 6 0 0 0
/proc/23338/statm: 101544 5882 348 364 0 37444 0
[pid=23340] ppid=23338 vsize=2628 CPUtime=2.33 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 692 0 0 0 233 0 0 0 25 0 1 0 649502041 2691072 432 33554432000 4194304 4955518 140735587326960 18446744073709551615 4238717 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23340/statm: 657 432 104 186 0 470 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 787 0 0 0 0 0 0 0 15 0 1 0 649502044 426426368 5892 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5892 354 364 0 40005 0
[pid=23344] ppid=23342 vsize=2628 CPUtime=2.3 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 696 0 0 0 230 0 0 0 25 0 1 0 649502044 2691072 431 33554432000 4194304 4955518 140733277793936 18446744073709551615 4206520 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23344/statm: 657 431 104 186 0 470 0
Current children cumulated CPU time (s) 15.27
Current children cumulated vsize (KiB) 2629416

[startup+6.30112 s]
/proc/loadavg: 7.70 8.42 8.47 7/462 23344
/proc/meminfo: memFree=17811488/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=0.84 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15419 3911 1 0 59 12 12 1 22 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23322] ppid=23313 vsize=289360 CPUtime=0.01 cores=0-7
/proc/23322/stat : 23322 (python) S 23313 23313 22174 0 -1 4202560 897 0 0 0 1 0 0 0 15 0 1 0 649502025 296304640 5720 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617489553504305410 0 0 17 1 0 0 0
/proc/23322/statm: 72340 5720 371 364 0 8255 0
[pid=23324] ppid=23322 vsize=3480 CPUtime=5.7 cores=0-7
/proc/23324/stat : 23324 (march_hi) R 23322 23324 23324 0 -1 4202496 877 0 0 0 570 0 0 0 25 0 1 0 649502025 3563520 598 33554432000 4194304 4955518 140733677059392 18446744073709551615 4209669 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23324/statm: 870 598 104 186 0 683 0
[pid=23325] ppid=23313 vsize=365152 CPUtime=0.02 cores=0-7
/proc/23325/stat : 23325 (python) S 23313 23313 22174 0 -1 4202560 789 0 0 0 1 1 0 0 15 0 1 0 649502028 373915648 5897 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617480757411283202 0 0 17 1 0 0 0
/proc/23325/statm: 91288 5897 371 364 0 27200 0
[pid=23327] ppid=23325 vsize=3480 CPUtime=5.66 cores=0-7
/proc/23327/stat : 23327 (march_hi) R 23325 23327 23327 0 -1 4202496 864 0 0 0 566 0 0 0 25 0 1 0 649502028 3563520 597 33554432000 4194304 4955518 140734032183072 18446744073709551615 4224682 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23327/statm: 870 597 104 186 0 683 0
[pid=23329] ppid=23313 vsize=375408 CPUtime=0.02 cores=0-7
/proc/23329/stat : 23329 (python) S 23313 23313 22174 0 -1 4202560 808 0 0 0 1 1 0 0 15 0 1 0 649502031 384417792 5858 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 14380820447233016066 0 0 17 7 0 0 0
/proc/23329/statm: 93852 5858 371 364 0 29761 0
[pid=23331] ppid=23329 vsize=3480 CPUtime=5.64 cores=0-7
/proc/23331/stat : 23331 (march_hi) R 23329 23331 23331 0 -1 4202496 868 0 0 0 564 0 0 0 25 0 1 0 649502031 3563520 598 33554432000 4194304 4955518 140734606082528 18446744073709551615 4227654 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23331/statm: 870 598 104 186 0 683 0
[pid=23332] ppid=23313 vsize=385664 CPUtime=0.01 cores=0-7
/proc/23332/stat : 23332 (python) S 23313 23313 22174 0 -1 4202560 808 0 0 0 0 1 0 0 15 0 1 0 649502034 394919936 5880 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3371071868730573057 0 0 17 0 0 0 0
/proc/23332/statm: 96416 5880 371 364 0 32322 0
[pid=23334] ppid=23332 vsize=3480 CPUtime=5.6 cores=0-7
/proc/23334/stat : 23334 (march_hi) R 23332 23334 23334 0 -1 4202496 868 0 0 0 560 0 0 0 25 0 1 0 649502035 3563520 596 33554432000 4194304 4955518 140734308731104 18446744073709551615 4209688 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23334/statm: 870 596 104 186 0 683 0
[pid=23338] ppid=23313 vsize=406176 CPUtime=0.02 cores=0-7
/proc/23338/stat : 23338 (python) S 23313 23313 22174 0 -1 4202560 830 0 0 0 1 1 0 0 15 0 1 0 649502041 415924224 5904 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3800840177702633730 0 0 17 6 0 0 0
/proc/23338/statm: 101544 5904 370 364 0 37444 0
[pid=23340] ppid=23338 vsize=3480 CPUtime=5.53 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 867 0 0 0 553 0 0 0 25 0 1 0 649502041 3563520 593 33554432000 4194304 4955518 140735587326960 18446744073709551615 4209733 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23340/statm: 870 593 104 186 0 683 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0.01 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 867 0 0 0 1 0 0 0 15 0 1 0 649502044 426426368 5915 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5915 377 364 0 40005 0
[pid=23344] ppid=23342 vsize=3480 CPUtime=5.5 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 870 0 0 0 550 0 0 0 25 0 1 0 649502044 3563520 591 33554432000 4194304 4955518 140733277793936 18446744073709551615 4224638 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23344/statm: 870 591 104 186 0 683 0
Current children cumulated CPU time (s) 34.56
Current children cumulated vsize (KiB) 2634528

[startup+12.7009 s]

################
# More data... #
################

[pid=23340] ppid=23338 vsize=8740 CPUtime=37.46 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 1952 0 0 0 3746 0 0 0 25 0 1 0 649502041 8949760 1678 33554432000 4194304 4955518 140735587326960 18446744073709551615 4284871 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23340/statm: 2185 1678 104 186 0 1998 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0.13 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 874 0 0 0 7 6 0 0 15 0 1 0 649502044 426426368 5915 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5915 377 364 0 40005 0
[pid=23344] ppid=23342 vsize=8740 CPUtime=37.46 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 1952 0 0 0 3746 0 0 0 25 0 1 0 649502044 8949760 1673 33554432000 4194304 4955518 140733277793936 18446744073709551615 4224642 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23344/statm: 2185 1673 104 186 0 1998 0
Current children cumulated CPU time (s) 227.13
Current children cumulated vsize (KiB) 2666088

[startup+44.7008 s]
/proc/loadavg: 7.01 8.15 8.37 7/462 23345
/proc/meminfo: memFree=17783472/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=0.84 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15419 3911 1 0 59 12 12 1 22 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23322] ppid=23313 vsize=289360 CPUtime=0.17 cores=0-7
/proc/23322/stat : 23322 (python) S 23313 23313 22174 0 -1 4202560 921 0 0 0 9 8 0 0 15 0 1 0 649502025 296304640 5720 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617489553504305410 0 0 17 1 0 0 0
/proc/23322/statm: 72340 5720 371 364 0 8255 0
[pid=23324] ppid=23322 vsize=10220 CPUtime=44.08 cores=0-7
/proc/23324/stat : 23324 (march_hi) R 23322 23324 23324 0 -1 4202496 2158 0 0 0 4408 0 0 0 25 0 1 0 649502025 10465280 1879 33554432000 4194304 4955518 140733677059392 18446744073709551615 4224738 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23324/statm: 2555 1879 104 186 0 2368 0
[pid=23325] ppid=23313 vsize=365152 CPUtime=0.16 cores=0-7
/proc/23325/stat : 23325 (python) S 23313 23313 22174 0 -1 4202560 794 0 0 0 8 8 0 0 15 0 1 0 649502028 373915648 5897 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617480757411283202 0 0 17 7 0 0 0
/proc/23325/statm: 91288 5897 371 364 0 27200 0
[pid=23327] ppid=23325 vsize=10220 CPUtime=44.06 cores=0-7
/proc/23327/stat : 23327 (march_hi) R 23325 23327 23327 0 -1 4202496 2148 0 0 0 4406 0 0 0 25 0 1 0 649502028 10465280 1881 33554432000 4194304 4955518 140734032183072 18446744073709551615 4235390 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23327/statm: 2555 1881 104 186 0 2368 0
[pid=23329] ppid=23313 vsize=375408 CPUtime=0.15 cores=0-7
/proc/23329/stat : 23329 (python) S 23313 23313 22174 0 -1 4202560 816 0 0 0 8 7 0 0 15 0 1 0 649502031 384417792 5858 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 14380820447233016066 0 0 17 2 0 0 0
/proc/23329/statm: 93852 5858 371 364 0 29761 0
[pid=23331] ppid=23329 vsize=10220 CPUtime=44.03 cores=0-7
/proc/23331/stat : 23331 (march_hi) R 23329 23331 23331 0 -1 4202496 2152 0 0 0 4403 0 0 0 25 0 1 0 649502031 10465280 1882 33554432000 4194304 4955518 140734606082528 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23331/statm: 2555 1882 104 186 0 2368 0
[pid=23332] ppid=23313 vsize=385664 CPUtime=0.16 cores=0-7
/proc/23332/stat : 23332 (python) S 23313 23313 22174 0 -1 4202560 815 0 0 0 8 8 0 0 15 0 1 0 649502034 394919936 5880 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3371071868730573057 0 0 17 0 0 0 0
/proc/23332/statm: 96416 5880 371 364 0 32322 0
[pid=23334] ppid=23332 vsize=10220 CPUtime=43.95 cores=0-7
/proc/23334/stat : 23334 (march_hi) R 23332 23334 23334 0 -1 4202496 2149 0 0 0 4395 0 0 0 25 0 1 0 649502035 10465280 1877 33554432000 4194304 4955518 140734308731104 18446744073709551615 4209664 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23334/statm: 2555 1877 104 186 0 2368 0
[pid=23338] ppid=23313 vsize=406176 CPUtime=0.16 cores=0-7
/proc/23338/stat : 23338 (python) S 23313 23313 22174 0 -1 4202560 840 0 0 0 9 7 0 0 15 0 1 0 649502041 415924224 5905 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3800840177702633730 0 0 17 6 0 0 0
/proc/23338/statm: 101544 5905 371 364 0 37444 0
[pid=23340] ppid=23338 vsize=10220 CPUtime=43.85 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 2148 0 0 0 4385 0 0 0 25 0 1 0 649502041 10465280 1874 33554432000 4194304 4955518 140735587326960 18446744073709551615 4286784 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23340/statm: 2555 1874 104 186 0 2368 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0.16 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 874 0 0 0 9 7 0 0 15 0 1 0 649502044 426426368 5915 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5915 377 364 0 40005 0
[pid=23344] ppid=23342 vsize=10220 CPUtime=43.85 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 2146 0 0 0 4385 0 0 0 25 0 1 0 649502044 10465280 1867 33554432000 4194304 4955518 140733277793936 18446744073709551615 4227583 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23344/statm: 2555 1867 104 186 0 2368 0
Current children cumulated CPU time (s) 265.62
Current children cumulated vsize (KiB) 2674968

[startup+46.3005 s]
/proc/loadavg: 7.01 8.15 8.37 7/462 23345
/proc/meminfo: memFree=17781920/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=0.84 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15419 3911 1 0 59 12 12 1 22 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23322] ppid=23313 vsize=289360 CPUtime=0.17 cores=0-7
/proc/23322/stat : 23322 (python) S 23313 23313 22174 0 -1 4202560 921 0 0 0 9 8 0 0 15 0 1 0 649502025 296304640 5720 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617489553504305410 0 0 17 1 0 0 0
/proc/23322/statm: 72340 5720 371 364 0 8255 0
[pid=23324] ppid=23322 vsize=10220 CPUtime=45.68 cores=0-7
/proc/23324/stat : 23324 (march_hi) R 23322 23324 23324 0 -1 4202496 2206 0 0 0 4568 0 0 0 25 0 1 0 649502025 10465280 1927 33554432000 4194304 4955518 140733677059392 18446744073709551615 4224631 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23324/statm: 2555 1927 104 186 0 2368 0
[pid=23325] ppid=23313 vsize=365152 CPUtime=0.16 cores=0-7
/proc/23325/stat : 23325 (python) S 23313 23313 22174 0 -1 4202560 794 0 0 0 8 8 0 0 15 0 1 0 649502028 373915648 5897 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 15617480757411283202 0 0 17 7 0 0 0
/proc/23325/statm: 91288 5897 371 364 0 27200 0
[pid=23327] ppid=23325 vsize=10220 CPUtime=45.66 cores=0-7
/proc/23327/stat : 23327 (march_hi) R 23325 23327 23327 0 -1 4202496 2197 0 0 0 4566 0 0 0 25 0 1 0 649502028 10465280 1930 33554432000 4194304 4955518 140734032183072 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23327/statm: 2555 1930 104 186 0 2368 0
[pid=23329] ppid=23313 vsize=375408 CPUtime=0.15 cores=0-7
/proc/23329/stat : 23329 (python) S 23313 23313 22174 0 -1 4202560 816 0 0 0 8 7 0 0 15 0 1 0 649502031 384417792 5858 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 14380820447233016066 0 0 17 2 0 0 0
/proc/23329/statm: 93852 5858 371 364 0 29761 0
[pid=23331] ppid=23329 vsize=10220 CPUtime=45.63 cores=0-7
/proc/23331/stat : 23331 (march_hi) R 23329 23331 23331 0 -1 4202496 2202 0 0 0 4563 0 0 0 25 0 1 0 649502031 10465280 1932 33554432000 4194304 4955518 140734606082528 18446744073709551615 4235369 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23331/statm: 2555 1932 104 186 0 2368 0
[pid=23332] ppid=23313 vsize=385664 CPUtime=0.16 cores=0-7
/proc/23332/stat : 23332 (python) S 23313 23313 22174 0 -1 4202560 815 0 0 0 8 8 0 0 15 0 1 0 649502034 394919936 5880 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3371071868730573057 0 0 17 0 0 0 0
/proc/23332/statm: 96416 5880 371 364 0 32322 0
[pid=23334] ppid=23332 vsize=10220 CPUtime=45.55 cores=0-7
/proc/23334/stat : 23334 (march_hi) R 23332 23334 23334 0 -1 4202496 2198 0 0 0 4555 0 0 0 25 0 1 0 649502035 10465280 1926 33554432000 4194304 4955518 140734308731104 18446744073709551615 4227522 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23334/statm: 2555 1926 104 186 0 2368 0
[pid=23338] ppid=23313 vsize=406176 CPUtime=0.16 cores=0-7
/proc/23338/stat : 23338 (python) S 23313 23313 22174 0 -1 4202560 840 0 0 0 9 7 0 0 15 0 1 0 649502041 415924224 5905 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 3800840177702633730 0 0 17 6 0 0 0
/proc/23338/statm: 101544 5905 371 364 0 37444 0
[pid=23340] ppid=23338 vsize=10220 CPUtime=45.45 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 2196 0 0 0 4545 0 0 0 25 0 1 0 649502041 10465280 1922 33554432000 4194304 4955518 140735587326960 18446744073709551615 4209699 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/23340/statm: 2555 1922 104 186 0 2368 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0.16 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 874 0 0 0 9 7 0 0 15 0 1 0 649502044 426426368 5915 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5915 377 364 0 40005 0
[pid=23344] ppid=23342 vsize=10220 CPUtime=45.45 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 2188 0 0 0 4545 0 0 0 25 0 1 0 649502044 10465280 1909 33554432000 4194304 4955518 140733277793936 18446744073709551615 4224704 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23344/statm: 2555 1909 104 186 0 2368 0
Current children cumulated CPU time (s) 275.22
Current children cumulated vsize (KiB) 2674968

[startup+47.9012 s]
/proc/loadavg: 6.93 8.11 8.36 7/462 23345
/proc/meminfo: memFree=17781300/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=0.84 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15419 3911 1 0 59 12 12 1 22 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 6 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23322] ppid=23313 vsize=289360 CPUtime=0.19 cores=0-7
/proc/23322/stat : 23322 (python) S 23313 23313 22174 0 -1 4202560 925 0 0 0 10 9 0 0 15 0 1 0 649502025 296304640 5720 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 9950967259465744642 0 0 17 1 0 0 0
/proc/23322/statm: 72340 5720 371 364 0 8255 0
[pid=23324] ppid=23322 vsize=10220 CPUtime=47.28 cores=0-7
/proc/23324/stat : 23324 (march_hi) R 23322 23324 23324 0 -1 4202496 2256 0 0 0 4728 0 0 0 25 0 1 0 649502025 10465280 1977 33554432000 4194304 4955518 140733677059392 18446744073709551615 4209711 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/23324/statm: 2555 1977 104 186 0 2368 0
[pid=23325] ppid=23313 vsize=365152 CPUtime=0.18 cores=0-7
/proc/23325/stat : 23325 (python) S 23313 23313 22174 0 -1 4202560 794 0 0 0 9 9 0 0 15 0 1 0 649502028 373915648 5897 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 9950967259465744642 0 0 17 7 0 0 0
/proc/23325/statm: 91288 5897 371 364 0 27200 0
[pid=23327] ppid=23325 vsize=10220 CPUtime=47.26 cores=0-7
/proc/23327/stat : 23327 (march_hi) R 23325 23327 23327 0 -1 4202496 2247 0 0 0 4726 0 0 0 25 0 1 0 649502028 10465280 1980 33554432000 4194304 4955518 140734032183072 18446744073709551615 4238484 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23327/statm: 2555 1980 104 186 0 2368 0
[pid=23329] ppid=23313 vsize=375408 CPUtime=0.18 cores=0-7
/proc/23329/stat : 23329 (python) S 23313 23313 22174 0 -1 4202560 816 0 0 0 10 8 0 0 15 0 1 0 649502031 384417792 5858 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 9950967259465744642 0 0 17 2 0 0 0
/proc/23329/statm: 93852 5858 371 364 0 29761 0
[pid=23331] ppid=23329 vsize=10220 CPUtime=47.23 cores=0-7
/proc/23331/stat : 23331 (march_hi) R 23329 23331 23331 0 -1 4202496 2252 0 0 0 4723 0 0 0 25 0 1 0 649502031 10465280 1982 33554432000 4194304 4955518 140734606082528 18446744073709551615 4286784 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/23331/statm: 2555 1982 104 186 0 2368 0
[pid=23332] ppid=23313 vsize=385664 CPUtime=0.19 cores=0-7
/proc/23332/stat : 23332 (python) S 23313 23313 22174 0 -1 4202560 815 0 0 0 10 9 0 0 15 0 1 0 649502034 394919936 5880 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 9950967259465744641 0 0 17 0 0 0 0
/proc/23332/statm: 96416 5880 371 364 0 32322 0
[pid=23334] ppid=23332 vsize=10220 CPUtime=47.15 cores=0-7
/proc/23334/stat : 23334 (march_hi) R 23332 23334 23334 0 -1 4202496 2247 0 0 0 4715 0 0 0 25 0 1 0 649502035 10465280 1975 33554432000 4194304 4955518 140734308731104 18446744073709551615 4209747 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23334/statm: 2555 1975 104 186 0 2368 0
[pid=23338] ppid=23313 vsize=406176 CPUtime=0.18 cores=0-7
/proc/23338/stat : 23338 (python) S 23313 23313 22174 0 -1 4202560 840 0 0 0 10 8 0 0 15 0 1 0 649502041 415924224 5905 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 9950967259465744642 0 0 17 6 0 0 0
/proc/23338/statm: 101544 5905 371 364 0 37444 0
[pid=23340] ppid=23338 vsize=10220 CPUtime=47.05 cores=0-7
/proc/23340/stat : 23340 (march_hi) R 23338 23340 23340 0 -1 4202496 2245 0 0 0 4705 0 0 0 25 0 1 0 649502041 10465280 1971 33554432000 4194304 4955518 140735587326960 18446744073709551615 4244449 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/23340/statm: 2555 1971 104 186 0 2368 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0.18 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 875 0 0 0 10 8 0 0 15 0 1 0 649502044 426426368 5915 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 9950967259465744643 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5915 377 364 0 40005 0
[pid=23344] ppid=23342 vsize=10220 CPUtime=47.05 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 2236 0 0 0 4705 0 0 0 25 0 1 0 649502044 10465280 1957 33554432000 4194304 4955518 140733277793936 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23344/statm: 2555 1957 104 186 0 2368 0
Current children cumulated CPU time (s) 284.96
Current children cumulated vsize (KiB) 2674968

[startup+48.7011 s]
/proc/loadavg: 6.93 8.11 8.36 5/460 23347
/proc/meminfo: memFree=17799784/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375456 CPUtime=240.3 cores=0-7
/proc/23313/stat : 23313 (python) S 23311 23313 22174 0 -1 4202496 15421 20866 1 0 59 12 23908 51 15 0 1 0 649501966 384466944 7413 33554432000 4194304 5683644 140735208512544 18446744073709551615 264738235199 0 0 16781312 2 18446744071562232103 0 0 17 0 0 0 0
/proc/23313/statm: 93864 7413 1891 364 0 29761 0
[pid=23342] ppid=23313 vsize=416432 CPUtime=0.18 cores=0-7
/proc/23342/stat : 23342 (python) S 23313 23313 22174 0 -1 4202560 875 0 0 0 10 8 0 0 15 0 1 0 649502044 426426368 5915 33554432000 4194304 5683644 140735208512544 18446744073709551615 264726425158 0 0 16781312 514 1656691348469678339 0 0 17 0 0 0 0
/proc/23342/statm: 104108 5915 377 364 0 40005 0
[pid=23344] ppid=23342 vsize=10220 CPUtime=47.84 cores=0-7
/proc/23344/stat : 23344 (march_hi) R 23342 23344 23344 0 -1 4202496 2260 0 0 0 4784 0 0 0 25 0 1 0 649502044 10465280 1981 33554432000 4194304 4955518 140733277793936 18446744073709551615 4224651 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23344/statm: 2555 1981 104 186 0 2368 0
Current children cumulated CPU time (s) 288.32
Current children cumulated vsize (KiB) 802108

[startup+49.1011 s]
/proc/loadavg: 6.93 8.11 8.36 5/460 23347
/proc/meminfo: memFree=17799784/32951124 swapFree=63083184/67111528
[pid=23313] ppid=23311 vsize=375348 CPUtime=288.74 cores=0-7
/proc/23313/stat : 23313 (python) R 23311 23313 22174 0 -1 4202496 18610 24304 1 0 60 12 28742 60 15 0 1 0 649501966 384356352 7362 33554432000 4194304 5683644 140735208512544 18446744073709551615 4501813 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23313/statm: 93837 7362 1874 364 0 29761 0
Current children cumulated CPU time (s) 288.74
Current children cumulated vsize (KiB) 375348

Child status: 1
Real time (s): 49.1114
CPU time (s): 288.766
CPU user time (s): 288.032
CPU system time (s): 0.733888
CPU usage (%): 587.982
Max. virtual memory (cumulated for all children) (KiB): 2674968

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 288.032
system time used= 0.733888
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 43950
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3597
involuntary context switches= 1581

runsolver used 0.180972 second user time and 0.539917 second system time

The end

Launcher Data 

Begin job on node141 at 2011-05-16 20:34:23
IDJOB=3385612
IDBENCH=82877
IDSOLVER=1870
FILE ID=node141/3385612-1305570863
RUNJOBID= node141-1305564961-22188
PBS_JOBID= 13350109
Free space on /tmp= 73260 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v400-c1704-S1050579435-065.SATISFIABLE.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385612-1305570863/watcher-3385612-1305570863 -o /tmp/evaluation-result-3385612-1305570863/solver-3385612-1305570863 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385612-1305570863.cnf -seed 455596738 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a9aff6c7ce1658eb3edc84aa7f38478c
RANDOM SEED=455596738

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17865084 kB
Buffers:        293720 kB
Cached:        5692792 kB
SwapCached:      15716 kB
Active:        9415788 kB
Inactive:      5386280 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17865084 kB
SwapTotal:    67111528 kB
SwapFree:     63083184 kB
Dirty:           31868 kB
Writeback:           0 kB
AnonPages:     8807576 kB
Mapped:          19656 kB
Slab:           149644 kB
PageTables:      61664 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 124936236 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73228 MiB
End job on node141 at 2011-05-16 20:35:13