Trace number 3322294

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-randomUNSAT 2300.23 320.631

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v100-c2130-S815408325-054.UNKNOWN.cnf
MD5SUM9830e1bf79d13e57ceeaa969f4476a02
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark313.313
Satisfiable
(Un)Satisfiability was proved
Number of variables100
Number of clauses2130
Sum of the clauses size10650
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52130
Number of clauses of size over 50

Solver Data 

0.30/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.47	c INFO: solving HOME/instance-3322294-1304380005.cnf
0.30/0.47	c INFO: computing task features
0.30/0.48	c INFO: computed variable-clause graph statistics
0.30/0.48	c INFO: computed clause balance statistics
0.40/0.53	c INFO: computed variable balance statistics
0.40/0.56	c INFO: computed small-clause counts
0.40/0.57	c INFO: computed Horn-clause counts
0.40/0.57	c INFO: computed variable graph statistics
0.40/0.57	c INFO: collected features for HOME/instance-3322294-1304380005.cnf in 0.10 s
0.50/0.60	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.62)
0.50/0.65	c INFO: running march_hi@0 for 194 with 39999 remaining (b = 0.19)
0.50/0.68	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.07)
0.59/0.72	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.29)
0.59/0.76	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.26)
0.69/0.80	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.27)
0.69/0.84	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.24)
0.69/0.88	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.21)
700.81/100.94	c INFO: running kcnfs-2006@0 for 582 with 39899 remaining (b = 0.21)
1036.34/148.96	c INFO: running march_hi@0 for 388 with 39851 remaining (b = 0.14)
1408.78/198.76	c INFO: running march_hi@0 for 97 with 39801 remaining (b = 0.04)
2155.27/299.84	c INFO: running march_hi@97 for 97 with 39700 remaining (b = 0.04)
1662.51/320.60	s UNSATISFIABLE

Verifier Data 

No possible verification on an UNSAT instance

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322294-1304380005/watcher-3322294-1304380005 -o /tmp/evaluation-result-3322294-1304380005/solver-3322294-1304380005 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322294-1304380005.cnf -seed 660750185 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.69 9.22 9.41 2/304 8512
/proc/meminfo: memFree=12947952/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=15776 CPUtime=0 cores=0-7
/proc/8512/stat : 8512 (python) S 8510 8512 6630 0 -1 4202496 928 0 0 0 0 0 0 0 24 0 1 0 549600960 16154624 807 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446943232 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/8512/statm: 3944 807 356 364 0 431 0

[startup+0.0334959 s]
/proc/loadavg: 8.69 9.22 9.41 2/304 8512
/proc/meminfo: memFree=12947952/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=18884 CPUtime=0.01 cores=0-7
/proc/8512/stat : 8512 (python) R 8510 8512 6630 0 -1 4202496 1284 0 0 0 1 0 0 0 24 0 1 0 549600960 19337216 1136 33554432000 4194304 5683644 140733475178832 18446744073709551615 4870096 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/8512/statm: 4721 1136 421 364 0 690 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 18884

[startup+0.100483 s]
/proc/loadavg: 8.69 9.22 9.41 2/304 8512
/proc/meminfo: memFree=12947952/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=104224 CPUtime=0.07 cores=0-7
/proc/8512/stat : 8512 (python) R 8510 8512 6630 0 -1 4202496 3053 0 0 0 5 2 0 0 24 0 1 0 549600960 106725376 2736 33554432000 4194304 5683644 140733475178832 18446744073709551615 4539757 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/8512/statm: 26056 2740 847 364 0 1804 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104224

[startup+0.300414 s]
/proc/loadavg: 8.69 9.22 9.41 2/304 8512
/proc/meminfo: memFree=12947952/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=170752 CPUtime=0.21 cores=0-7
/proc/8512/stat : 8512 (python) S 8510 8512 6630 0 -1 4202496 6728 934 1 0 16 5 0 0 17 0 1 0 549600960 174850048 4805 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446943232 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/8512/statm: 42688 4805 1336 364 0 3453 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 170752

[startup+0.700356 s]
/proc/loadavg: 8.69 9.22 9.41 2/304 8512
/proc/meminfo: memFree=12947952/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=309776 CPUtime=0.59 cores=0-7
/proc/8512/stat : 8512 (python) R 8510 8512 6630 0 -1 4202496 11880 934 1 0 50 9 0 0 19 0 4 0 549600960 317210624 7214 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446306054 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/8512/statm: 77444 7214 1878 364 0 13356 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 309776

[startup+1.50118 s]
/proc/loadavg: 8.69 9.22 9.41 12/329 8541
/proc/meminfo: memFree=12891068/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=361224 CPUtime=0.77 cores=0-7
/proc/8512/stat : 8512 (python) S 8510 8512 6630 0 -1 4202496 15704 934 1 0 67 10 0 0 25 0 9 0 549600960 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/8512/statm: 90306 7271 1889 364 0 26203 0
[pid=8518] ppid=8512 vsize=278584 CPUtime=0 cores=0-7
/proc/8518/stat : 8518 (python) S 8512 8512 6630 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 549601021 285270016 5559 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965263 0 0 16781312 514 10110862917596348415 0 0 17 4 0 0 0
/proc/8518/statm: 69646 5559 354 364 0 5564 0
[pid=8520] ppid=8518 vsize=1017800 CPUtime=0.66 cores=0-7
/proc/8520/stat : 8520 (gnovelty+2) R 8518 8520 8520 0 -1 4202496 436 0 0 0 66 0 0 0 21 0 1 0 549601021 1042227200 171 33554432000 134512640 135663704 4288802688 18446744073709551615 134525684 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8520/statm: 254450 171 113 282 0 254166 0
[pid=8521] ppid=8512 vsize=289276 CPUtime=0 cores=0-7
/proc/8521/stat : 8521 (python) S 8512 8512 6630 0 -1 4202560 745 0 0 0 0 0 0 0 15 0 1 0 549601025 296218624 5676 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 5602269475916185858 0 0 17 2 0 0 0
/proc/8521/statm: 72319 5676 348 364 0 8234 0
[pid=8523] ppid=8521 vsize=1560 CPUtime=0.84 cores=0-7
/proc/8523/stat : 8523 (march_hi) R 8521 8523 8523 0 -1 4202496 506 0 0 0 84 0 0 0 25 0 1 0 549601026 1597440 240 33554432000 4194304 4955518 140734909562464 18446744073709551615 4208704 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/8523/statm: 390 240 106 186 0 203 0
[pid=8524] ppid=8512 vsize=299532 CPUtime=0 cores=0-7
/proc/8524/stat : 8524 (python) S 8512 8512 6630 0 -1 4202560 740 0 0 0 0 0 0 0 19 0 1 0 549601029 306720768 5680 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 6366968650414424063 0 0 17 0 0 0 0
/proc/8524/statm: 74883 5680 348 364 0 10795 0
[pid=8526] ppid=8524 vsize=1017800 CPUtime=0.79 cores=0-7
/proc/8526/stat : 8526 (gnovelty+2) R 8524 8526 8526 0 -1 4202496 435 0 0 0 79 0 0 0 22 0 1 0 549601030 1042227200 172 33554432000 134512640 135663704 4291936160 18446744073709551615 134524540 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8526/statm: 254450 172 113 282 0 254166 0
[pid=8527] ppid=8512 vsize=309788 CPUtime=0 cores=0-7
/proc/8527/stat : 8527 (python) S 8512 8512 6630 0 -1 4202560 753 0 0 0 0 0 0 0 18 0 1 0 549601033 317222912 5684 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 10301707074974744833 0 0 17 4 0 0 0
/proc/8527/statm: 77447 5684 348 364 0 13356 0
[pid=8529] ppid=8527 vsize=1560 CPUtime=0.76 cores=0-7
/proc/8529/stat : 8529 (march_hi) R 8527 8529 8529 0 -1 4202496 504 0 0 0 76 0 0 0 25 0 1 0 549601033 1597440 239 33554432000 4194304 4955518 140734502472512 18446744073709551615 4308598 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/8529/statm: 390 239 106 186 0 203 0
[pid=8530] ppid=8512 vsize=320044 CPUtime=0 cores=0-7
/proc/8530/stat : 8530 (python) S 8512 8512 6630 0 -1 4202560 769 0 0 0 0 0 0 0 16 0 1 0 549601037 327725056 5697 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 12912686551128866815 0 0 17 1 0 0 0
/proc/8530/statm: 80011 5697 348 364 0 15917 0
[pid=8532] ppid=8530 vsize=1560 CPUtime=0.72 cores=0-7
/proc/8532/stat : 8532 (march_hi) R 8530 8532 8532 0 -1 4202496 506 0 0 0 72 0 0 0 22 0 1 0 549601037 1597440 240 33554432000 4194304 4955518 140734864802688 18446744073709551615 4237786 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8532/statm: 390 240 106 186 0 203 0
[pid=8533] ppid=8512 vsize=330468 CPUtime=0 cores=0-7
/proc/8533/stat : 8533 (python) S 8512 8512 6630 0 -1 4202560 779 0 0 0 0 0 0 0 17 0 1 0 549601041 338399232 5718 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 17475092458399301890 0 0 17 6 0 0 0
/proc/8533/statm: 82617 5718 348 364 0 18520 0
[pid=8535] ppid=8533 vsize=1560 CPUtime=0.67 cores=0-7
/proc/8535/stat : 8535 (march_hi) R 8533 8535 8535 0 -1 4202496 506 0 0 0 67 0 0 0 25 0 1 0 549601041 1597440 240 33554432000 4194304 4955518 140737159099760 18446744073709551615 4235634 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/8535/statm: 390 240 106 186 0 203 0
[pid=8536] ppid=8512 vsize=340724 CPUtime=0 cores=0-7
/proc/8536/stat : 8536 (python) S 8512 8512 6630 0 -1 4202560 780 0 0 0 0 0 0 0 18 0 1 0 549601045 348901376 5722 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 5602269475916185858 0 0 17 0 0 0 0
/proc/8536/statm: 85181 5722 348 364 0 21081 0
[pid=8538] ppid=8536 vsize=1560 CPUtime=0.21 cores=0-7
/proc/8538/stat : 8538 (march_hi) R 8536 8538 8538 0 -1 4202496 505 0 0 0 21 0 0 0 25 0 1 0 549601047 1597440 240 33554432000 4194304 4955518 140735653073808 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8538/statm: 390 240 106 186 0 203 0
[pid=8539] ppid=8512 vsize=350980 CPUtime=0 cores=0-7
/proc/8539/stat : 8539 (python) S 8512 8512 6630 0 -1 4202560 812 0 0 0 0 0 0 0 23 0 1 0 549601049 359403520 5732 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 14445102295039312132 0 0 17 0 0 0 0
/proc/8539/statm: 87745 5732 354 364 0 23642 0
[pid=8541] ppid=8539 vsize=1560 CPUtime=0.6 cores=0-7
/proc/8541/stat : 8541 (march_hi) R 8539 8541 8541 0 -1 4202496 510 0 0 0 60 0 0 0 25 0 1 0 549601049 1597440 240 33554432000 4194304 4955518 140735132206352 18446744073709551615 4296838 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/8541/statm: 390 240 106 186 0 203 0
[pid=8512/tid=8519] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8519/stat : 8519 (python) S 8510 8512 6630 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 9 0 549601021 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 2 0 0 0
[pid=8512/tid=8522] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8522/stat : 8522 (python) S 8510 8512 6630 0 -1 4202560 22 934 0 0 0 0 0 0 21 0 9 0 549601025 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8525] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8525/stat : 8525 (python) S 8510 8512 6630 0 -1 4202560 19 934 0 0 0 0 0 0 20 0 9 0 549601029 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8528] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8528/stat : 8528 (python) S 8510 8512 6630 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 9 0 549601033 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 4 0 0 0
[pid=8512/tid=8531] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8531/stat : 8531 (python) S 8510 8512 6630 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 9 0 549601037 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8534] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8534/stat : 8534 (python) S 8510 8512 6630 0 -1 4202560 28 934 0 0 0 0 0 0 22 0 9 0 549601041 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 6 0 0 0
[pid=8512/tid=8537] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8537/stat : 8537 (python) S 8510 8512 6630 0 -1 4202560 25 934 0 0 0 0 0 0 24 0 9 0 549601045 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8540] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8540/stat : 8540 (python) S 8510 8512 6630 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 9 0 549601049 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 6.02
Current children cumulated vsize (KiB) 4925580

[startup+3.10089 s]
/proc/loadavg: 8.69 9.22 9.41 10/329 8541
/proc/meminfo: memFree=12890200/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=361224 CPUtime=0.77 cores=0-7
/proc/8512/stat : 8512 (python) S 8510 8512 6630 0 -1 4202496 15704 934 1 0 67 10 0 0 25 0 9 0 549600960 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/8512/statm: 90306 7271 1889 364 0 26203 0
[pid=8518] ppid=8512 vsize=278584 CPUtime=0 cores=0-7
/proc/8518/stat : 8518 (python) S 8512 8512 6630 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 549601021 285270016 5559 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965263 0 0 16781312 514 10110862917596315904 0 0 17 4 0 0 0
/proc/8518/statm: 69646 5559 354 364 0 5564 0
[pid=8520] ppid=8518 vsize=1017800 CPUtime=1.46 cores=0-7
/proc/8520/stat : 8520 (gnovelty+2) R 8518 8520 8520 0 -1 4202496 436 0 0 0 146 0 0 0 23 0 1 0 549601021 1042227200 171 33554432000 134512640 135663704 4288802688 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8520/statm: 254450 171 113 282 0 254166 0
[pid=8521] ppid=8512 vsize=289276 CPUtime=0 cores=0-7
/proc/8521/stat : 8521 (python) S 8512 8512 6630 0 -1 4202560 748 0 0 0 0 0 0 0 15 0 1 0 549601025 296218624 5676 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 12943499264985628930 0 0 17 2 0 0 0
/proc/8521/statm: 72319 5676 348 364 0 8234 0
[pid=8523] ppid=8521 vsize=1560 CPUtime=2.44 cores=0-7
/proc/8523/stat : 8523 (march_hi) R 8521 8523 8523 0 -1 4202496 506 0 0 0 244 0 0 0 25 0 1 0 549601026 1597440 240 33554432000 4194304 4955518 140734909562464 18446744073709551615 4224658 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/8523/statm: 390 240 106 186 0 203 0
[pid=8524] ppid=8512 vsize=299532 CPUtime=0 cores=0-7
/proc/8524/stat : 8524 (python) S 8512 8512 6630 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 549601029 306720768 5680 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 6366968650414391554 0 0 17 0 0 0 0
/proc/8524/statm: 74883 5680 348 364 0 10795 0
[pid=8526] ppid=8524 vsize=1017800 CPUtime=2.38 cores=0-7
/proc/8526/stat : 8526 (gnovelty+2) R 8524 8526 8526 0 -1 4202496 435 0 0 0 238 0 0 0 25 0 1 0 549601030 1042227200 172 33554432000 134512640 135663704 4291936160 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8526/statm: 254450 172 113 282 0 254166 0
[pid=8527] ppid=8512 vsize=309788 CPUtime=0 cores=0-7
/proc/8527/stat : 8527 (python) S 8512 8512 6630 0 -1 4202560 755 0 0 0 0 0 0 0 15 0 1 0 549601033 317222912 5684 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 10301707074974744833 0 0 17 4 0 0 0
/proc/8527/statm: 77447 5684 348 364 0 13356 0
[pid=8529] ppid=8527 vsize=1560 CPUtime=2.36 cores=0-7
/proc/8529/stat : 8529 (march_hi) R 8527 8529 8529 0 -1 4202496 504 0 0 0 236 0 0 0 25 0 1 0 549601033 1597440 239 33554432000 4194304 4955518 140734502472512 18446744073709551615 4297288 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/8529/statm: 390 239 106 186 0 203 0
[pid=8530] ppid=8512 vsize=320044 CPUtime=0 cores=0-7
/proc/8530/stat : 8530 (python) S 8512 8512 6630 0 -1 4202560 779 0 0 0 0 0 0 0 15 0 1 0 549601037 327725056 5697 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 14853170642525585665 0 0 17 1 0 0 0
/proc/8530/statm: 80011 5697 348 364 0 15917 0
[pid=8532] ppid=8530 vsize=1560 CPUtime=2.32 cores=0-7
/proc/8532/stat : 8532 (march_hi) R 8530 8532 8532 0 -1 4202496 506 0 0 0 232 0 0 0 25 0 1 0 549601037 1597440 240 33554432000 4194304 4955518 140734864802688 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8532/statm: 390 240 106 186 0 203 0
[pid=8533] ppid=8512 vsize=330468 CPUtime=0 cores=0-7
/proc/8533/stat : 8533 (python) S 8512 8512 6630 0 -1 4202560 781 0 0 0 0 0 0 0 15 0 1 0 549601041 338399232 5718 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 17475092458399301890 0 0 17 6 0 0 0
/proc/8533/statm: 82617 5718 348 364 0 18520 0
[pid=8535] ppid=8533 vsize=1560 CPUtime=2.27 cores=0-7
/proc/8535/stat : 8535 (march_hi) R 8533 8535 8535 0 -1 4202496 506 0 0 0 227 0 0 0 25 0 1 0 549601041 1597440 240 33554432000 4194304 4955518 140737159099760 18446744073709551615 4208624 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/8535/statm: 390 240 106 186 0 203 0
[pid=8536] ppid=8512 vsize=340724 CPUtime=0 cores=0-7
/proc/8536/stat : 8536 (python) S 8512 8512 6630 0 -1 4202560 784 0 0 0 0 0 0 0 15 0 1 0 549601045 348901376 5722 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 7070880117685190914 0 0 17 0 0 0 0
/proc/8536/statm: 85181 5722 348 364 0 21081 0
[pid=8538] ppid=8536 vsize=1560 CPUtime=1.01 cores=0-7
/proc/8538/stat : 8538 (march_hi) R 8536 8538 8538 0 -1 4202496 505 0 0 0 101 0 0 0 25 0 1 0 549601047 1597440 240 33554432000 4194304 4955518 140735653073808 18446744073709551615 4224471 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8538/statm: 390 240 106 186 0 203 0
[pid=8539] ppid=8512 vsize=350980 CPUtime=0 cores=0-7
/proc/8539/stat : 8539 (python) S 8512 8512 6630 0 -1 4202560 818 0 0 0 0 0 0 0 15 0 1 0 549601049 359403520 5732 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 14445102295039312132 0 0 17 0 0 0 0
/proc/8539/statm: 87745 5732 354 364 0 23642 0
[pid=8541] ppid=8539 vsize=1560 CPUtime=2.2 cores=0-7
/proc/8541/stat : 8541 (march_hi) R 8539 8541 8541 0 -1 4202496 510 0 0 0 220 0 0 0 25 0 1 0 549601049 1597440 240 33554432000 4194304 4955518 140735132206352 18446744073709551615 4224512 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/8541/statm: 390 240 106 186 0 203 0
[pid=8512/tid=8519] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8519/stat : 8519 (python) S 8510 8512 6630 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 9 0 549601021 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 2 0 0 0
[pid=8512/tid=8522] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8522/stat : 8522 (python) S 8510 8512 6630 0 -1 4202560 22 934 0 0 0 0 0 0 21 0 9 0 549601025 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8525] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8525/stat : 8525 (python) S 8510 8512 6630 0 -1 4202560 19 934 0 0 0 0 0 0 20 0 9 0 549601029 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8528] ppid=8510 vsize=361224 CPUtime=0 cores=0-7
/proc/8512/task/8528/stat : 8528 (python) S 8510 8512 6630 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 9 0 549601033 369893376 7271 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 4 0 0 0
[pid=8512/tid=8531] ppid=8510 vsize=361224 CPUtime=0 cores=0-7

################
# More data... #
################

[pid=8520] ppid=8518 vsize=1017800 CPUtime=98.39 cores=0-7
/proc/8520/stat : 8520 (gnovelty+2) T 8518 8520 8520 0 -1 4202496 436 0 0 0 9839 0 0 0 25 0 1 0 549601021 1042227200 171 33554432000 134512640 135663704 4288802688 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8520/statm: 254450 171 113 282 0 254166 0
[pid=8518/tid=8550] ppid=8512 vsize=288828 CPUtime=0 cores=0-7
/proc/8518/task/8550/stat : 8550 (python) S 8512 8512 6630 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 549615852 295759872 5620 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 514 18446604446676877376 0 0 -1 4 0 0 0
[pid=8521] ppid=8512 vsize=289276 CPUtime=0.81 cores=0-7
/proc/8521/stat : 8521 (python) S 8512 8512 6630 0 -1 4202560 1154 0 0 0 42 39 0 0 15 0 2 0 549601025 296218624 5729 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/8521/statm: 72319 5729 400 364 0 8234 0
[pid=8523] ppid=8521 vsize=5108 CPUtime=315.73 cores=0-7
/proc/8523/stat : 8523 (march_hi) R 8521 8523 8523 0 -1 4202496 1274 0 0 0 31572 1 0 0 25 0 1 0 549601026 5230592 1007 33554432000 4194304 4955518 140734909562464 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/8523/statm: 1277 1007 106 186 0 1090 0
[pid=8521/tid=8555] ppid=8512 vsize=289276 CPUtime=0 cores=0-7
/proc/8521/task/8555/stat : 8555 (python) S 8512 8512 6630 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 549620832 296218624 5729 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 514 18446604451624195968 0 0 -1 2 0 0 0
[pid=8524] ppid=8512 vsize=299532 CPUtime=0.21 cores=0-7
/proc/8524/stat : 8524 (python) S 8512 8512 6630 0 -1 4202560 1091 0 0 0 12 9 0 0 19 0 2 0 549601029 306720768 5733 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/8524/statm: 74883 5733 400 364 0 10795 0
[pid=8526] ppid=8524 vsize=1017800 CPUtime=99.25 cores=0-7
/proc/8526/stat : 8526 (gnovelty+2) T 8524 8526 8526 0 -1 4202496 435 0 0 0 9925 0 0 0 25 0 1 0 549601030 1042227200 172 33554432000 134512640 135663704 4291936160 18446744073709551615 134564723 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/8526/statm: 254450 172 113 282 0 254166 0
[pid=8524/tid=8545] ppid=8512 vsize=299532 CPUtime=0 cores=0-7
/proc/8524/task/8545/stat : 8545 (python) S 8512 8512 6630 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 549611050 306720768 5733 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 514 18446604450100151360 0 0 -1 4 0 0 0
[pid=8527] ppid=8512 vsize=309788 CPUtime=1.21 cores=0-7
/proc/8527/stat : 8527 (python) R 8512 8512 6630 0 -1 4202560 1032 0 0 0 81 40 0 0 16 0 1 0 549601033 317222912 5708 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 0 0 0 17 4 0 0 0
/proc/8527/statm: 77447 5708 372 364 0 13356 0
[pid=8529] ppid=8527 vsize=0 CPUtime=318.22 cores=0-7
/proc/8529/stat : 8529 (march_hi) R 8527 8529 8529 0 -1 4202500 1290 0 0 0 31822 0 0 0 25 0 1 0 549601033 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 4 0 0 0
/proc/8529/statm: 0 0 0 0 0 0 0
[pid=8530] ppid=8512 vsize=320044 CPUtime=0.8 cores=0-7
/proc/8530/stat : 8530 (python) S 8512 8512 6630 0 -1 4202560 1025 0 0 0 44 36 0 0 15 0 1 0 549601037 327725056 5720 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 14853170642525585665 0 0 17 1 0 0 0
/proc/8530/statm: 80011 5720 371 364 0 15917 0
[pid=8532] ppid=8530 vsize=5108 CPUtime=295.38 cores=0-7
/proc/8532/stat : 8532 (march_hi) R 8530 8532 8532 0 -1 4202496 1163 0 0 0 29538 0 0 0 25 0 1 0 549601037 5230592 896 33554432000 4194304 4955518 140734864802688 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8532/statm: 1277 896 106 186 0 1090 0
[pid=8533] ppid=8512 vsize=330468 CPUtime=2.82 cores=0-7
/proc/8533/stat : 8533 (python) R 8512 8512 6630 0 -1 4202560 1038 0 0 0 222 60 0 0 18 0 1 0 549601041 338399232 5741 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 0 0 0 17 6 0 0 0
/proc/8533/statm: 82617 5741 371 364 0 18520 0
[pid=8535] ppid=8533 vsize=0 CPUtime=316.82 cores=0-7
/proc/8535/stat : 8535 (march_hi) R 8533 8535 8535 0 -1 4202500 1292 0 0 0 31681 1 0 0 25 0 1 0 549601041 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 6 0 0 0
/proc/8535/statm: 0 0 0 0 0 0 0
[pid=8536] ppid=8512 vsize=340724 CPUtime=0.79 cores=0-7
/proc/8536/stat : 8536 (python) S 8512 8512 6630 0 -1 4202560 1030 0 0 0 41 38 0 0 15 0 1 0 549601045 348901376 5745 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 7070880117685190914 0 0 17 7 0 0 0
/proc/8536/statm: 85181 5745 371 364 0 21081 0
[pid=8538] ppid=8536 vsize=3956 CPUtime=245.57 cores=0-7
/proc/8538/stat : 8538 (march_hi) R 8536 8538 8538 0 -1 4202496 986 0 0 0 24557 0 0 0 25 0 1 0 549601047 4050944 720 33554432000 4194304 4955518 140735653073808 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8538/statm: 989 720 106 186 0 802 0
[pid=8539] ppid=8512 vsize=350980 CPUtime=0.8 cores=0-7
/proc/8539/stat : 8539 (python) S 8512 8512 6630 0 -1 4202560 1080 0 0 0 44 36 0 0 15 0 1 0 549601049 359403520 5755 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 14445102295039312132 0 0 17 2 0 0 0
/proc/8539/statm: 87745 5755 377 364 0 23642 0
[pid=8541] ppid=8539 vsize=5108 CPUtime=318.02 cores=0-7
/proc/8541/stat : 8541 (march_hi) R 8539 8541 8541 0 -1 4202496 1291 0 0 0 31801 1 0 0 25 0 1 0 549601049 5230592 1020 33554432000 4194304 4955518 140735132206352 18446744073709551615 4224707 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/8541/statm: 1277 1020 106 186 0 1090 0
[pid=8546] ppid=8512 vsize=361236 CPUtime=0.46 cores=0-7
/proc/8546/stat : 8546 (python) S 8512 8512 6630 0 -1 4202560 1031 0 0 0 26 20 0 0 18 0 1 0 549611054 369905664 5753 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 6950918996753350914 0 0 17 1 0 0 0
/proc/8546/statm: 90309 5753 371 364 0 26203 0
[pid=8548] ppid=8546 vsize=181248 CPUtime=194.22 cores=0-7
/proc/8548/stat : 8548 (kcnfs-2006) R 8546 8548 8548 0 -1 4202496 629 0 0 0 19422 0 0 0 25 0 1 0 549611055 185597952 359 33554432000 134512640 135163436 4287878720 18446744073709551615 134612121 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8548/statm: 45312 359 76 159 0 45152 0
[pid=8551] ppid=8512 vsize=371224 CPUtime=0.4 cores=0-7
/proc/8551/stat : 8551 (python) S 8512 8512 6630 0 -1 4202560 1035 0 0 0 21 19 0 0 15 0 1 0 549615856 380133376 5714 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 4550751088310976771 0 0 17 1 0 0 0
/proc/8551/statm: 92806 5714 371 364 0 28697 0
[pid=8553] ppid=8551 vsize=2340 CPUtime=89.05 cores=0-7
/proc/8553/stat : 8553 (march_hi) R 8551 8553 8553 0 -1 4202496 662 0 0 0 8905 0 0 0 25 0 1 0 549615857 2396160 390 33554432000 4194304 4955518 140733931640592 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8553/statm: 585 390 106 186 0 398 0
[pid=8512/tid=8519] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8519/stat : 8519 (python) S 8510 8512 6630 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 11 0 549601021 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 2 0 0 0
[pid=8512/tid=8522] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8522/stat : 8522 (python) S 8510 8512 6630 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 11 0 549601025 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8525] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8525/stat : 8525 (python) S 8510 8512 6630 0 -1 4202560 19 934 0 0 0 0 0 0 20 0 11 0 549601029 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8528] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8528/stat : 8528 (python) S 8510 8512 6630 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 11 0 549601033 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 4 0 0 0
[pid=8512/tid=8531] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8531/stat : 8531 (python) S 8510 8512 6630 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 11 0 549601037 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8534] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8534/stat : 8534 (python) S 8510 8512 6630 0 -1 4202560 28 934 0 0 0 0 0 0 22 0 11 0 549601041 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 6 0 0 0
[pid=8512/tid=8537] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8537/stat : 8537 (python) S 8510 8512 6630 0 -1 4202560 25 934 0 0 0 0 0 0 24 0 11 0 549601045 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8540] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8540/stat : 8540 (python) S 8510 8512 6630 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 11 0 549601049 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8547] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8547/stat : 8547 (python) S 8510 8512 6630 0 -1 4202560 31 934 0 0 0 0 0 0 19 0 11 0 549611055 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8552] ppid=8510 vsize=382040 CPUtime=0 cores=0-7
/proc/8512/task/8552/stat : 8552 (python) S 8510 8512 6630 0 -1 4202560 30 934 0 0 0 0 0 0 19 0 11 0 549615856 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2300.23
Current children cumulated vsize (KiB) 5882608

[startup+320.602 s]
/proc/loadavg: 9.00 9.08 9.29 11/338 8558
/proc/meminfo: memFree=12818748/32951124 swapFree=62859076/67111528
[pid=8512] ppid=8510 vsize=382040 CPUtime=199.16 cores=0-7
/proc/8512/stat : 8512 (python) S 8510 8512 6630 0 -1 4202496 18184 4048 1 0 85 11 19796 24 17 0 11 0 549600960 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458775359 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/8512/statm: 95510 7360 1894 364 0 31401 0
[pid=8521] ppid=8512 vsize=289276 CPUtime=0.81 cores=0-7
/proc/8521/stat : 8521 (python) S 8512 8512 6630 0 -1 4202560 1154 0 0 0 42 39 0 0 15 0 2 0 549601025 296218624 5729 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/8521/statm: 72319 5729 400 364 0 8234 0
[pid=8523] ppid=8521 vsize=5108 CPUtime=315.83 cores=0-7
/proc/8523/stat : 8523 (march_hi) R 8521 8523 8523 0 -1 4202496 1275 0 0 0 31582 1 0 0 25 0 1 0 549601026 5230592 1008 33554432000 4194304 4955518 140734909562464 18446744073709551615 4235634 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/8523/statm: 1277 1008 106 186 0 1090 0
[pid=8521/tid=8555] ppid=8512 vsize=289276 CPUtime=0 cores=0-7
/proc/8521/task/8555/stat : 8555 (python) S 8512 8512 6630 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 549620832 296218624 5729 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 514 18446604451624195968 0 0 -1 2 0 0 0
[pid=8530] ppid=8512 vsize=320044 CPUtime=0.8 cores=0-7
/proc/8530/stat : 8530 (python) R 8512 8512 6630 0 -1 4202560 1034 0 0 0 44 36 0 0 15 0 1 0 549601037 327725056 5720 33554432000 4194304 5683644 140733475178832 18446744073709551615 4532142 0 0 16781312 514 0 0 0 17 1 0 0 0
/proc/8530/statm: 80011 5720 371 364 0 15917 0
[pid=8532] ppid=8530 vsize=5108 CPUtime=295.48 cores=0-7
/proc/8532/stat : 8532 (march_hi) R 8530 8532 8532 0 -1 4203524 1164 0 0 0 29548 0 0 0 25 0 1 0 549601037 5230592 897 33554432000 4194304 4955518 140734864802688 18446744073709551615 4224512 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8532/statm: -- couldn't open statm file --[pid=8533] ppid=8512 vsize=0 CPUtime=319.73 cores=0-7
/proc/8533/stat : 8533 (python) Z 8512 8512 6630 0 -1 4202588 1265 1292 0 0 230 61 31681 1 19 0 1 0 549601041 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 6 0 0 0
/proc/8533/statm: 0 0 0 0 0 0 0
[pid=8536] ppid=8512 vsize=340724 CPUtime=0.79 cores=0-7
/proc/8536/stat : 8536 (python) S 8512 8512 6630 0 -1 4202560 1030 0 0 0 41 38 0 0 15 0 1 0 549601045 348901376 5745 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 7070880117685190914 0 0 17 7 0 0 0
/proc/8536/statm: 85181 5745 371 364 0 21081 0
[pid=8538] ppid=8536 vsize=3956 CPUtime=245.63 cores=0-7
/proc/8538/stat : 8538 (march_hi) R 8536 8538 8538 0 -1 4202496 986 0 0 0 24563 0 0 0 25 0 1 0 549601047 4050944 720 33554432000 4194304 4955518 140735653073808 18446744073709551615 4235327 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8538/statm: 989 720 106 186 0 802 0
[pid=8546] ppid=8512 vsize=361236 CPUtime=0.46 cores=0-7
/proc/8546/stat : 8546 (python) S 8512 8512 6630 0 -1 4202560 1031 0 0 0 26 20 0 0 18 0 1 0 549611054 369905664 5753 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 6950918996753350914 0 0 17 1 0 0 0
/proc/8546/statm: 90309 5753 371 364 0 26203 0
[pid=8548] ppid=8546 vsize=181248 CPUtime=194.32 cores=0-7
/proc/8548/stat : 8548 (kcnfs-2006) R 8546 8548 8548 0 -1 4202496 629 0 0 0 19432 0 0 0 25 0 1 0 549611055 185597952 359 33554432000 134512640 135163436 4287878720 18446744073709551615 134684342 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8548/statm: 45312 359 76 159 0 45152 0
[pid=8551] ppid=8512 vsize=371224 CPUtime=0.4 cores=0-7
/proc/8551/stat : 8551 (python) S 8512 8512 6630 0 -1 4202560 1035 0 0 0 21 19 0 0 15 0 1 0 549615856 380133376 5714 33554432000 4194304 5683644 140733475178832 18446744073709551615 244446965318 0 0 16781312 514 4550751088310976771 0 0 17 1 0 0 0
/proc/8551/statm: 92806 5714 371 364 0 28697 0
[pid=8553] ppid=8551 vsize=2340 CPUtime=89.1 cores=0-7
/proc/8553/stat : 8553 (march_hi) R 8551 8553 8553 0 -1 4202496 662 0 0 0 8910 0 0 0 25 0 1 0 549615857 2396160 390 33554432000 4194304 4955518 140733931640592 18446744073709551615 4224679 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8553/statm: 585 390 106 186 0 398 0
[pid=8512/tid=8519] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8519/stat : 8519 (python) S 8510 8512 6630 0 -1 4202560 29 9012 0 0 0 0 83565 105 19 0 11 0 549601021 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 2 0 0 0
[pid=8512/tid=8522] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8522/stat : 8522 (python) S 8510 8512 6630 0 -1 4202560 26 9012 0 0 0 0 83565 105 18 0 11 0 549601025 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8525] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8525/stat : 8525 (python) S 8510 8512 6630 0 -1 4202560 19 9012 0 0 0 0 83565 105 20 0 11 0 549601029 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8528] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8528/stat : 8528 (python) S 8510 8512 6630 0 -1 4202560 25 9012 0 0 0 0 83565 105 22 0 11 0 549601033 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 4 0 0 0
[pid=8512/tid=8531] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8531/stat : 8531 (python) S 8510 8512 6630 0 -1 4202560 27 9012 0 0 0 0 83565 105 20 0 11 0 549601037 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8534] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8534/stat : 8534 (python) S 8510 8512 6630 0 -1 4202560 28 9012 0 0 0 0 83565 105 22 0 11 0 549601041 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 6 0 0 0
[pid=8512/tid=8537] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8537/stat : 8537 (python) S 8510 8512 6630 0 -1 4202560 25 9012 0 0 0 0 83565 105 24 0 11 0 549601045 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8540] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8540/stat : 8540 (python) S 8510 8512 6630 0 -1 4202560 27 9012 0 0 0 0 83565 105 25 0 11 0 549601049 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8547] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8547/stat : 8547 (python) S 8510 8512 6630 0 -1 4202560 31 9012 0 0 0 0 83565 105 19 0 11 0 549611055 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
[pid=8512/tid=8552] ppid=8510 vsize=382040 CPUtime=836.7 cores=0-7
/proc/8512/task/8552/stat : 8552 (python) S 8510 8512 6630 0 -1 4202560 30 9012 0 0 0 0 83565 105 19 0 11 0 549615856 391208960 7360 33554432000 4194304 5683644 140733475178832 18446744073709551615 244458768817 0 0 16781312 2 18446604453795933120 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1662.51
Current children cumulated vsize (KiB) 2262304

Child status: 0

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 8512 and gives
#  childrusage.ru_utime.tv_sec=2297
#  childrusage.ru_utime.tv_usec=795681
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=380486
# CPU time returned by wait4() is 2301.18
# while last known CPU time is 1662.51
#
# Using CPU time of the last sample as value...


# WARNING:
# The CPU time of some children was not reported to their father
# (probably because of a missing or aborted wait()).
# This 'lost CPU time' is added to the watched process CPU time.
#  lost CPU time (s): 637.72
#  lost CPU user time (s): 636.97
#  lost CPU system time (s): 0.75

Real time (s): 320.631
CPU time (s): 2300.23
CPU user time (s): 2296.98
CPU system time (s): 3.25
CPU usage (%): 717.407
Max. virtual memory (cumulated for all children) (KiB): 5892824

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2297.8
system time used= 3.38049
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 44008
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6966
involuntary context switches= 15831

runsolver used 1.67874 second user time and 3.04854 second system time

The end

Launcher Data 

Begin job on node105 at 2011-05-03 01:46:45
IDJOB=3322294
IDBENCH=83072
IDSOLVER=1712
FILE ID=node105/3322294-1304380005
RUNJOBID= node105-1304366469-6644
PBS_JOBID= 13181765
Free space on /tmp= 73624 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v100-c2130-S815408325-054.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322294-1304380005/watcher-3322294-1304380005 -o /tmp/evaluation-result-3322294-1304380005/solver-3322294-1304380005 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322294-1304380005.cnf -seed 660750185 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 9830e1bf79d13e57ceeaa969f4476a02
RANDOM SEED=660750185

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      12947984 kB
Buffers:         24312 kB
Cached:         244224 kB
SwapCached:      26984 kB
Active:        8227040 kB
Inactive:     11559052 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      12947984 kB
SwapTotal:    67111528 kB
SwapFree:     62859076 kB
Dirty:           32380 kB
Writeback:           0 kB
AnonPages:    19493612 kB
Mapped:          21836 kB
Slab:            83828 kB
PageTables:      65240 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 60711612 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73592 MiB
End job on node105 at 2011-05-03 01:52:06