Trace number 3378427

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 4776.01 1197.56

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S1061710238-001.UNKNOWN.cnf
MD5SUM23ebfbdf095807765a162fcb1d02f57a
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.30/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.48	c INFO: solving HOME/instance-3378427-1305465347.cnf
0.30/0.48	c INFO: computing task features
0.30/0.49	c INFO: computed variable-clause graph statistics
0.30/0.49	c INFO: computed clause balance statistics
0.39/0.56	c INFO: computed variable balance statistics
0.39/0.59	c INFO: computed small-clause counts
0.39/0.59	c INFO: computed Horn-clause counts
0.39/0.59	c INFO: computed variable graph statistics
0.39/0.60	c INFO: collected features for HOME/instance-3378427-1305465347.cnf in 0.12 s
0.49/0.60	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.63)
0.49/0.61	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.49/0.62	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.29)
0.49/0.64	c INFO: running march_hi@0 for 679 with 1199 remaining (b = 0.36)
400.23/100.83	c INFO: running march_hi@0 for 679 with 1099 remaining (b = 0.33)
400.23/100.84	c INFO: running march_hi@0 for 679 with 1099 remaining (b = 0.29)
1581.27/396.87	c INFO: running kcnfs-2006@0 for 582 with 803 remaining (b = 0.24)
2714.42/680.90	c INFO: running march_hi@0 for 388 with 519 remaining (b = 0.17)
3133.31/785.81	c INFO: running march_hi@0 for 291 with 414 remaining (b = 0.12)
3141.31/787.82	c INFO: running march_hi@99 for 291 with 412 remaining (b = 0.13)
3924.53/984.16	c INFO: running march_hi@392 for 97 with 216 remaining (b = 0.06)
4292.33/1076.34	c INFO: running march_hi@100 for 124 with 124 remaining (b = 0.05)
4298.72/1077.90	c INFO: running march_hi@683 for 122 with 122 remaining (b = 0.05)
4305.89/1079.78	c INFO: running kcnfs-2006@679 for 120 with 120 remaining (b = 0.04)
4321.47/1083.61	c INFO: running march_hi@680 for 116 with 116 remaining (b = 0.04)
4775.69/1197.54	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378427-1305465347/watcher-3378427-1305465347 -o /tmp/evaluation-result-3378427-1305465347/solver-3378427-1305465347 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378427-1305465347.cnf -seed 873408899 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.84 0.67 0.48 5/319 18490
/proc/meminfo: memFree=15779004/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=15912 CPUtime=0 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 958 0 0 0 0 0 0 0 25 0 1 0 658137491 16293888 834 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370955264 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/18490/statm: 3978 834 357 364 0 465 0

[startup+0.108861 s]
/proc/loadavg: 2.84 0.67 0.48 5/319 18490
/proc/meminfo: memFree=15779004/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=104620 CPUtime=0.07 cores=1,3,5,7
/proc/18490/stat : 18490 (python) R 18488 18490 18387 0 -1 4202496 3154 0 0 0 5 2 0 0 17 0 1 0 658137491 107130880 2829 33554432000 4194304 5683644 140733493568864 18446744073709551615 4874594 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/18490/statm: 26155 2829 847 364 0 1903 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104620

[startup+0.200835 s]
/proc/loadavg: 2.84 0.67 0.48 5/319 18490
/proc/meminfo: memFree=15779004/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=128872 CPUtime=0.14 cores=1,3,5,7
/proc/18490/stat : 18490 (python) R 18488 18490 18387 0 -1 4202496 5553 935 1 0 11 3 0 0 18 0 1 0 658137491 131964928 4019 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370957056 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/18490/statm: 32218 4019 1011 364 0 2997 0
Current children cumulated CPU time (s) 0.14
Current children cumulated vsize (KiB) 128872

[startup+0.300809 s]
/proc/loadavg: 2.84 0.67 0.48 5/319 18490
/proc/meminfo: memFree=15779004/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=163652 CPUtime=0.22 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 6664 935 1 0 16 6 0 0 17 0 1 0 658137491 167579648 4755 33554432000 4194304 5683644 140733493568864 18446744073709551615 218366033543 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/18490/statm: 40913 4755 1311 364 0 3454 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 163652

[startup+0.700709 s]
/proc/loadavg: 2.84 0.67 0.48 5/319 18490
/proc/meminfo: memFree=15779004/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=320100 CPUtime=0.52 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 12230 935 1 0 42 10 0 0 22 0 5 0 658137491 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/18490/statm: 80025 7230 1873 364 0 15934 0
Current children cumulated CPU time (s) 0.52
Current children cumulated vsize (KiB) 320100

[startup+1.50057 s]
/proc/loadavg: 2.84 0.67 0.48 9/332 18507
/proc/meminfo: memFree=15739640/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=320100 CPUtime=0.52 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 12230 935 1 0 42 10 0 0 22 0 5 0 658137491 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/18490/statm: 80025 7230 1873 364 0 15934 0
[pid=18496] ppid=18490 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/18496/stat : 18496 (python) S 18490 18490 18387 0 -1 4202560 731 0 0 0 0 0 0 0 18 0 1 0 658137551 285237248 5557 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977295 0 0 16781312 514 4485031903929303039 0 0 17 5 0 0 0
/proc/18496/statm: 69638 5557 354 364 0 5556 0
[pid=18498] ppid=18496 vsize=1017800 CPUtime=0.87 cores=1,3,5,7
/proc/18498/stat : 18498 (gnovelty+2) R 18496 18498 18498 0 -1 4202496 446 0 0 0 87 0 0 0 21 0 1 0 658137552 1042227200 181 33554432000 134512640 135663704 4292118656 18446744073709551615 134524507 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18498/statm: 254450 181 113 282 0 254166 0
[pid=18499] ppid=18490 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/18499/stat : 18499 (python) S 18490 18490 18387 0 -1 4202560 735 0 0 0 0 0 0 0 20 0 1 0 658137552 296288256 5676 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 12157022171050278911 0 0 17 5 0 0 0
/proc/18499/statm: 72336 5676 348 364 0 8251 0
[pid=18501] ppid=18499 vsize=1017800 CPUtime=0.87 cores=1,3,5,7
/proc/18501/stat : 18501 (gnovelty+2) R 18499 18501 18501 0 -1 4202496 445 0 0 0 87 0 0 0 25 0 1 0 658137553 1042227200 181 33554432000 134512640 135663704 4292321072 18446744073709551615 134544186 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18501/statm: 254450 181 113 282 0 254166 0
[pid=18502] ppid=18490 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/18502/stat : 18502 (python) S 18490 18490 18387 0 -1 4202560 756 0 0 0 0 0 0 0 15 0 1 0 658137553 306790400 5697 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 5501094172569862407 0 0 17 5 0 0 0
/proc/18502/statm: 74900 5697 348 364 0 10812 0
[pid=18504] ppid=18502 vsize=2092 CPUtime=0.86 cores=1,3,5,7
/proc/18504/stat : 18504 (march_hi) R 18502 18504 18504 0 -1 4202496 607 0 0 0 86 0 0 0 23 0 1 0 658137554 2142208 344 33554432000 4194304 4955518 140735053166416 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18504/statm: 523 344 106 186 0 336 0
[pid=18505] ppid=18490 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/18505/stat : 18505 (python) S 18490 18490 18387 0 -1 4202560 758 0 0 0 0 0 0 0 15 0 1 0 658137554 317292544 5701 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 5501098570616373510 0 0 17 3 0 0 0
/proc/18505/statm: 77464 5701 348 364 0 13373 0
[pid=18507] ppid=18505 vsize=2092 CPUtime=0.85 cores=1,3,5,7
/proc/18507/stat : 18507 (march_hi) R 18505 18507 18507 0 -1 4202496 605 0 0 0 85 0 0 0 25 0 1 0 658137555 2142208 343 33554432000 4194304 4955518 140735882680704 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18507/statm: 523 343 106 186 0 336 0
[pid=18490/tid=18497] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18497/stat : 18497 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 19 0 5 0 658137552 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18500] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18500/stat : 18500 (python) S 18488 18490 18387 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 5 0 658137552 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18503] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18503/stat : 18503 (python) S 18488 18490 18387 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 5 0 658137553 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 5 0 0 0
[pid=18490/tid=18506] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18506/stat : 18506 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 5 0 658137555 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3.97
Current children cumulated vsize (KiB) 3537236

[startup+3.10128 s]
/proc/loadavg: 2.84 0.67 0.48 9/332 18507
/proc/meminfo: memFree=15738468/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=320100 CPUtime=0.52 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 12230 935 1 0 42 10 0 0 22 0 5 0 658137491 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/18490/statm: 80025 7230 1873 364 0 15934 0
[pid=18496] ppid=18490 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/18496/stat : 18496 (python) S 18490 18490 18387 0 -1 4202560 731 0 0 0 0 0 0 0 18 0 1 0 658137551 285237248 5557 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977295 0 0 16781312 514 4485031903929270535 0 0 17 5 0 0 0
/proc/18496/statm: 69638 5557 354 364 0 5556 0
[pid=18498] ppid=18496 vsize=1017800 CPUtime=2.46 cores=1,3,5,7
/proc/18498/stat : 18498 (gnovelty+2) R 18496 18498 18498 0 -1 4202496 446 0 0 0 246 0 0 0 25 0 1 0 658137552 1042227200 181 33554432000 134512640 135663704 4292118656 18446744073709551615 134525218 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18498/statm: 254450 181 113 282 0 254166 0
[pid=18499] ppid=18490 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/18499/stat : 18499 (python) S 18490 18490 18387 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 658137552 296288256 5676 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 12157022171050246407 0 0 17 5 0 0 0
/proc/18499/statm: 72336 5676 348 364 0 8251 0
[pid=18501] ppid=18499 vsize=1017800 CPUtime=2.47 cores=1,3,5,7
/proc/18501/stat : 18501 (gnovelty+2) R 18499 18501 18501 0 -1 4202496 445 0 0 0 247 0 0 0 25 0 1 0 658137553 1042227200 181 33554432000 134512640 135663704 4292321072 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18501/statm: 254450 181 113 282 0 254166 0
[pid=18502] ppid=18490 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/18502/stat : 18502 (python) S 18490 18490 18387 0 -1 4202560 760 0 0 0 0 0 0 0 15 0 1 0 658137553 306790400 5697 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 5501094172569862407 0 0 17 5 0 0 0
/proc/18502/statm: 74900 5697 348 364 0 10812 0
[pid=18504] ppid=18502 vsize=2472 CPUtime=2.46 cores=1,3,5,7
/proc/18504/stat : 18504 (march_hi) R 18502 18504 18504 0 -1 4202496 702 0 0 0 246 0 0 0 25 0 1 0 658137554 2531328 439 33554432000 4194304 4955518 140735053166416 18446744073709551615 4297231 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18504/statm: 618 439 106 186 0 431 0
[pid=18505] ppid=18490 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/18505/stat : 18505 (python) S 18490 18490 18387 0 -1 4202560 762 0 0 0 0 0 0 0 15 0 1 0 658137554 317292544 5701 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 5501098570616373510 0 0 17 3 0 0 0
/proc/18505/statm: 77464 5701 348 364 0 13373 0
[pid=18507] ppid=18505 vsize=2472 CPUtime=2.45 cores=1,3,5,7
/proc/18507/stat : 18507 (march_hi) R 18505 18507 18507 0 -1 4202496 701 0 0 0 245 0 0 0 25 0 1 0 658137555 2531328 439 33554432000 4194304 4955518 140735882680704 18446744073709551615 4224768 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18507/statm: 618 439 106 186 0 431 0
[pid=18490/tid=18497] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18497/stat : 18497 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 19 0 5 0 658137552 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18500] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18500/stat : 18500 (python) S 18488 18490 18387 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 5 0 658137552 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18503] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18503/stat : 18503 (python) S 18488 18490 18387 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 5 0 658137553 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 5 0 0 0
[pid=18490/tid=18506] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18506/stat : 18506 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 5 0 658137555 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 10.36
Current children cumulated vsize (KiB) 3537996

[startup+6.30071 s]
/proc/loadavg: 3.26 0.79 0.52 9/332 18507
/proc/meminfo: memFree=15735044/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=320100 CPUtime=0.52 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 12230 935 1 0 42 10 0 0 22 0 5 0 658137491 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/18490/statm: 80025 7230 1873 364 0 15934 0
[pid=18496] ppid=18490 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/18496/stat : 18496 (python) S 18490 18490 18387 0 -1 4202560 890 0 0 0 0 0 0 0 18 0 1 0 658137551 285237248 5580 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977295 0 0 16781312 514 4485031903929270535 0 0 17 5 0 0 0
/proc/18496/statm: 69638 5580 377 364 0 5556 0
[pid=18498] ppid=18496 vsize=1017800 CPUtime=5.63 cores=1,3,5,7
/proc/18498/stat : 18498 (gnovelty+2) R 18496 18498 18498 0 -1 4202496 446 0 0 0 563 0 0 0 25 0 1 0 658137552 1042227200 181 33554432000 134512640 135663704 4292118656 18446744073709551615 134525323 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18498/statm: 254450 181 113 282 0 254166 0
[pid=18499] ppid=18490 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/18499/stat : 18499 (python) S 18490 18490 18387 0 -1 4202560 906 0 0 0 0 0 0 0 18 0 1 0 658137552 296288256 5699 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 12157022171050246407 0 0 17 5 0 0 0
/proc/18499/statm: 72336 5699 371 364 0 8251 0
[pid=18501] ppid=18499 vsize=1017800 CPUtime=5.67 cores=1,3,5,7
/proc/18501/stat : 18501 (gnovelty+2) R 18499 18501 18501 0 -1 4202496 445 0 0 0 567 0 0 0 25 0 1 0 658137553 1042227200 181 33554432000 134512640 135663704 4292321072 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18501/statm: 254450 181 113 282 0 254166 0
[pid=18502] ppid=18490 vsize=299600 CPUtime=0.01 cores=1,3,5,7
/proc/18502/stat : 18502 (python) S 18490 18490 18387 0 -1 4202560 909 0 0 0 0 1 0 0 15 0 1 0 658137553 306790400 5720 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 5501094172569862407 0 0 17 5 0 0 0
/proc/18502/statm: 74900 5720 371 364 0 10812 0
[pid=18504] ppid=18502 vsize=3324 CPUtime=5.62 cores=1,3,5,7
/proc/18504/stat : 18504 (march_hi) R 18502 18504 18504 0 -1 4202496 872 0 0 0 562 0 0 0 25 0 1 0 658137554 3403776 609 33554432000 4194304 4955518 140735053166416 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18504/statm: 831 609 106 186 0 644 0
[pid=18505] ppid=18490 vsize=309856 CPUtime=0.01 cores=1,3,5,7
/proc/18505/stat : 18505 (python) S 18490 18490 18387 0 -1 4202560 910 0 0 0 0 1 0 0 15 0 1 0 658137554 317292544 5724 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 5501098570616373510 0 0 17 3 0 0 0
/proc/18505/statm: 77464 5724 371 364 0 13373 0
[pid=18507] ppid=18505 vsize=3324 CPUtime=5.64 cores=1,3,5,7
/proc/18507/stat : 18507 (march_hi) R 18505 18507 18507 0 -1 4202496 872 0 0 0 564 0 0 0 25 0 1 0 658137555 3403776 610 33554432000 4194304 4955518 140735882680704 18446744073709551615 4208638 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18507/statm: 831 610 106 186 0 644 0
[pid=18490/tid=18497] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18497/stat : 18497 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 19 0 5 0 658137552 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18500] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18500/stat : 18500 (python) S 18488 18490 18387 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 5 0 658137552 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18503] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18503/stat : 18503 (python) S 18488 18490 18387 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 5 0 658137553 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 5 0 0 0
[pid=18490/tid=18506] ppid=18488 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18506/stat : 18506 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 5 0 658137555 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 23.1
Current children cumulated vsize (KiB) 3539700

[startup+12.7006 s]
/proc/loadavg: 3.64 0.91 0.56 9/332 18507
/proc/meminfo: memFree=15731332/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=320100 CPUtime=0.52 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 12230 935 1 0 42 10 0 0 22 0 5 0 658137491 327782400 7230 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/18490/statm: 80025 7230 1873 364 0 15934 0
[pid=18496] ppid=18490 vsize=278552 CPUtime=0.02 cores=1,3,5,7
/proc/18496/stat : 18496 (python) S 18490 18490 18387 0 -1 4202560 897 0 0 0 1 1 0 0 18 0 1 0 658137551 285237248 5580 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977295 0 0 16781312 514 678178772012204295 0 0 17 5 0 0 0
/proc/18496/statm: 69638 5580 377 364 0 5556 0
[pid=18498] ppid=18496 vsize=1017800 CPUtime=11.97 cores=1,3,5,7
/proc/18498/stat : 18498 (gnovelty+2) R 18496 18498 18498 0 -1 4202496 446 0 0 0 1197 0 0 0 25 0 1 0 658137552 1042227200 181 33554432000 134512640 135663704 4292118656 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18498/statm: 254450 181 113 282 0 254166 0
[pid=18499] ppid=18490 vsize=289344 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

/proc/18490/task/18538/stat : 18538 (python) S 18488 18490 18387 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 9 0 658177179 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18556] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18556/stat : 18556 (python) S 18488 18490 18387 0 -1 4202560 26 935 0 0 0 0 0 0 18 0 9 0 658205581 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4774.5
Current children cumulated vsize (KiB) 5842844

[startup+1197.4 s]
/proc/loadavg: 8.04 7.97 6.00 9/360 18717
/proc/meminfo: memFree=14909684/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=361416 CPUtime=0.61 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 16249 935 1 0 50 11 0 0 18 0 9 0 658137491 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/18490/statm: 90354 7338 1882 364 0 26251 0
[pid=18496] ppid=18490 vsize=288796 CPUtime=0.22 cores=1,3,5,7
/proc/18496/stat : 18496 (python) S 18490 18490 18387 0 -1 4202560 1174 0 0 0 11 11 0 0 19 0 2 0 658137551 295727104 5624 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18496/statm: 72199 5624 416 364 0 8117 0
[pid=18498] ppid=18496 vsize=1017800 CPUtime=99.28 cores=1,3,5,7
/proc/18498/stat : 18498 (gnovelty+2) T 18496 18498 18498 0 -1 4202496 446 0 0 0 9928 0 0 0 25 0 1 0 658137552 1042227200 181 33554432000 134512640 135663704 4292118656 18446744073709551615 134525278 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/18498/statm: 254450 181 113 282 0 254166 0
[pid=18496/tid=18518] ppid=18490 vsize=288796 CPUtime=0 cores=1,3,5,7
/proc/18496/task/18518/stat : 18518 (python) S 18490 18490 18387 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658147573 295727104 5624 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604471136815296 0 0 -1 5 0 0 0
[pid=18499] ppid=18490 vsize=289344 CPUtime=0.22 cores=1,3,5,7
/proc/18499/stat : 18499 (python) S 18490 18490 18387 0 -1 4202560 1094 0 0 0 13 9 0 0 19 0 2 0 658137552 296288256 5729 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/18499/statm: 72336 5729 400 364 0 8251 0
[pid=18501] ppid=18499 vsize=1017800 CPUtime=100.2 cores=1,3,5,7
/proc/18501/stat : 18501 (gnovelty+2) T 18499 18501 18501 0 -1 4202496 445 0 0 0 10020 0 0 0 25 0 1 0 658137553 1042227200 181 33554432000 134512640 135663704 4292321072 18446744073709551615 134525684 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/18501/statm: 254450 181 113 282 0 254166 0
[pid=18499/tid=18522] ppid=18490 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/18499/task/18522/stat : 18522 (python) S 18490 18490 18387 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658147574 296288256 5729 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604455743220544 0 0 -1 7 0 0 0
[pid=18502] ppid=18490 vsize=299600 CPUtime=1.96 cores=1,3,5,7
/proc/18502/stat : 18502 (python) S 18490 18490 18387 0 -1 4202560 1150 0 0 0 103 93 0 0 15 0 2 0 658137553 306790400 5750 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18502/statm: 74900 5750 400 364 0 10812 0
[pid=18504] ppid=18502 vsize=141204 CPUtime=794.1 cores=1,3,5,7
/proc/18504/stat : 18504 (march_hi) R 18502 18504 18504 0 -1 4202496 34420 0 0 0 79403 7 0 0 25 0 1 0 658137554 144592896 34157 33554432000 4194304 4955518 140735053166416 18446744073709551615 4224471 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/18504/statm: 35301 34157 106 186 0 35114 0
[pid=18502/tid=18536] ppid=18490 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/18502/task/18536/stat : 18536 (python) S 18490 18490 18387 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 658177178 306790400 5750 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604455743219648 0 0 -1 5 0 0 0
[pid=18505] ppid=18490 vsize=309856 CPUtime=2.36 cores=1,3,5,7
/proc/18505/stat : 18505 (python) S 18490 18490 18387 0 -1 4202560 1150 0 0 0 128 108 0 0 15 0 2 0 658137554 317292544 5754 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18505/statm: 77464 5754 400 364 0 13373 0
[pid=18507] ppid=18505 vsize=214316 CPUtime=971.96 cores=1,3,5,7
/proc/18507/stat : 18507 (march_hi) T 18505 18507 18507 0 -1 4202496 41573 0 0 0 97189 7 0 0 25 0 1 0 658137555 219459584 41311 33554432000 4194304 4955518 140735882680704 18446744073709551615 4208620 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/18507/statm: 53579 41311 106 186 0 53392 0
[pid=18505/tid=18554] ppid=18490 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/18505/task/18554/stat : 18554 (python) S 18490 18490 18387 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658205580 317292544 5754 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604455743217856 0 0 -1 3 0 0 0
[pid=18519] ppid=18490 vsize=320112 CPUtime=1.96 cores=1,3,5,7
/proc/18519/stat : 18519 (python) S 18490 18490 18387 0 -1 4202560 1155 0 0 0 103 93 0 0 15 0 2 0 658147574 327794688 5758 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18519/statm: 80028 5758 400 364 0 15934 0
[pid=18521] ppid=18519 vsize=141204 CPUtime=804.24 cores=1,3,5,7
/proc/18521/stat : 18521 (march_hi) R 18519 18521 18521 0 -1 4202496 35052 0 0 0 80418 6 0 0 25 0 1 0 658147574 144592896 34788 33554432000 4194304 4955518 140736904838464 18446744073709551615 4237772 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18521/statm: 35301 34788 106 186 0 35114 0
[pid=18519/tid=18563] ppid=18490 vsize=320112 CPUtime=0 cores=1,3,5,7
/proc/18519/task/18563/stat : 18563 (python) S 18490 18490 18387 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 658216071 327794688 5758 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604470096154944 0 0 -1 5 0 0 0
[pid=18523] ppid=18490 vsize=330512 CPUtime=1.88 cores=1,3,5,7
/proc/18523/stat : 18523 (python) S 18490 18490 18387 0 -1 4202560 1149 0 0 0 102 86 0 0 15 0 2 0 658147575 338444288 5781 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18523/statm: 82628 5781 400 364 0 18531 0
[pid=18525] ppid=18523 vsize=141204 CPUtime=779.65 cores=1,3,5,7
/proc/18525/stat : 18525 (march_hi) T 18523 18525 18525 0 -1 4202496 33834 0 0 0 77958 7 0 0 25 0 1 0 658147575 144592896 33572 33554432000 4194304 4955518 140735660801296 18446744073709551615 4208624 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/18525/statm: 35301 33572 106 186 0 35114 0
[pid=18523/tid=18564] ppid=18490 vsize=330512 CPUtime=0 cores=1,3,5,7
/proc/18523/task/18564/stat : 18564 (python) S 18490 18490 18387 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 658216272 338444288 5781 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604470096157632 0 0 -1 3 0 0 0
[pid=18537] ppid=18490 vsize=340324 CPUtime=1.64 cores=1,3,5,7
/proc/18537/stat : 18537 (python) S 18490 18490 18387 0 -1 4202560 1155 0 0 0 80 84 0 0 18 0 2 0 658177178 348491776 5706 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18537/statm: 85081 5706 400 364 0 20981 0
[pid=18539] ppid=18537 vsize=181380 CPUtime=700.88 cores=1,3,5,7
/proc/18539/stat : 18539 (kcnfs-2006) R 18537 18539 18539 0 -1 4202496 680 0 0 0 70088 0 0 0 25 0 1 0 658177179 185733120 420 33554432000 134512640 135163436 4287742640 18446744073709551615 134572059 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18539/statm: 45345 420 76 159 0 45185 0
[pid=18537/tid=18702] ppid=18490 vsize=340324 CPUtime=0 cores=1,3,5,7
/proc/18537/task/18702/stat : 18702 (python) S 18490 18490 18387 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658235906 348491776 5706 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604463156196608 0 0 -1 1 0 0 0
[pid=18555] ppid=18490 vsize=350972 CPUtime=1.25 cores=1,3,5,7
/proc/18555/stat : 18555 (python) S 18490 18490 18387 0 -1 4202560 1205 0 0 0 66 59 0 0 15 0 2 0 658205581 359395328 5799 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18555/statm: 87743 5799 406 364 0 23640 0
[pid=18557] ppid=18555 vsize=97004 CPUtime=512.89 cores=1,3,5,7
/proc/18557/stat : 18557 (march_hi) R 18555 18557 18557 0 -1 4202496 23155 0 0 0 51285 4 0 0 25 0 1 0 658205581 99332096 22882 33554432000 4194304 4955518 140734146644192 18446744073709551615 4297490 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18557/statm: 24251 22882 106 186 0 24064 0
[pid=18555/tid=18713] ppid=18490 vsize=350972 CPUtime=0 cores=1,3,5,7
/proc/18555/task/18713/stat : 18713 (python) S 18490 18490 18387 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658245125 359395328 5799 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604470096155840 0 0 -1 3 0 0 0
[pid=18490/tid=18497] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18497/stat : 18497 (python) S 18488 18490 18387 0 -1 4202560 27 935 0 0 0 0 0 0 19 0 9 0 658137552 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18500] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18500/stat : 18500 (python) S 18488 18490 18387 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 9 0 658137552 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18503] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18503/stat : 18503 (python) S 18488 18490 18387 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 658137553 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 5 0 0 0
[pid=18490/tid=18506] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18506/stat : 18506 (python) S 18488 18490 18387 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 9 0 658137555 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18520] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18520/stat : 18520 (python) S 18488 18490 18387 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 9 0 658147574 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18524] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18524/stat : 18524 (python) S 18488 18490 18387 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 9 0 658147575 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18538] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18538/stat : 18538 (python) S 18488 18490 18387 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 9 0 658177179 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18556] ppid=18488 vsize=361416 CPUtime=0 cores=1,3,5,7
/proc/18490/task/18556/stat : 18556 (python) S 18488 18490 18387 0 -1 4202560 26 935 0 0 0 0 0 0 18 0 9 0 658205581 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4775.3
Current children cumulated vsize (KiB) 5842844

[startup+1197.5 s]
/proc/loadavg: 8.04 7.97 6.00 9/360 18717
/proc/meminfo: memFree=14909684/32951124 swapFree=52307768/67111528
[pid=18490] ppid=18488 vsize=361416 CPUtime=100.11 cores=1,3,5,7
/proc/18490/stat : 18490 (python) S 18488 18490 18387 0 -1 4202496 16262 2574 1 0 50 11 9939 11 18 0 9 0 658137491 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382787391 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/18490/statm: 90354 7338 1882 364 0 26251 0
[pid=18499] ppid=18490 vsize=0 CPUtime=100.42 cores=1,3,5,7
/proc/18499/stat : 18499 (python) R 18490 18490 18387 0 -1 4202564 1120 445 0 0 13 9 10020 0 18 0 1 0 658137552 0 0 33554432000 0 0 0 0 0 0 0 16781824 2 18446744073709551615 0 0 17 7 0 0 0
/proc/18499/statm: 0 0 0 0 0 0 0
[pid=18502] ppid=18490 vsize=299600 CPUtime=1.96 cores=1,3,5,7
/proc/18502/stat : 18502 (python) S 18490 18490 18387 0 -1 4202560 1150 0 0 0 103 93 0 0 15 0 2 0 658137553 306790400 5750 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18502/statm: 74900 5750 400 364 0 10812 0
[pid=18504] ppid=18502 vsize=141204 CPUtime=794.19 cores=1,3,5,7
/proc/18504/stat : 18504 (march_hi) R 18502 18504 18504 0 -1 4202496 34424 0 0 0 79412 7 0 0 25 0 1 0 658137554 144592896 34161 33554432000 4194304 4955518 140735053166416 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/18504/statm: 35301 34161 106 186 0 35114 0
[pid=18502/tid=18536] ppid=18490 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/18502/task/18536/stat : 18536 (python) S 18490 18490 18387 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 658177178 306790400 5750 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604455743219648 0 0 -1 5 0 0 0
[pid=18505] ppid=18490 vsize=309856 CPUtime=2.36 cores=1,3,5,7
/proc/18505/stat : 18505 (python) S 18490 18490 18387 0 -1 4202560 1150 0 0 0 128 108 0 0 15 0 2 0 658137554 317292544 5754 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18505/statm: 77464 5754 400 364 0 13373 0
[pid=18507] ppid=18505 vsize=214316 CPUtime=971.96 cores=1,3,5,7
/proc/18507/stat : 18507 (march_hi) T 18505 18507 18507 0 -1 4202496 41573 0 0 0 97189 7 0 0 25 0 1 0 658137555 219459584 41311 33554432000 4194304 4955518 140735882680704 18446744073709551615 4208620 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/18507/statm: 53579 41311 106 186 0 53392 0
[pid=18505/tid=18554] ppid=18490 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/18505/task/18554/stat : 18554 (python) S 18490 18490 18387 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658205580 317292544 5754 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604455743217856 0 0 -1 3 0 0 0
[pid=18519] ppid=18490 vsize=320112 CPUtime=1.96 cores=1,3,5,7
/proc/18519/stat : 18519 (python) S 18490 18490 18387 0 -1 4202560 1155 0 0 0 103 93 0 0 15 0 2 0 658147574 327794688 5758 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/18519/statm: 80028 5758 400 364 0 15934 0
[pid=18521] ppid=18519 vsize=141204 CPUtime=804.34 cores=1,3,5,7
/proc/18521/stat : 18521 (march_hi) R 18519 18521 18521 0 -1 4202496 35056 0 0 0 80428 6 0 0 25 0 1 0 658147574 144592896 34792 33554432000 4194304 4955518 140736904838464 18446744073709551615 4224549 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/18521/statm: 35301 34792 106 186 0 35114 0
[pid=18519/tid=18563] ppid=18490 vsize=320112 CPUtime=0 cores=1,3,5,7
/proc/18519/task/18563/stat : 18563 (python) S 18490 18490 18387 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 658216071 327794688 5758 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604470096154944 0 0 -1 5 0 0 0
[pid=18523] ppid=18490 vsize=330512 CPUtime=1.88 cores=1,3,5,7
/proc/18523/stat : 18523 (python) S 18490 18490 18387 0 -1 4202560 1149 0 0 0 102 86 0 0 15 0 2 0 658147575 338444288 5781 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18523/statm: 82628 5781 400 364 0 18531 0
[pid=18525] ppid=18523 vsize=141204 CPUtime=779.65 cores=1,3,5,7
/proc/18525/stat : 18525 (march_hi) T 18523 18525 18525 0 -1 4202496 33834 0 0 0 77958 7 0 0 25 0 1 0 658147575 144592896 33572 33554432000 4194304 4955518 140735660801296 18446744073709551615 4208624 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/18525/statm: 35301 33572 106 186 0 35114 0
[pid=18523/tid=18564] ppid=18490 vsize=330512 CPUtime=0 cores=1,3,5,7
/proc/18523/task/18564/stat : 18564 (python) S 18490 18490 18387 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 658216272 338444288 5781 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604470096157632 0 0 -1 3 0 0 0
[pid=18537] ppid=18490 vsize=340324 CPUtime=1.64 cores=1,3,5,7
/proc/18537/stat : 18537 (python) S 18490 18490 18387 0 -1 4202560 1155 0 0 0 80 84 0 0 18 0 2 0 658177178 348491776 5706 33554432000 4194304 5683644 140733493568864 18446744073709551615 218370977350 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18537/statm: 85081 5706 400 364 0 20981 0
[pid=18539] ppid=18537 vsize=181380 CPUtime=700.97 cores=1,3,5,7
/proc/18539/stat : 18539 (kcnfs-2006) R 18537 18539 18539 0 -1 4202496 680 0 0 0 70097 0 0 0 25 0 1 0 658177179 185733120 420 33554432000 134512640 135163436 4287742640 18446744073709551615 134609236 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/18539/statm: 45345 420 76 159 0 45185 0
[pid=18537/tid=18702] ppid=18490 vsize=340324 CPUtime=0 cores=1,3,5,7
/proc/18537/task/18702/stat : 18702 (python) S 18490 18490 18387 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658235906 348491776 5706 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604463156196608 0 0 -1 1 0 0 0
[pid=18555] ppid=18490 vsize=350972 CPUtime=1.27 cores=1,3,5,7
/proc/18555/stat : 18555 (python) S 18490 18490 18387 0 -1 4202560 1206 0 0 0 67 60 0 0 15 0 2 0 658205581 359395328 5799 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382783947 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/18555/statm: 87743 5799 406 364 0 23640 0
[pid=18557] ppid=18555 vsize=97004 CPUtime=512.98 cores=1,3,5,7
/proc/18557/stat : 18557 (march_hi) T 18555 18557 18557 0 -1 4202496 23159 0 0 0 51294 4 0 0 25 0 1 0 658205581 99332096 22886 33554432000 4194304 4955518 140734146644192 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/18557/statm: 24251 22886 106 186 0 24064 0
[pid=18555/tid=18713] ppid=18490 vsize=350972 CPUtime=0 cores=1,3,5,7
/proc/18555/task/18713/stat : 18713 (python) S 18490 18490 18387 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658245125 359395328 5799 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 514 18446604470096155840 0 0 -1 3 0 0 0
[pid=18490/tid=18497] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18497/stat : 18497 (python) S 18488 18490 18387 0 -1 4202560 27 2574 0 0 0 0 9939 11 19 0 9 0 658137552 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18500] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18500/stat : 18500 (python) S 18488 18490 18387 0 -1 4202560 24 2574 0 0 0 0 9939 11 20 0 9 0 658137552 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 1 0 0 0
[pid=18490/tid=18503] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18503/stat : 18503 (python) S 18488 18490 18387 0 -1 4202560 29 2574 0 0 0 0 9939 11 18 0 9 0 658137553 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 5 0 0 0
[pid=18490/tid=18506] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18506/stat : 18506 (python) S 18488 18490 18387 0 -1 4202560 32 2574 0 0 0 0 9939 11 18 0 9 0 658137555 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18520] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18520/stat : 18520 (python) S 18488 18490 18387 0 -1 4202560 34 2574 0 0 0 0 9939 11 18 0 9 0 658147574 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18524] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18524/stat : 18524 (python) S 18488 18490 18387 0 -1 4202560 33 2574 0 0 0 0 9939 11 18 0 9 0 658147575 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18538] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18538/stat : 18538 (python) S 18488 18490 18387 0 -1 4202560 30 2574 0 0 0 0 9939 11 18 0 9 0 658177179 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
[pid=18490/tid=18556] ppid=18488 vsize=361416 CPUtime=99.5 cores=1,3,5,7
/proc/18490/task/18556/stat : 18556 (python) S 18488 18490 18387 0 -1 4202560 26 2574 0 0 0 0 9939 11 18 0 9 0 658205581 370089984 7338 33554432000 4194304 5683644 140733493568864 18446744073709551615 218382780849 0 0 16781312 2 18446604466013954432 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4775.69
Current children cumulated vsize (KiB) 3229104

Child status: 0
Real time (s): 1197.56
CPU time (s): 4776.01
CPU user time (s): 4770.01
CPU system time (s): 5.99409
CPU usage (%): 398.812
Max. virtual memory (cumulated for all children) (KiB): 5842848

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4770.01
system time used= 5.99409
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 200275
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7965
involuntary context switches= 37127

runsolver used 4.08538 second user time and 8.16076 second system time

The end

Launcher Data 

Begin job on node101 at 2011-05-15 15:15:47
IDJOB=3378427
IDBENCH=82836
IDSOLVER=1859
FILE ID=node101/3378427-1305465347
RUNJOBID= node101-1305465320-18408
PBS_JOBID= 13325061
Free space on /tmp= 73724 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S1061710238-001.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378427-1305465347/watcher-3378427-1305465347 -o /tmp/evaluation-result-3378427-1305465347/solver-3378427-1305465347 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378427-1305465347.cnf -seed 873408899 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 23ebfbdf095807765a162fcb1d02f57a
RANDOM SEED=873408899

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15779160 kB
Buffers:        343016 kB
Cached:        6259484 kB
SwapCached:      14848 kB
Active:       11687292 kB
Inactive:      5191084 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15779160 kB
SwapTotal:    67111528 kB
SwapFree:     52307768 kB
Dirty:           64496 kB
Writeback:           0 kB
AnonPages:    10272556 kB
Mapped:          22676 kB
Slab:           160300 kB
PageTables:      66784 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 44864564 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73696 MiB
End job on node101 at 2011-05-15 15:35:45