Trace number 3322123

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-randomUNSAT 2891.06 362.715

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v100-c2130-S360859187-040.UNKNOWN.cnf
MD5SUM5da80be8e24322484381341b0d82cc61
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark358.158
Satisfiable
(Un)Satisfiability was proved
Number of variables100
Number of clauses2130
Sum of the clauses size10650
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52130
Number of clauses of size over 50

Solver Data 

0.32/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.46	c INFO: solving HOME/instance-3322123-1304374910.cnf
0.32/0.46	c INFO: computing task features
0.32/0.47	c INFO: computed variable-clause graph statistics
0.32/0.48	c INFO: computed clause balance statistics
0.41/0.53	c INFO: computed variable balance statistics
0.41/0.56	c INFO: computed small-clause counts
0.41/0.56	c INFO: computed Horn-clause counts
0.41/0.56	c INFO: computed variable graph statistics
0.41/0.56	c INFO: collected features for HOME/instance-3322123-1304374910.cnf in 0.10 s
0.51/0.60	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.62)
0.51/0.63	c INFO: running march_hi@0 for 194 with 39999 remaining (b = 0.19)
0.51/0.67	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.07)
0.61/0.71	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.30)
0.61/0.75	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.26)
0.61/0.79	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.28)
0.71/0.83	c INFO: running march_hi@0 for 485 with 39999 remaining (b = 0.25)
0.71/0.87	c INFO: running march_hi@0 for 388 with 39999 remaining (b = 0.18)
800.32/100.82	c INFO: running kcnfs-2006@0 for 582 with 39899 remaining (b = 0.21)
800.32/100.89	c INFO: running march_hi@0 for 388 with 39899 remaining (b = 0.14)
1586.55/199.32	c INFO: running march_hi@0 for 97 with 39801 remaining (b = 0.04)
2340.94/293.82	c INFO: running march_hi@93 for 97 with 39706 remaining (b = 0.04)
2890.14/362.69	s UNSATISFIABLE

Verifier Data 

No possible verification on an UNSAT instance

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322123-1304374910/watcher-3322123-1304374910 -o /tmp/evaluation-result-3322123-1304374910/solver-3322123-1304374910 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322123-1304374910.cnf -seed 1469432572 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.17 8.21 8.43 1/255 21300
/proc/meminfo: memFree=18960412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=15776 CPUtime=0 cores=0-7
/proc/21300/stat : 21300 (python) R 21298 21300 21159 0 -1 4202496 889 0 0 0 0 0 0 0 17 0 1 0 549088985 16154624 770 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122374853 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/21300/statm: 3944 771 343 364 0 431 0

[startup+0.0489721 s]
/proc/loadavg: 7.17 8.21 8.43 1/255 21300
/proc/meminfo: memFree=18960412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=29792 CPUtime=0.03 cores=0-7
/proc/21300/stat : 21300 (python) R 21298 21300 21159 0 -1 4202496 1453 0 0 0 3 0 0 0 17 0 1 0 549088985 30507008 1294 33554432000 4194304 5683644 140735468663968 18446744073709551615 4516228 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/21300/statm: 7448 1294 443 364 0 818 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29792

[startup+0.10097 s]
/proc/loadavg: 7.17 8.21 8.43 1/255 21300
/proc/meminfo: memFree=18960412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=104232 CPUtime=0.07 cores=0-7
/proc/21300/stat : 21300 (python) R 21298 21300 21159 0 -1 4202496 3088 0 0 0 5 2 0 0 17 0 1 0 549088985 106733568 2768 33554432000 4194304 5683644 140735468663968 18446744073709551615 4429136 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/21300/statm: 26058 2768 847 364 0 1806 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104232

[startup+0.300929 s]
/proc/loadavg: 7.17 8.21 8.43 1/255 21300
/proc/meminfo: memFree=18960412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=187688 CPUtime=0.25 cores=0-7
/proc/21300/stat : 21300 (python) S 21298 21300 21159 0 -1 4202496 6827 934 1 0 18 7 0 0 18 0 1 0 549088985 192192512 4882 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122712064 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/21300/statm: 46923 4883 1399 364 0 3724 0
Current children cumulated CPU time (s) 0.25
Current children cumulated vsize (KiB) 187688

[startup+0.700847 s]
/proc/loadavg: 7.17 8.21 8.43 1/255 21300
/proc/meminfo: memFree=18960412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=375312 CPUtime=0.61 cores=0-7
/proc/21300/stat : 21300 (python) R 21298 21300 21159 0 -1 4202496 11760 934 1 0 51 10 0 0 18 0 4 0 549088985 384319488 7348 33554432000 4194304 5683644 140735468663968 18446744073709551615 46943470507338 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/21300/statm: 93828 7348 1878 364 0 29740 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 375312

[startup+1.50167 s]
/proc/loadavg: 7.17 8.21 8.43 9/280 21329
/proc/meminfo: memFree=18907156/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=426592 CPUtime=0.78 cores=0-7
/proc/21300/stat : 21300 (python) S 21298 21300 21159 0 -1 4202496 15280 934 1 0 67 11 0 0 24 0 9 0 549088985 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/21300/statm: 106648 7423 1889 364 0 42545 0
[pid=21306] ppid=21300 vsize=278588 CPUtime=0 cores=0-7
/proc/21306/stat : 21306 (python) S 21300 21300 21159 0 -1 4202560 732 0 0 0 0 0 0 0 19 0 1 0 549089045 285274112 5560 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734095 0 0 16781312 514 4124466032175742975 0 0 17 1 0 0 0
/proc/21306/statm: 69647 5560 354 364 0 5565 0
[pid=21308] ppid=21306 vsize=1017800 CPUtime=0.89 cores=0-7
/proc/21308/stat : 21308 (gnovelty+2) R 21306 21308 21308 0 -1 4202496 435 0 0 0 89 0 0 0 23 0 1 0 549089045 1042227200 172 33554432000 134512640 135663704 4288951760 18446744073709551615 134524854 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21308/statm: 254450 172 113 282 0 254166 0
[pid=21309] ppid=21300 vsize=289276 CPUtime=0 cores=0-7
/proc/21309/stat : 21309 (python) S 21300 21300 21159 0 -1 4202560 740 0 0 0 0 0 0 0 15 0 1 0 549089048 296218624 5676 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 8358971183764308227 0 0 17 7 0 0 0
/proc/21309/statm: 72319 5676 348 364 0 8234 0
[pid=21311] ppid=21309 vsize=1560 CPUtime=0.85 cores=0-7
/proc/21311/stat : 21311 (march_hi) R 21309 21311 21311 0 -1 4202496 504 0 0 0 85 0 0 0 25 0 1 0 549089049 1597440 239 33554432000 4194304 4955518 140735593698416 18446744073709551615 4224443 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21311/statm: 390 239 106 186 0 203 0
[pid=21312] ppid=21300 vsize=365068 CPUtime=0 cores=0-7
/proc/21312/stat : 21312 (python) S 21300 21300 21159 0 -1 4202560 707 0 0 0 0 0 0 0 18 0 1 0 549089052 373829632 5853 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 4124458335594348543 0 0 17 1 0 0 0
/proc/21312/statm: 91267 5853 348 364 0 27179 0
[pid=21314] ppid=21312 vsize=1017800 CPUtime=0.82 cores=0-7
/proc/21314/stat : 21314 (gnovelty+2) R 21312 21314 21314 0 -1 4202496 428 0 0 0 82 0 0 0 21 0 1 0 549089052 1042227200 172 33554432000 134512640 135663704 4286750576 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21314/statm: 254450 172 113 282 0 254166 0
[pid=21315] ppid=21300 vsize=375324 CPUtime=0 cores=0-7
/proc/21315/stat : 21315 (python) S 21300 21300 21159 0 -1 4202560 723 0 0 0 0 0 0 0 15 0 1 0 549089056 384331776 5811 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 5602247593057812740 0 0 17 2 0 0 0
/proc/21315/statm: 93831 5811 348 364 0 29740 0
[pid=21317] ppid=21315 vsize=1560 CPUtime=0.77 cores=0-7
/proc/21317/stat : 21317 (march_hi) R 21315 21317 21317 0 -1 4202496 494 0 0 0 77 0 0 0 23 0 1 0 549089056 1597440 239 33554432000 4194304 4955518 140737045514784 18446744073709551615 4237697 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21317/statm: 390 239 106 186 0 203 0
[pid=21318] ppid=21300 vsize=385580 CPUtime=0 cores=0-7
/proc/21318/stat : 21318 (python) S 21300 21300 21159 0 -1 4202560 728 0 0 0 0 0 0 0 15 0 1 0 549089060 394833920 5935 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 16368227296856736001 0 0 17 0 0 0 0
/proc/21318/statm: 96395 5935 348 364 0 32301 0
[pid=21320] ppid=21318 vsize=1560 CPUtime=0.74 cores=0-7
/proc/21320/stat : 21320 (march_hi) R 21318 21320 21320 0 -1 4202496 496 0 0 0 74 0 0 0 25 0 1 0 549089060 1597440 239 33554432000 4194304 4955518 140733304243472 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21320/statm: 390 239 106 186 0 203 0
[pid=21321] ppid=21300 vsize=395836 CPUtime=0 cores=0-7
/proc/21321/stat : 21321 (python) S 21300 21300 21159 0 -1 4202560 738 0 0 0 0 0 0 0 15 0 1 0 549089064 405336064 5939 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 2573024339058196738 0 0 17 0 0 0 0
/proc/21321/statm: 98959 5939 348 364 0 34862 0
[pid=21323] ppid=21321 vsize=1560 CPUtime=0.7 cores=0-7
/proc/21323/stat : 21323 (march_hi) R 21321 21323 21323 0 -1 4202496 496 0 0 0 70 0 0 0 22 0 1 0 549089064 1597440 238 33554432000 4194304 4955518 140736146739520 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21323/statm: 390 238 106 186 0 203 0
[pid=21324] ppid=21300 vsize=406092 CPUtime=0 cores=0-7
/proc/21324/stat : 21324 (python) S 21300 21300 21159 0 -1 4202560 744 0 0 0 0 0 0 0 17 0 1 0 549089068 415838208 5857 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 4436525024324583682 0 0 17 6 0 0 0
/proc/21324/statm: 101523 5857 348 364 0 37423 0
[pid=21326] ppid=21324 vsize=1560 CPUtime=0.66 cores=0-7
/proc/21326/stat : 21326 (march_hi) R 21324 21326 21326 0 -1 4202496 497 0 0 0 66 0 0 0 25 0 1 0 549089068 1597440 239 33554432000 4194304 4955518 140736684226400 18446744073709551615 4297067 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21326/statm: 390 239 106 186 0 203 0
[pid=21327] ppid=21300 vsize=416348 CPUtime=0 cores=0-7
/proc/21327/stat : 21327 (python) S 21300 21300 21159 0 -1 4202560 772 0 0 0 0 0 0 0 19 0 1 0 549089072 426340352 5884 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 17285167217863786753 0 0 17 0 0 0 0
/proc/21327/statm: 104087 5884 354 364 0 39984 0
[pid=21329] ppid=21327 vsize=1560 CPUtime=0.61 cores=0-7
/proc/21329/stat : 21329 (march_hi) R 21327 21329 21329 0 -1 4202496 505 0 0 0 61 0 0 0 25 0 1 0 549089072 1597440 239 33554432000 4194304 4955518 140737345189840 18446744073709551615 4224471 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21329/statm: 390 239 106 186 0 203 0
[pid=21300/tid=21307] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21307/stat : 21307 (python) S 21298 21300 21159 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 9 0 549089045 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21310] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21310/stat : 21310 (python) S 21298 21300 21159 0 -1 4202560 22 934 0 0 0 0 0 0 18 0 9 0 549089049 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 3 0 0 0
[pid=21300/tid=21313] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21313/stat : 21313 (python) S 21298 21300 21159 0 -1 4202560 77 934 0 0 0 0 0 0 19 0 9 0 549089052 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21316] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21316/stat : 21316 (python) S 21298 21300 21159 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 549089056 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 2 0 0 0
[pid=21300/tid=21319] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21319/stat : 21319 (python) S 21298 21300 21159 0 -1 4202560 34 934 0 0 0 0 0 0 22 0 9 0 549089060 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 6 0 0 0
[pid=21300/tid=21322] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21322/stat : 21322 (python) S 21298 21300 21159 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 549089064 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 6 0 0 0
[pid=21300/tid=21325] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21325/stat : 21325 (python) S 21298 21300 21159 0 -1 4202560 34 934 0 0 0 0 0 0 22 0 9 0 549089068 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21328] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21328/stat : 21328 (python) S 21298 21300 21159 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 9 0 549089072 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 6.82
Current children cumulated vsize (KiB) 5383664

[startup+3.10138 s]
/proc/loadavg: 7.17 8.21 8.43 9/280 21329
/proc/meminfo: memFree=18905420/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=426592 CPUtime=0.78 cores=0-7
/proc/21300/stat : 21300 (python) S 21298 21300 21159 0 -1 4202496 15280 934 1 0 67 11 0 0 24 0 9 0 549088985 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/21300/statm: 106648 7423 1889 364 0 42545 0
[pid=21306] ppid=21300 vsize=278588 CPUtime=0 cores=0-7
/proc/21306/stat : 21306 (python) S 21300 21300 21159 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 549089045 285274112 5560 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734095 0 0 16781312 514 4124466032175710468 0 0 17 1 0 0 0
/proc/21306/statm: 69647 5560 354 364 0 5565 0
[pid=21308] ppid=21306 vsize=1017800 CPUtime=2.48 cores=0-7
/proc/21308/stat : 21308 (gnovelty+2) R 21306 21308 21308 0 -1 4202496 435 0 0 0 248 0 0 0 25 0 1 0 549089045 1042227200 172 33554432000 134512640 135663704 4288951760 18446744073709551615 134524888 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21308/statm: 254450 172 113 282 0 254166 0
[pid=21309] ppid=21300 vsize=289276 CPUtime=0 cores=0-7
/proc/21309/stat : 21309 (python) S 21300 21300 21159 0 -1 4202560 741 0 0 0 0 0 0 0 15 0 1 0 549089048 296218624 5676 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 8358971183764308227 0 0 17 7 0 0 0
/proc/21309/statm: 72319 5676 348 364 0 8234 0
[pid=21311] ppid=21309 vsize=1560 CPUtime=2.45 cores=0-7
/proc/21311/stat : 21311 (march_hi) R 21309 21311 21311 0 -1 4202496 507 0 0 0 245 0 0 0 25 0 1 0 549089049 1597440 242 33554432000 4194304 4955518 140735593698416 18446744073709551615 4208656 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21311/statm: 390 242 106 186 0 203 0
[pid=21312] ppid=21300 vsize=365068 CPUtime=0 cores=0-7
/proc/21312/stat : 21312 (python) S 21300 21300 21159 0 -1 4202560 707 0 0 0 0 0 0 0 18 0 1 0 549089052 373829632 5853 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 4124458335594316033 0 0 17 1 0 0 0
/proc/21312/statm: 91267 5853 348 364 0 27179 0
[pid=21314] ppid=21312 vsize=1017800 CPUtime=2.42 cores=0-7
/proc/21314/stat : 21314 (gnovelty+2) R 21312 21314 21314 0 -1 4202496 428 0 0 0 242 0 0 0 25 0 1 0 549089052 1042227200 172 33554432000 134512640 135663704 4286750576 18446744073709551615 134545572 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21314/statm: 254450 172 113 282 0 254166 0
[pid=21315] ppid=21300 vsize=375324 CPUtime=0 cores=0-7
/proc/21315/stat : 21315 (python) S 21300 21300 21159 0 -1 4202560 723 0 0 0 0 0 0 0 15 0 1 0 549089056 384331776 5811 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 11189716646031753476 0 0 17 2 0 0 0
/proc/21315/statm: 93831 5811 348 364 0 29740 0
[pid=21317] ppid=21315 vsize=1560 CPUtime=2.36 cores=0-7
/proc/21317/stat : 21317 (march_hi) R 21315 21317 21317 0 -1 4202496 497 0 0 0 236 0 0 0 25 0 1 0 549089056 1597440 242 33554432000 4194304 4955518 140737045514784 18446744073709551615 4208638 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21317/statm: 390 242 106 186 0 203 0
[pid=21318] ppid=21300 vsize=385580 CPUtime=0 cores=0-7
/proc/21318/stat : 21318 (python) S 21300 21300 21159 0 -1 4202560 728 0 0 0 0 0 0 0 15 0 1 0 549089060 394833920 5935 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 16368227296856736001 0 0 17 0 0 0 0
/proc/21318/statm: 96395 5935 348 364 0 32301 0
[pid=21320] ppid=21318 vsize=1560 CPUtime=2.34 cores=0-7
/proc/21320/stat : 21320 (march_hi) R 21318 21320 21320 0 -1 4202496 499 0 0 0 234 0 0 0 25 0 1 0 549089060 1597440 242 33554432000 4194304 4955518 140733304243472 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21320/statm: 390 242 106 186 0 203 0
[pid=21321] ppid=21300 vsize=395836 CPUtime=0 cores=0-7
/proc/21321/stat : 21321 (python) S 21300 21300 21159 0 -1 4202560 738 0 0 0 0 0 0 0 15 0 1 0 549089064 405336064 5939 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 2573024339058196738 0 0 17 0 0 0 0
/proc/21321/statm: 98959 5939 348 364 0 34862 0
[pid=21323] ppid=21321 vsize=1560 CPUtime=2.3 cores=0-7
/proc/21323/stat : 21323 (march_hi) R 21321 21323 21323 0 -1 4202496 499 0 0 0 230 0 0 0 25 0 1 0 549089064 1597440 241 33554432000 4194304 4955518 140736146739520 18446744073709551615 4224811 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21323/statm: 390 241 106 186 0 203 0
[pid=21324] ppid=21300 vsize=406092 CPUtime=0 cores=0-7
/proc/21324/stat : 21324 (python) S 21300 21300 21159 0 -1 4202560 746 0 0 0 0 0 0 0 15 0 1 0 549089068 415838208 5857 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 4436525024324583682 0 0 17 6 0 0 0
/proc/21324/statm: 101523 5857 348 364 0 37423 0
[pid=21326] ppid=21324 vsize=1560 CPUtime=2.26 cores=0-7
/proc/21326/stat : 21326 (march_hi) R 21324 21326 21326 0 -1 4202496 500 0 0 0 226 0 0 0 25 0 1 0 549089068 1597440 242 33554432000 4194304 4955518 140736684226400 18446744073709551615 4306969 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21326/statm: 390 242 106 186 0 203 0
[pid=21327] ppid=21300 vsize=416348 CPUtime=0 cores=0-7
/proc/21327/stat : 21327 (python) S 21300 21300 21159 0 -1 4202560 774 0 0 0 0 0 0 0 15 0 1 0 549089072 426340352 5884 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 17285167217863786753 0 0 17 0 0 0 0
/proc/21327/statm: 104087 5884 354 364 0 39984 0
[pid=21329] ppid=21327 vsize=1560 CPUtime=2.21 cores=0-7
/proc/21329/stat : 21329 (march_hi) R 21327 21329 21329 0 -1 4202496 508 0 0 0 221 0 0 0 25 0 1 0 549089072 1597440 242 33554432000 4194304 4955518 140737345189840 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21329/statm: 390 242 106 186 0 203 0
[pid=21300/tid=21307] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21307/stat : 21307 (python) S 21298 21300 21159 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 9 0 549089045 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21310] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21310/stat : 21310 (python) S 21298 21300 21159 0 -1 4202560 22 934 0 0 0 0 0 0 18 0 9 0 549089049 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 3 0 0 0
[pid=21300/tid=21313] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21313/stat : 21313 (python) S 21298 21300 21159 0 -1 4202560 77 934 0 0 0 0 0 0 19 0 9 0 549089052 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21316] ppid=21298 vsize=426592 CPUtime=0 cores=0-7
/proc/21300/task/21316/stat : 21316 (python) S 21298 21300 21159 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 9 0 549089056 436830208 7423 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 2 0 0 0
[pid=21300/tid=21319] ppid=21298 vsize=426592 CPUtime=0 cores=0-7

################
# More data... #
################

[pid=21312/tid=21337] ppid=21300 vsize=365068 CPUtime=0 cores=0-7
/proc/21312/task/21337/stat : 21337 (python) S 21300 21300 21159 0 -1 4202560 105 0 0 0 0 0 0 0 19 0 2 0 549099070 373829632 5906 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 514 18446604449578026240 0 0 -1 1 0 0 0
[pid=21315] ppid=21300 vsize=375324 CPUtime=0.74 cores=0-7
/proc/21315/stat : 21315 (python) S 21300 21300 21159 0 -1 4202560 808 0 0 0 41 33 0 0 15 0 1 0 549089056 384331776 5834 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 11189716646031753476 0 0 17 7 0 0 0
/proc/21315/statm: 93831 5834 371 364 0 29740 0
[pid=21317] ppid=21315 vsize=13516 CPUtime=355.21 cores=0-7
/proc/21317/stat : 21317 (march_hi) R 21315 21317 21317 0 -1 4202496 3060 0 0 0 35520 1 0 0 25 0 1 0 549089056 13840384 2805 33554432000 4194304 4955518 140737045514784 18446744073709551615 4290789 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21317/statm: 3379 2805 106 186 0 3192 0
[pid=21318] ppid=21300 vsize=385580 CPUtime=0.76 cores=0-7
/proc/21318/stat : 21318 (python) S 21300 21300 21159 0 -1 4202560 809 0 0 0 43 33 0 0 15 0 1 0 549089060 394833920 5958 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 16368227296856736001 0 0 17 2 0 0 0
/proc/21318/statm: 96395 5958 371 364 0 32301 0
[pid=21320] ppid=21318 vsize=13516 CPUtime=359.51 cores=0-7
/proc/21320/stat : 21320 (march_hi) R 21318 21320 21320 0 -1 4202496 3183 0 0 0 35951 0 0 0 25 0 1 0 549089060 13840384 2926 33554432000 4194304 4955518 140733304243472 18446744073709551615 4297472 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21320/statm: 3379 2926 106 186 0 3192 0
[pid=21321] ppid=21300 vsize=395836 CPUtime=0.74 cores=0-7
/proc/21321/stat : 21321 (python) S 21300 21300 21159 0 -1 4202560 829 0 0 0 39 35 0 0 15 0 1 0 549089064 405336064 5962 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 2573024339058196738 0 0 17 1 0 0 0
/proc/21321/statm: 98959 5962 371 364 0 34862 0
[pid=21323] ppid=21321 vsize=13516 CPUtime=359.92 cores=0-7
/proc/21323/stat : 21323 (march_hi) R 21321 21323 21323 0 -1 4202496 3163 0 0 0 35991 1 0 0 25 0 1 0 549089064 13840384 2905 33554432000 4194304 4955518 140736146739520 18446744073709551615 4224427 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21323/statm: 3379 2905 106 186 0 3192 0
[pid=21324] ppid=21300 vsize=406092 CPUtime=0.94 cores=0-7
/proc/21324/stat : 21324 (python) R 21300 21300 21159 0 -1 4202560 849 0 0 0 57 37 0 0 16 0 1 0 549089068 415838208 5880 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134544191 0 0 16781312 514 0 0 0 17 6 0 0 0
/proc/21324/statm: 101523 5880 371 364 0 37423 0
[pid=21326] ppid=21324 vsize=0 CPUtime=359.67 cores=0-7
/proc/21326/stat : 21326 (march_hi) R 21324 21326 21326 0 -1 4202500 3203 0 0 0 35966 1 0 0 25 0 1 0 549089068 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 6 0 0 0
/proc/21326/statm: 0 0 0 0 0 0 0
[pid=21327] ppid=21300 vsize=416348 CPUtime=0.77 cores=0-7
/proc/21327/stat : 21327 (python) S 21300 21300 21159 0 -1 4202560 871 0 0 0 44 33 0 0 15 0 1 0 549089072 426340352 5907 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 17285167217863786753 0 0 17 4 0 0 0
/proc/21327/statm: 104087 5907 377 364 0 39984 0
[pid=21329] ppid=21327 vsize=13516 CPUtime=359.78 cores=0-7
/proc/21329/stat : 21329 (march_hi) R 21327 21329 21329 0 -1 4202496 3171 0 0 0 35976 2 0 0 25 0 1 0 549089072 13840384 2905 33554432000 4194304 4955518 140737345189840 18446744073709551615 4226483 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21329/statm: 3379 2905 106 186 0 3192 0
[pid=21334] ppid=21300 vsize=426604 CPUtime=0.46 cores=0-7
/proc/21334/stat : 21334 (python) S 21300 21300 21159 0 -1 4202560 839 0 0 0 29 17 0 0 18 0 1 0 549099067 436842496 5888 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 16946763922818433283 0 0 17 0 0 0 0
/proc/21334/statm: 106651 5888 371 364 0 42545 0
[pid=21336] ppid=21334 vsize=181248 CPUtime=260.67 cores=0-7
/proc/21336/stat : 21336 (kcnfs-2006) R 21334 21336 21336 0 -1 4202496 622 0 0 0 26067 0 0 0 25 0 1 0 549099067 185597952 359 33554432000 134512640 135163436 4292765232 18446744073709551615 134608590 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21336/statm: 45312 359 76 159 0 45152 0
[pid=21338] ppid=21300 vsize=436860 CPUtime=0.56 cores=0-7
/proc/21338/stat : 21338 (python) S 21300 21300 21159 0 -1 4202560 872 0 0 0 30 26 0 0 15 0 1 0 549099074 447344640 5902 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 4169691144449392898 0 0 17 3 0 0 0
/proc/21338/statm: 109215 5902 372 364 0 45106 0
[pid=21340] ppid=21338 vsize=9456 CPUtime=259.49 cores=0-7
/proc/21340/stat : 21340 (march_hi) R 21338 21340 21340 0 -1 4202496 1995 0 0 0 25947 2 0 0 25 0 1 0 549099074 9682944 1730 33554432000 4194304 4955518 140733340552400 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21340/statm: 2364 1730 106 186 0 2177 0
[pid=21300/tid=21307] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21307/stat : 21307 (python) S 21298 21300 21159 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 11 0 549089045 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21310] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21310/stat : 21310 (python) S 21298 21300 21159 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 11 0 549089049 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 3 0 0 0
[pid=21300/tid=21313] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21313/stat : 21313 (python) S 21298 21300 21159 0 -1 4202560 77 934 0 0 0 0 0 0 19 0 11 0 549089052 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21316] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21316/stat : 21316 (python) S 21298 21300 21159 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 11 0 549089056 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 2 0 0 0
[pid=21300/tid=21319] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21319/stat : 21319 (python) S 21298 21300 21159 0 -1 4202560 34 934 0 0 0 0 0 0 22 0 11 0 549089060 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 6 0 0 0
[pid=21300/tid=21322] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21322/stat : 21322 (python) S 21298 21300 21159 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 11 0 549089064 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 6 0 0 0
[pid=21300/tid=21325] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21325/stat : 21325 (python) S 21298 21300 21159 0 -1 4202560 34 934 0 0 0 0 0 0 22 0 11 0 549089068 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21328] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21328/stat : 21328 (python) S 21298 21300 21159 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 11 0 549089072 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21335] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21335/stat : 21335 (python) S 21298 21300 21159 0 -1 4202560 39 934 0 0 0 0 0 0 20 0 11 0 549099067 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21339] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21339/stat : 21339 (python) S 21298 21300 21159 0 -1 4202560 35 934 0 0 0 0 0 0 21 0 11 0 549099074 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2881.35
Current children cumulated vsize (KiB) 6592344

[startup+362.302 s]
/proc/loadavg: 8.21 8.18 8.34 10/289 21348
/proc/meminfo: memFree=18839412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=447104 CPUtime=0.97 cores=0-7
/proc/21300/stat : 21300 (python) S 21298 21300 21159 0 -1 4202496 17613 934 1 0 85 12 0 0 18 0 11 0 549088985 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/21300/statm: 111776 7462 1894 364 0 47667 0
[pid=21306] ppid=21300 vsize=288832 CPUtime=0.18 cores=0-7
/proc/21306/stat : 21306 (python) S 21300 21300 21159 0 -1 4202560 1090 0 0 0 10 8 0 0 19 0 2 0 549089045 295763968 5621 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21306/statm: 72208 5621 410 364 0 8126 0
[pid=21308] ppid=21306 vsize=1017800 CPUtime=100.16 cores=0-7
/proc/21308/stat : 21308 (gnovelty+2) T 21306 21308 21308 0 -1 4202496 435 0 0 0 10016 0 0 0 25 0 1 0 549089045 1042227200 172 33554432000 134512640 135663704 4288951760 18446744073709551615 134524548 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/21308/statm: 254450 172 113 282 0 254166 0
[pid=21306/tid=21333] ppid=21300 vsize=288832 CPUtime=0 cores=0-7
/proc/21306/task/21333/stat : 21333 (python) S 21300 21300 21159 0 -1 4202560 52 0 0 0 0 0 0 0 19 0 2 0 549099063 295763968 5621 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 514 18446604449578028032 0 0 -1 4 0 0 0
[pid=21309] ppid=21300 vsize=354812 CPUtime=0.76 cores=0-7
/proc/21309/stat : 21309 (python) S 21300 21300 21159 0 -1 4202560 1151 0 0 0 40 36 0 0 15 0 2 0 549089048 363327488 5731 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21309/statm: 88703 5731 400 364 0 24618 0
[pid=21311] ppid=21309 vsize=13516 CPUtime=360.51 cores=0-7
/proc/21311/stat : 21311 (march_hi) R 21309 21311 21311 0 -1 4202496 3199 0 0 0 36049 2 0 0 25 0 1 0 549089049 13840384 2934 33554432000 4194304 4955518 140735593698416 18446744073709551615 4237772 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21311/statm: 3379 2934 106 186 0 3192 0
[pid=21309/tid=21344] ppid=21300 vsize=354812 CPUtime=0 cores=0-7
/proc/21309/task/21344/stat : 21344 (python) S 21300 21300 21159 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 549108912 363327488 5731 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 514 18446604449578027136 0 0 -1 0 0 0 0
[pid=21312] ppid=21300 vsize=365068 CPUtime=0.18 cores=0-7
/proc/21312/stat : 21312 (python) S 21300 21300 21159 0 -1 4202560 991 0 0 0 9 9 0 0 19 0 2 0 549089052 373829632 5906 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/21312/statm: 91267 5906 400 364 0 27179 0
[pid=21314] ppid=21312 vsize=1017800 CPUtime=100.17 cores=0-7
/proc/21314/stat : 21314 (gnovelty+2) T 21312 21314 21314 0 -1 4202496 428 0 0 0 10017 0 0 0 25 0 1 0 549089052 1042227200 172 33554432000 134512640 135663704 4286750576 18446744073709551615 135007736 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21314/statm: 254450 172 113 282 0 254166 0
[pid=21312/tid=21337] ppid=21300 vsize=365068 CPUtime=0 cores=0-7
/proc/21312/task/21337/stat : 21337 (python) S 21300 21300 21159 0 -1 4202560 105 0 0 0 0 0 0 0 19 0 2 0 549099070 373829632 5906 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 514 18446604449578026240 0 0 -1 1 0 0 0
[pid=21315] ppid=21300 vsize=375324 CPUtime=0.74 cores=0-7
/proc/21315/stat : 21315 (python) S 21300 21300 21159 0 -1 4202560 808 0 0 0 41 33 0 0 15 0 1 0 549089056 384331776 5834 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 11189716646031753476 0 0 17 7 0 0 0
/proc/21315/statm: 93831 5834 371 364 0 29740 0
[pid=21317] ppid=21315 vsize=13516 CPUtime=355.99 cores=0-7
/proc/21317/stat : 21317 (march_hi) R 21315 21317 21317 0 -1 4202496 3073 0 0 0 35598 1 0 0 25 0 1 0 549089056 13840384 2818 33554432000 4194304 4955518 140737045514784 18446744073709551615 4208638 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/21317/statm: 3379 2818 106 186 0 3192 0
[pid=21318] ppid=21300 vsize=385580 CPUtime=0.76 cores=0-7
/proc/21318/stat : 21318 (python) S 21300 21300 21159 0 -1 4202560 809 0 0 0 43 33 0 0 15 0 1 0 549089060 394833920 5958 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 16368227296856736001 0 0 17 2 0 0 0
/proc/21318/statm: 96395 5958 371 364 0 32301 0
[pid=21320] ppid=21318 vsize=13516 CPUtime=360.31 cores=0-7
/proc/21320/stat : 21320 (march_hi) R 21318 21320 21320 0 -1 4202496 3198 0 0 0 36031 0 0 0 25 0 1 0 549089060 13840384 2941 33554432000 4194304 4955518 140733304243472 18446744073709551615 4224551 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21320/statm: 3379 2941 106 186 0 3192 0
[pid=21321] ppid=21300 vsize=395836 CPUtime=0.74 cores=0-7
/proc/21321/stat : 21321 (python) S 21300 21300 21159 0 -1 4202560 829 0 0 0 39 35 0 0 15 0 1 0 549089064 405336064 5962 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 2573024339058196738 0 0 17 1 0 0 0
/proc/21321/statm: 98959 5962 371 364 0 34862 0
[pid=21323] ppid=21321 vsize=13516 CPUtime=360.72 cores=0-7
/proc/21323/stat : 21323 (march_hi) R 21321 21323 21323 0 -1 4202496 3176 0 0 0 36071 1 0 0 25 0 1 0 549089064 13840384 2918 33554432000 4194304 4955518 140736146739520 18446744073709551615 4208736 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21323/statm: 3379 2918 106 186 0 3192 0
[pid=21324] ppid=21300 vsize=406092 CPUtime=1.73 cores=0-7
/proc/21324/stat : 21324 (python) R 21300 21300 21159 0 -1 4202560 849 0 0 0 126 47 0 0 17 0 1 0 549089068 415838208 5880 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 0 0 0 17 6 0 0 0
/proc/21324/statm: 101523 5880 371 364 0 37423 0
[pid=21326] ppid=21324 vsize=0 CPUtime=359.67 cores=0-7
/proc/21326/stat : 21326 (march_hi) R 21324 21326 21326 0 -1 4202500 3203 0 0 0 35966 1 0 0 25 0 1 0 549089068 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 6 0 0 0
/proc/21326/statm: 0 0 0 0 0 0 0
[pid=21327] ppid=21300 vsize=416348 CPUtime=0.77 cores=0-7
/proc/21327/stat : 21327 (python) S 21300 21300 21159 0 -1 4202560 871 0 0 0 44 33 0 0 15 0 1 0 549089072 426340352 5907 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 5602249139246039297 0 0 17 4 0 0 0
/proc/21327/statm: 104087 5907 377 364 0 39984 0
[pid=21329] ppid=21327 vsize=13516 CPUtime=360.58 cores=0-7
/proc/21329/stat : 21329 (march_hi) R 21327 21329 21329 0 -1 4202496 3184 0 0 0 36056 2 0 0 25 0 1 0 549089072 13840384 2918 33554432000 4194304 4955518 140737345189840 18446744073709551615 4224471 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21329/statm: 3379 2918 106 186 0 3192 0
[pid=21334] ppid=21300 vsize=426604 CPUtime=0.46 cores=0-7
/proc/21334/stat : 21334 (python) S 21300 21300 21159 0 -1 4202560 839 0 0 0 29 17 0 0 18 0 1 0 549099067 436842496 5888 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 16946763922818433283 0 0 17 0 0 0 0
/proc/21334/statm: 106651 5888 371 364 0 42545 0
[pid=21336] ppid=21334 vsize=181248 CPUtime=261.47 cores=0-7
/proc/21336/stat : 21336 (kcnfs-2006) R 21334 21336 21336 0 -1 4202496 622 0 0 0 26147 0 0 0 25 0 1 0 549099067 185597952 359 33554432000 134512640 135163436 4292765232 18446744073709551615 134568876 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21336/statm: 45312 359 76 159 0 45152 0
[pid=21338] ppid=21300 vsize=436860 CPUtime=0.56 cores=0-7
/proc/21338/stat : 21338 (python) S 21300 21300 21159 0 -1 4202560 872 0 0 0 30 26 0 0 15 0 1 0 549099074 447344640 5902 33554432000 4194304 5683644 140735468663968 18446744073709551615 228122734150 0 0 16781312 514 4169691144449392898 0 0 17 3 0 0 0
/proc/21338/statm: 109215 5902 372 364 0 45106 0
[pid=21340] ppid=21338 vsize=9456 CPUtime=260.29 cores=0-7
/proc/21340/stat : 21340 (march_hi) R 21338 21340 21340 0 -1 4202496 2005 0 0 0 26027 2 0 0 25 0 1 0 549099074 9682944 1740 33554432000 4194304 4955518 140733340552400 18446744073709551615 4224450 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21340/statm: 2364 1740 106 186 0 2177 0
[pid=21300/tid=21307] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21307/stat : 21307 (python) S 21298 21300 21159 0 -1 4202560 29 934 0 0 0 0 0 0 19 0 11 0 549089045 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21310] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21310/stat : 21310 (python) S 21298 21300 21159 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 11 0 549089049 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 3 0 0 0
[pid=21300/tid=21313] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21313/stat : 21313 (python) S 21298 21300 21159 0 -1 4202560 77 934 0 0 0 0 0 0 19 0 11 0 549089052 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21316] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21316/stat : 21316 (python) S 21298 21300 21159 0 -1 4202560 30 934 0 0 0 0 0 0 20 0 11 0 549089056 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 2 0 0 0
[pid=21300/tid=21319] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21319/stat : 21319 (python) S 21298 21300 21159 0 -1 4202560 34 934 0 0 0 0 0 0 22 0 11 0 549089060 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 6 0 0 0
[pid=21300/tid=21322] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21322/stat : 21322 (python) S 21298 21300 21159 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 11 0 549089064 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 6 0 0 0
[pid=21300/tid=21325] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21325/stat : 21325 (python) S 21298 21300 21159 0 -1 4202560 34 934 0 0 0 0 0 0 22 0 11 0 549089068 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21328] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21328/stat : 21328 (python) S 21298 21300 21159 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 11 0 549089072 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21335] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21335/stat : 21335 (python) S 21298 21300 21159 0 -1 4202560 39 934 0 0 0 0 0 0 20 0 11 0 549099067 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
[pid=21300/tid=21339] ppid=21298 vsize=447104 CPUtime=0 cores=0-7
/proc/21300/task/21339/stat : 21339 (python) S 21298 21300 21159 0 -1 4202560 35 934 0 0 0 0 0 0 21 0 11 0 549099074 457834496 7462 33554432000 4194304 5683644 140735468663968 18446744073709551615 228134537649 0 0 16781312 2 18446604445328610560 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2887.72
Current children cumulated vsize (KiB) 6592344

[startup+362.702 s]
/proc/loadavg: 8.21 8.18 8.34 10/289 21348
/proc/meminfo: memFree=18839412/32951124 swapFree=67111444/67111528
[pid=21300] ppid=21298 vsize=375264 CPUtime=2891.02 cores=0-7
/proc/21300/stat : 21300 (python) R 21298 21300 21159 0 -1 4202496 19580 33494 1 0 85 12 288700 305 17 0 1 0 549088985 384270336 7390 33554432000 4194304 5683644 140735468663968 18446744073709551615 5114130 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21300/statm: 93816 7390 1873 364 0 29740 0
Current children cumulated CPU time (s) 2891.02
Current children cumulated vsize (KiB) 375264

Child status: 0
Real time (s): 362.715
CPU time (s): 2891.06
CPU user time (s): 2887.87
CPU system time (s): 3.18751
CPU usage (%): 797.061
Max. virtual memory (cumulated for all children) (KiB): 6605860

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2887.87
system time used= 3.18751
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 55262
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7935
involuntary context switches= 18893

runsolver used 1.74573 second user time and 3.60345 second system time

The end

Launcher Data 

Begin job on node138 at 2011-05-03 00:21:50
IDJOB=3322123
IDBENCH=82901
IDSOLVER=1712
FILE ID=node138/3322123-1304374910
RUNJOBID= node138-1304374901-21173
PBS_JOBID= 13181780
Free space on /tmp= 73200 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v100-c2130-S360859187-040.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322123-1304374910/watcher-3322123-1304374910 -o /tmp/evaluation-result-3322123-1304374910/solver-3322123-1304374910 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3322123-1304374910.cnf -seed 1469432572 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 5da80be8e24322484381341b0d82cc61
RANDOM SEED=1469432572

node138.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18960576 kB
Buffers:        744100 kB
Cached:        2887744 kB
SwapCached:         84 kB
Active:       10510108 kB
Inactive:      3036268 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18960576 kB
SwapTotal:    67111528 kB
SwapFree:     67111444 kB
Dirty:           32544 kB
Writeback:           0 kB
AnonPages:     9914724 kB
Mapped:          23120 kB
Slab:           344576 kB
PageTables:      33128 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 31878612 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73172 MiB
End job on node138 at 2011-05-03 00:27:53