Trace number 3378382

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 3473.59 1167.33

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v550-c2343-S1663503036-045.UNKNOWN.cnf
MD5SUM8c477af50318fe10cbec1ad215bff97c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables550
Number of clauses2343
Sum of the clauses size7029
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32343
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.29/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.29/0.47	c INFO: solving HOME/instance-3378382-1305465186.cnf
0.29/0.47	c INFO: computing task features
0.29/0.48	c INFO: computed variable-clause graph statistics
0.29/0.48	c INFO: computed clause balance statistics
0.38/0.52	c INFO: computed variable balance statistics
0.38/0.55	c INFO: computed small-clause counts
0.38/0.55	c INFO: computed Horn-clause counts
0.38/0.56	c INFO: computed variable graph statistics
0.38/0.56	c INFO: collected features for HOME/instance-3378382-1305465186.cnf in 0.09 s
0.38/0.57	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.57)
0.38/0.58	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.32)
0.38/0.59	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.48/0.60	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.31)
398.13/148.99	c INFO: running march_hi@0 for 679 with 1051 remaining (b = 0.37)
521.28/196.28	c INFO: running march_hi@0 for 679 with 1004 remaining (b = 0.33)
1306.56/495.21	c INFO: running kcnfs-2006@0 for 582 with 705 remaining (b = 0.28)
1698.07/643.82	c INFO: running march_hi@0 for 556 with 556 remaining (b = 0.23)
2911.71/1005.86	c INFO: running march_hi@0 for 97 with 194 remaining (b = 0.05)
3473.09/1167.31	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378382-1305465186/watcher-3378382-1305465186 -o /tmp/evaluation-result-3378382-1305465186/solver-3378382-1305465186 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378382-1305465186.cnf -seed 106280113 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.98 9.03 7.95 11/389 21834
/proc/meminfo: memFree=25052952/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=99376 CPUtime=0.06 cores=0,2,4,6
/proc/21834/stat : 21834 (python) R 21832 21834 21227 0 -1 4202496 2866 0 0 0 5 1 0 0 25 0 1 0 390709911 101761024 2565 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660660224 0 2147483391 16781312 2 0 0 0 17 0 0 0 0
/proc/21834/statm: 24844 2565 818 364 0 1695 0

[startup+0.200528 s]
/proc/loadavg: 8.98 9.03 7.95 11/389 21834
/proc/meminfo: memFree=25052952/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=126668 CPUtime=0.13 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 5085 936 1 0 11 2 0 0 25 0 1 0 390709911 129708032 3925 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660660224 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/21834/statm: 31667 3925 1003 364 0 2962 0
Current children cumulated CPU time (s) 0.13
Current children cumulated vsize (KiB) 126668

[startup+0.300711 s]
/proc/loadavg: 8.98 9.03 7.95 11/389 21834
/proc/meminfo: memFree=25052952/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=163652 CPUtime=0.21 cores=0,2,4,6
/proc/21834/stat : 21834 (python) R 21832 21834 21227 0 -1 4202496 6632 936 1 0 16 5 0 0 25 0 1 0 390709911 167579648 4731 33554432000 4194304 5683644 140735545757248 18446744073709551615 268655694922 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21834/statm: 40913 4731 1286 364 0 3454 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 163652

[startup+0.400694 s]
/proc/loadavg: 8.98 9.03 7.95 11/389 21834
/proc/meminfo: memFree=25052952/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=246052 CPUtime=0.29 cores=0,2,4,6
/proc/21834/stat : 21834 (python) R 21832 21834 21227 0 -1 4202496 8076 936 1 0 23 6 0 0 25 0 1 0 390709911 251957248 5974 33554432000 4194304 5683644 140735545757248 18446744073709551615 5307655 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/21834/statm: 61513 5974 1621 364 0 4515 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 246052

[startup+0.700641 s]
/proc/loadavg: 8.98 9.03 7.95 11/389 21834
/proc/meminfo: memFree=25052952/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=320228 CPUtime=0.48 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 12217 936 1 0 39 9 0 0 25 0 5 0 390709911 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21834/statm: 80057 7246 1873 364 0 15966 0
Current children cumulated CPU time (s) 0.48
Current children cumulated vsize (KiB) 320228

[startup+1.5015 s]
/proc/loadavg: 8.98 9.03 7.95 13/402 21851
/proc/meminfo: memFree=25020736/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=320228 CPUtime=0.48 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 12217 936 1 0 39 9 0 0 25 0 5 0 390709911 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21834/statm: 80057 7246 1873 364 0 15966 0
[pid=21840] ppid=21834 vsize=278608 CPUtime=0 cores=0,2,4,6
/proc/21840/stat : 21840 (python) S 21834 21834 21227 0 -1 4202560 733 0 0 0 0 0 0 0 25 0 1 0 390709968 285294592 5559 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682255 0 0 16781312 514 13668131578637713407 0 0 17 0 0 0 0
/proc/21840/statm: 69652 5559 354 364 0 5570 0
[pid=21844] ppid=21840 vsize=1017800 CPUtime=0.41 cores=0,2,4,6
/proc/21844/stat : 21844 (gnovelty+2) R 21840 21844 21844 0 -1 4202496 447 0 0 0 41 0 0 0 25 0 1 0 390709970 1042227200 182 33554432000 134512640 135663704 4287264816 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21844/statm: 254450 182 113 282 0 254166 0
[pid=21842] ppid=21834 vsize=289372 CPUtime=0 cores=0,2,4,6
/proc/21842/stat : 21842 (python) S 21834 21834 21227 0 -1 4202560 750 0 0 0 0 0 0 0 17 0 1 0 390709969 296316928 5701 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 7527582063491252483 0 0 17 2 0 0 0
/proc/21842/statm: 72343 5701 348 364 0 8258 0
[pid=21848] ppid=21842 vsize=2360 CPUtime=0.43 cores=0,2,4,6
/proc/21848/stat : 21848 (march_hi) R 21842 21848 21848 0 -1 4202496 657 0 0 0 43 0 0 0 25 0 1 0 390709972 2416640 393 33554432000 4194304 4955518 140735196650240 18446744073709551615 4227060 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21848/statm: 590 393 104 186 0 403 0
[pid=21845] ppid=21834 vsize=299120 CPUtime=0 cores=0,2,4,6
/proc/21845/stat : 21845 (python) S 21834 21834 21227 0 -1 4202560 741 0 0 0 0 0 0 0 25 0 1 0 390709971 306298880 5593 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 9728680372362805247 0 0 17 6 0 0 0
/proc/21845/statm: 74780 5593 348 364 0 10692 0
[pid=21850] ppid=21845 vsize=1017800 CPUtime=0.4 cores=0,2,4,6
/proc/21850/stat : 21850 (gnovelty+2) R 21845 21850 21850 0 -1 4202496 439 0 0 0 40 0 0 0 25 0 1 0 390709972 1042227200 182 33554432000 134512640 135663704 4294742288 18446744073709551615 134544183 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21850/statm: 254450 182 113 282 0 254166 0
[pid=21847] ppid=21834 vsize=309984 CPUtime=0 cores=0,2,4,6
/proc/21847/stat : 21847 (python) S 21834 21834 21227 0 -1 4202560 749 0 0 0 0 0 0 0 19 0 1 0 390709972 317423616 5717 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 5409601503592708 0 0 17 6 0 0 0
/proc/21847/statm: 77496 5717 348 364 0 13405 0
[pid=21851] ppid=21847 vsize=2564 CPUtime=0.87 cores=0,2,4,6
/proc/21851/stat : 21851 (march_hi) R 21847 21851 21851 0 -1 4202496 694 0 0 0 87 0 0 0 25 0 1 0 390709972 2625536 435 33554432000 4194304 4955518 140737197919136 18446744073709551615 4224646 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21851/statm: 641 435 104 186 0 454 0
[pid=21834/tid=21841] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21841/stat : 21841 (python) S 21832 21834 21227 0 -1 4202560 42 936 0 0 0 0 0 0 25 0 5 0 390709969 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21843] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21843/stat : 21843 (python) S 21832 21834 21227 0 -1 4202560 35 936 0 0 0 0 0 0 25 0 5 0 390709970 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21846] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21846/stat : 21846 (python) S 21832 21834 21227 0 -1 4202560 36 936 0 0 0 0 0 0 25 0 5 0 390709971 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21849] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21849/stat : 21849 (python) S 21832 21834 21227 0 -1 4202560 20 936 0 0 0 0 0 0 25 0 5 0 390709972 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2.59
Current children cumulated vsize (KiB) 3537836

[startup+3.10122 s]
/proc/loadavg: 8.98 9.03 7.95 12/402 21851
/proc/meminfo: memFree=25019000/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=320228 CPUtime=0.48 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 12217 936 1 0 39 9 0 0 25 0 5 0 390709911 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21834/statm: 80057 7246 1873 364 0 15966 0
[pid=21840] ppid=21834 vsize=278608 CPUtime=0 cores=0,2,4,6
/proc/21840/stat : 21840 (python) S 21834 21834 21227 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 390709968 285294592 5559 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682255 0 0 16781312 514 13668131578637680899 0 0 17 0 0 0 0
/proc/21840/statm: 69652 5559 354 364 0 5570 0
[pid=21844] ppid=21840 vsize=1017800 CPUtime=1.21 cores=0,2,4,6
/proc/21844/stat : 21844 (gnovelty+2) R 21840 21844 21844 0 -1 4202496 447 0 0 0 121 0 0 0 25 0 1 0 390709970 1042227200 182 33554432000 134512640 135663704 4287264816 18446744073709551615 134525298 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21844/statm: 254450 182 113 282 0 254166 0
[pid=21842] ppid=21834 vsize=289372 CPUtime=0 cores=0,2,4,6
/proc/21842/stat : 21842 (python) S 21834 21834 21227 0 -1 4202560 750 0 0 0 0 0 0 0 15 0 1 0 390709969 296316928 5701 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 7527582063491252483 0 0 17 2 0 0 0
/proc/21842/statm: 72343 5701 348 364 0 8258 0
[pid=21848] ppid=21842 vsize=2564 CPUtime=1.23 cores=0,2,4,6
/proc/21848/stat : 21848 (march_hi) R 21842 21848 21848 0 -1 4202496 715 0 0 0 123 0 0 0 25 0 1 0 390709972 2625536 451 33554432000 4194304 4955518 140735196650240 18446744073709551615 4227824 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21848/statm: 641 451 104 186 0 454 0
[pid=21845] ppid=21834 vsize=299120 CPUtime=0 cores=0,2,4,6
/proc/21845/stat : 21845 (python) S 21834 21834 21227 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 390709971 306298880 5593 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 9728680372362772738 0 0 17 6 0 0 0
/proc/21845/statm: 74780 5593 348 364 0 10692 0
[pid=21850] ppid=21845 vsize=1017800 CPUtime=1.2 cores=0,2,4,6
/proc/21850/stat : 21850 (gnovelty+2) R 21845 21850 21850 0 -1 4202496 439 0 0 0 120 0 0 0 25 0 1 0 390709972 1042227200 182 33554432000 134512640 135663704 4294742288 18446744073709551615 134544268 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21850/statm: 254450 182 113 282 0 254166 0
[pid=21847] ppid=21834 vsize=309984 CPUtime=0 cores=0,2,4,6
/proc/21847/stat : 21847 (python) S 21834 21834 21227 0 -1 4202560 751 0 0 0 0 0 0 0 15 0 1 0 390709972 317423616 5717 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 5409601503592708 0 0 17 6 0 0 0
/proc/21847/statm: 77496 5717 348 364 0 13405 0
[pid=21851] ppid=21847 vsize=2816 CPUtime=2.45 cores=0,2,4,6
/proc/21851/stat : 21851 (march_hi) R 21847 21851 21851 0 -1 4202496 754 0 0 0 245 0 0 0 25 0 1 0 390709972 2883584 495 33554432000 4194304 4955518 140737197919136 18446744073709551615 4224541 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21851/statm: 704 495 104 186 0 517 0
[pid=21834/tid=21841] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21841/stat : 21841 (python) S 21832 21834 21227 0 -1 4202560 42 936 0 0 0 0 0 0 25 0 5 0 390709969 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21843] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21843/stat : 21843 (python) S 21832 21834 21227 0 -1 4202560 35 936 0 0 0 0 0 0 25 0 5 0 390709970 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21846] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21846/stat : 21846 (python) S 21832 21834 21227 0 -1 4202560 36 936 0 0 0 0 0 0 25 0 5 0 390709971 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21849] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21849/stat : 21849 (python) S 21832 21834 21227 0 -1 4202560 20 936 0 0 0 0 0 0 25 0 5 0 390709972 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 6.57
Current children cumulated vsize (KiB) 3538292

[startup+6.30066 s]
/proc/loadavg: 9.14 9.06 7.97 12/405 21855
/proc/meminfo: memFree=25011436/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=320228 CPUtime=0.48 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 12217 936 1 0 39 9 0 0 25 0 5 0 390709911 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21834/statm: 80057 7246 1873 364 0 15966 0
[pid=21840] ppid=21834 vsize=278608 CPUtime=0.01 cores=0,2,4,6
/proc/21840/stat : 21840 (python) S 21834 21834 21227 0 -1 4202560 893 0 0 0 1 0 0 0 18 0 1 0 390709968 285294592 5581 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682255 0 0 16781312 514 13668131578637680899 0 0 17 0 0 0 0
/proc/21840/statm: 69652 5581 376 364 0 5570 0
[pid=21844] ppid=21840 vsize=1017800 CPUtime=2.8 cores=0,2,4,6
/proc/21844/stat : 21844 (gnovelty+2) R 21840 21844 21844 0 -1 4202496 447 0 0 0 280 0 0 0 25 0 1 0 390709970 1042227200 182 33554432000 134512640 135663704 4287264816 18446744073709551615 134545689 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21844/statm: 254450 182 113 282 0 254166 0
[pid=21842] ppid=21834 vsize=289372 CPUtime=0.01 cores=0,2,4,6
/proc/21842/stat : 21842 (python) S 21834 21834 21227 0 -1 4202560 925 0 0 0 0 1 0 0 15 0 1 0 390709969 296316928 5724 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 7527582063491252483 0 0 17 2 0 0 0
/proc/21842/statm: 72343 5724 371 364 0 8258 0
[pid=21848] ppid=21842 vsize=2816 CPUtime=2.81 cores=0,2,4,6
/proc/21848/stat : 21848 (march_hi) R 21842 21848 21848 0 -1 4202496 773 0 0 0 281 0 0 0 25 0 1 0 390709972 2883584 509 33554432000 4194304 4955518 140735196650240 18446744073709551615 4346309 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/21848/statm: 704 509 104 186 0 517 0
[pid=21845] ppid=21834 vsize=299120 CPUtime=0 cores=0,2,4,6
/proc/21845/stat : 21845 (python) S 21834 21834 21227 0 -1 4202560 899 0 0 0 0 0 0 0 18 0 1 0 390709971 306298880 5615 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 9728680372362772738 0 0 17 6 0 0 0
/proc/21845/statm: 74780 5615 370 364 0 10692 0
[pid=21850] ppid=21845 vsize=1017800 CPUtime=2.8 cores=0,2,4,6
/proc/21850/stat : 21850 (gnovelty+2) R 21845 21850 21850 0 -1 4202496 439 0 0 0 280 0 0 0 25 0 1 0 390709972 1042227200 182 33554432000 134512640 135663704 4294742288 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/21850/statm: 254450 182 113 282 0 254166 0
[pid=21847] ppid=21834 vsize=309984 CPUtime=0.01 cores=0,2,4,6
/proc/21847/stat : 21847 (python) S 21834 21834 21227 0 -1 4202560 910 0 0 0 1 0 0 0 15 0 1 0 390709972 317423616 5740 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 17466167447638737156 0 0 17 6 0 0 0
/proc/21847/statm: 77496 5740 371 364 0 13405 0
[pid=21851] ppid=21847 vsize=3596 CPUtime=5.57 cores=0,2,4,6
/proc/21851/stat : 21851 (march_hi) R 21847 21851 21851 0 -1 4202496 881 0 0 0 557 0 0 0 25 0 1 0 390709972 3682304 622 33554432000 4194304 4955518 140737197919136 18446744073709551615 4224651 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21851/statm: 899 622 104 186 0 712 0
[pid=21834/tid=21841] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21841/stat : 21841 (python) S 21832 21834 21227 0 -1 4202560 42 936 0 0 0 0 0 0 25 0 5 0 390709969 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21843] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21843/stat : 21843 (python) S 21832 21834 21227 0 -1 4202560 35 936 0 0 0 0 0 0 25 0 5 0 390709970 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21846] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21846/stat : 21846 (python) S 21832 21834 21227 0 -1 4202560 36 936 0 0 0 0 0 0 25 0 5 0 390709971 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21849] ppid=21832 vsize=320228 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21849/stat : 21849 (python) S 21832 21834 21227 0 -1 4202560 20 936 0 0 0 0 0 0 25 0 5 0 390709972 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 14.49
Current children cumulated vsize (KiB) 3539324

[startup+12.7005 s]
/proc/loadavg: 9.29 9.10 7.98 12/405 21855
/proc/meminfo: memFree=25006024/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=320228 CPUtime=0.48 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 12217 936 1 0 39 9 0 0 25 0 5 0 390709911 327913472 7246 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21834/statm: 80057 7246 1873 364 0 15966 0
[pid=21840] ppid=21834 vsize=278608 CPUtime=0.03 cores=0,2,4,6
/proc/21840/stat : 21840 (python) S 21834 21834 21227 0 -1 4202560 896 0 0 0 2 1 0 0 18 0 1 0 390709968 285294592 5581 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682255 0 0 16781312 514 6884508494435680515 0 0 17 0 0 0 0
/proc/21840/statm: 69652 5581 376 364 0 5570 0
[pid=21844] ppid=21840 vsize=1017800 CPUtime=6 cores=0,2,4,6
/proc/21844/stat : 21844 (gnovelty+2) R 21840 21844 21844 0 -1 4202496 447 0 0 0 600 0 0 0 25 0 1 0 390709970 1042227200 182 33554432000 134512640 135663704 4287264816 18446744073709551615 134525347 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21844/statm: 254450 182 113 282 0 254166 0
[pid=21842] ppid=21834 vsize=289372 CPUtime=0.03 cores=0,2,4,6

################
# More data... #
################

[pid=21863] ppid=21834 vsize=320240 CPUtime=2.59 cores=0,2,4,6
/proc/21863/stat : 21863 (python) S 21834 21834 21227 0 -1 4202560 1142 0 0 0 140 119 0 0 17 0 2 0 390724811 327925760 5774 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21863/statm: 80060 5774 400 364 0 15966 0
[pid=21865] ppid=21863 vsize=122980 CPUtime=679.48 cores=0,2,4,6
/proc/21865/stat : 21865 (march_hi) T 21863 21865 21865 0 -1 4202496 21445 0 0 0 67935 13 0 0 25 0 1 0 390724812 125931520 21184 33554432000 4194304 4955518 140737196148608 18446744073709551615 4238717 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/21865/statm: 30745 21184 104 186 0 30558 0
[pid=21863/tid=22185] ppid=21834 vsize=320240 CPUtime=0 cores=0,2,4,6
/proc/21863/task/22185/stat : 22185 (python) S 21834 21834 21227 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 390810497 327925760 5774 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 514 18446604449651122304 0 0 -1 0 0 0 0
[pid=21869] ppid=21834 vsize=329972 CPUtime=2.9 cores=0,2,4,6
/proc/21869/stat : 21869 (python) S 21834 21834 21227 0 -1 4202560 995 0 0 0 148 142 0 0 15 0 1 0 390729540 337891328 5649 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 16893178128422109442 0 0 17 2 0 0 0
/proc/21869/statm: 82493 5649 371 364 0 18396 0
[pid=21871] ppid=21869 vsize=107044 CPUtime=647.45 cores=0,2,4,6
/proc/21871/stat : 21871 (march_hi) R 21869 21871 21871 0 -1 4202496 20466 0 0 0 64735 10 0 0 25 0 1 0 390729541 109613056 20208 33554432000 4194304 4955518 140735991226320 18446744073709551615 4235379 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21871/statm: 26761 20208 104 186 0 26574 0
[pid=21890] ppid=21834 vsize=340852 CPUtime=1.93 cores=0,2,4,6
/proc/21890/stat : 21890 (python) S 21834 21834 21227 0 -1 4202560 989 0 0 0 101 92 0 0 18 0 1 0 390759433 349032448 5777 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 6884508494435680516 0 0 17 0 0 0 0
/proc/21890/statm: 85213 5777 371 364 0 21113 0
[pid=21892] ppid=21890 vsize=182448 CPUtime=583.97 cores=0,2,4,6
/proc/21892/stat : 21892 (kcnfs-2006) R 21890 21892 21892 0 -1 4202496 609 0 0 0 58395 2 0 0 25 0 1 0 390759433 186826752 350 33554432000 134512640 135163436 4293806864 18446744073709551615 134572242 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21892/statm: 45612 350 77 159 0 45452 0
[pid=22165] ppid=21834 vsize=351108 CPUtime=1.58 cores=0,2,4,6
/proc/22165/stat : 22165 (python) S 21834 21834 21227 0 -1 4202560 1052 0 0 0 81 77 0 0 15 0 1 0 390774294 359534592 5809 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 6884508494435680515 0 0 17 2 0 0 0
/proc/22165/statm: 87777 5809 377 364 0 23674 0
[pid=22167] ppid=22165 vsize=77576 CPUtime=487.72 cores=0,2,4,6
/proc/22167/stat : 22167 (march_hi) R 22165 22167 22167 0 -1 4202496 15999 0 0 0 48766 6 0 0 25 0 1 0 390774294 79437824 15720 33554432000 4194304 4955518 140736360017472 18446744073709551615 4224512 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22167/statm: 19394 15720 104 186 0 19207 0
[pid=22186] ppid=21834 vsize=361552 CPUtime=0.49 cores=0,2,4,6
/proc/22186/stat : 22186 (python) S 21834 21834 21227 0 -1 4202560 1012 0 0 0 25 24 0 0 15 0 1 0 390810498 370229248 5838 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 6884508494435680514 0 0 17 2 0 0 0
/proc/22186/statm: 90388 5838 377 364 0 26282 0
[pid=22188] ppid=22186 vsize=16208 CPUtime=80.31 cores=0,2,4,6
/proc/22188/stat : 22188 (march_hi) R 22186 22188 22188 0 -1 4202496 3549 0 0 0 8031 0 0 0 25 0 1 0 390810498 16596992 3283 33554432000 4194304 4955518 140737458375600 18446744073709551615 4224682 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22188/statm: 4052 3283 104 186 0 3865 0
[pid=21834/tid=21841] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21841/stat : 21841 (python) S 21832 21834 21227 0 -1 4202560 42 936 0 0 0 0 0 0 25 0 10 0 390709969 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21843] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21843/stat : 21843 (python) S 21832 21834 21227 0 -1 4202560 35 936 0 0 0 0 0 0 25 0 10 0 390709970 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21846] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21846/stat : 21846 (python) S 21832 21834 21227 0 -1 4202560 36 936 0 0 0 0 0 0 25 0 10 0 390709971 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21849] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21849/stat : 21849 (python) S 21832 21834 21227 0 -1 4202560 20 936 0 0 0 0 0 0 25 0 10 0 390709972 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21864] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21864/stat : 21864 (python) S 21832 21834 21227 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 10 0 390724811 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21870] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21870/stat : 21870 (python) S 21832 21834 21227 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 10 0 390729540 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21891] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21891/stat : 21891 (python) S 21832 21834 21227 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 10 0 390759433 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=22166] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/22166/stat : 22166 (python) S 21832 21834 21227 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 10 0 390774294 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=22187] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/22187/stat : 22187 (python) S 21832 21834 21227 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 10 0 390810498 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3470.6
Current children cumulated vsize (KiB) 5959592

[startup+1166.9 s]
/proc/loadavg: 9.18 9.43 9.18 10/424 22193
/proc/meminfo: memFree=24327356/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=371796 CPUtime=0.52 cores=0,2,4,6
/proc/21834/stat : 21834 (python) S 21832 21834 21227 0 -1 4202496 16301 936 1 0 43 9 0 0 20 0 10 0 390709911 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/21834/statm: 92949 7349 1884 364 0 28843 0
[pid=21840] ppid=21834 vsize=288852 CPUtime=0.41 cores=0,2,4,6
/proc/21840/stat : 21840 (python) S 21834 21834 21227 0 -1 4202560 1179 0 0 0 22 19 0 0 19 0 2 0 390709968 295784448 5626 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21840/statm: 72213 5626 416 364 0 8131 0
[pid=21844] ppid=21840 vsize=1017800 CPUtime=102.47 cores=0,2,4,6
/proc/21844/stat : 21844 (gnovelty+2) T 21840 21844 21844 0 -1 4202496 447 0 0 0 10247 0 0 0 25 0 1 0 390709970 1042227200 182 33554432000 134512640 135663704 4287264816 18446744073709551615 134524843 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/21844/statm: 254450 182 113 282 0 254166 0
[pid=21840/tid=21862] ppid=21834 vsize=288852 CPUtime=0 cores=0,2,4,6
/proc/21840/task/21862/stat : 21862 (python) S 21834 21834 21227 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 390724810 295784448 5626 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 514 18446604443210151232 0 0 -1 0 0 0 0
[pid=21842] ppid=21834 vsize=289372 CPUtime=1.93 cores=0,2,4,6
/proc/21842/stat : 21842 (python) S 21834 21834 21227 0 -1 4202560 1159 0 0 0 108 85 0 0 17 0 2 0 390709969 296316928 5763 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/21842/statm: 72343 5763 409 364 0 8258 0
[pid=21848] ppid=21842 vsize=77444 CPUtime=388.85 cores=0,2,4,6
/proc/21848/stat : 21848 (march_hi) T 21842 21848 21848 0 -1 4202496 12971 0 0 0 38880 5 0 0 25 0 1 0 390709972 79302656 12707 33554432000 4194304 4955518 140735196650240 18446744073709551615 4207095 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/21848/statm: 19361 12707 104 186 0 19174 0
[pid=21842/tid=22164] ppid=21834 vsize=289372 CPUtime=0 cores=0,2,4,6
/proc/21842/task/22164/stat : 22164 (python) S 21834 21834 21227 0 -1 4202560 44 0 0 0 0 0 0 0 17 0 2 0 390774293 296316928 5763 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 514 18446604453927368704 0 0 -1 2 0 0 0
[pid=21845] ppid=21834 vsize=299120 CPUtime=0.55 cores=0,2,4,6
/proc/21845/stat : 21845 (python) S 21834 21834 21227 0 -1 4202560 1117 0 0 0 28 27 0 0 19 0 2 0 390709971 306298880 5646 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/21845/statm: 74780 5646 400 364 0 10692 0
[pid=21850] ppid=21845 vsize=1017800 CPUtime=97.79 cores=0,2,4,6
/proc/21850/stat : 21850 (gnovelty+2) T 21845 21850 21850 0 -1 4202496 439 0 0 0 9779 0 0 0 25 0 1 0 390709972 1042227200 182 33554432000 134512640 135663704 4294742288 18446744073709551615 134525334 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/21850/statm: 254450 182 113 282 0 254166 0
[pid=21845/tid=21868] ppid=21834 vsize=299120 CPUtime=0 cores=0,2,4,6
/proc/21845/task/21868/stat : 21868 (python) S 21834 21834 21227 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 390729539 306298880 5646 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 514 18446604450882860864 0 0 -1 6 0 0 0
[pid=21847] ppid=21834 vsize=309984 CPUtime=1.5 cores=0,2,4,6
/proc/21847/stat : 21847 (python) S 21834 21834 21227 0 -1 4202560 1167 0 0 0 82 68 0 0 16 0 2 0 390709972 317423616 5770 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/21847/statm: 77496 5770 400 364 0 13405 0
[pid=21851] ppid=21847 vsize=77444 CPUtime=388.16 cores=0,2,4,6
/proc/21851/stat : 21851 (march_hi) T 21847 21851 21851 0 -1 4202496 12928 0 0 0 38811 5 0 0 25 0 1 0 390709972 79302656 12669 33554432000 4194304 4955518 140737197919136 18446744073709551615 4209717 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/21851/statm: 19361 12669 104 186 0 19174 0
[pid=21847/tid=21889] ppid=21834 vsize=309984 CPUtime=0 cores=0,2,4,6
/proc/21847/task/21889/stat : 21889 (python) S 21834 21834 21227 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 390759432 317423616 5770 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 514 18446604452684384960 0 0 -1 6 0 0 0
[pid=21863] ppid=21834 vsize=320240 CPUtime=2.59 cores=0,2,4,6
/proc/21863/stat : 21863 (python) S 21834 21834 21227 0 -1 4202560 1142 0 0 0 140 119 0 0 17 0 2 0 390724811 327925760 5774 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672488907 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/21863/statm: 80060 5774 400 364 0 15966 0
[pid=21865] ppid=21863 vsize=122980 CPUtime=679.48 cores=0,2,4,6
/proc/21865/stat : 21865 (march_hi) T 21863 21865 21865 0 -1 4202496 21445 0 0 0 67935 13 0 0 25 0 1 0 390724812 125931520 21184 33554432000 4194304 4955518 140737196148608 18446744073709551615 4238717 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/21865/statm: 30745 21184 104 186 0 30558 0
[pid=21863/tid=22185] ppid=21834 vsize=320240 CPUtime=0 cores=0,2,4,6
/proc/21863/task/22185/stat : 22185 (python) S 21834 21834 21227 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 390810497 327925760 5774 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 514 18446604449651122304 0 0 -1 0 0 0 0
[pid=21869] ppid=21834 vsize=329972 CPUtime=2.9 cores=0,2,4,6
/proc/21869/stat : 21869 (python) S 21834 21834 21227 0 -1 4202560 995 0 0 0 148 142 0 0 15 0 1 0 390729540 337891328 5649 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 16893178128422109442 0 0 17 2 0 0 0
/proc/21869/statm: 82493 5649 371 364 0 18396 0
[pid=21871] ppid=21869 vsize=107044 CPUtime=647.85 cores=0,2,4,6
/proc/21871/stat : 21871 (march_hi) R 21869 21871 21871 0 -1 4202496 20477 0 0 0 64775 10 0 0 25 0 1 0 390729541 109613056 20219 33554432000 4194304 4955518 140735991226320 18446744073709551615 4209747 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/21871/statm: 26761 20219 104 186 0 26574 0
[pid=21890] ppid=21834 vsize=340852 CPUtime=1.93 cores=0,2,4,6
/proc/21890/stat : 21890 (python) S 21834 21834 21227 0 -1 4202560 989 0 0 0 101 92 0 0 18 0 1 0 390759433 349032448 5777 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 6884508494435680516 0 0 17 0 0 0 0
/proc/21890/statm: 85213 5777 371 364 0 21113 0
[pid=21892] ppid=21890 vsize=182448 CPUtime=584.37 cores=0,2,4,6
/proc/21892/stat : 21892 (kcnfs-2006) R 21890 21892 21892 0 -1 4202496 609 0 0 0 58435 2 0 0 25 0 1 0 390759433 186826752 350 33554432000 134512640 135163436 4293806864 18446744073709551615 134675205 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/21892/statm: 45612 350 77 159 0 45452 0
[pid=22165] ppid=21834 vsize=351108 CPUtime=1.58 cores=0,2,4,6
/proc/22165/stat : 22165 (python) S 21834 21834 21227 0 -1 4202560 1052 0 0 0 81 77 0 0 15 0 1 0 390774294 359534592 5809 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 6884508494435680515 0 0 17 2 0 0 0
/proc/22165/statm: 87777 5809 377 364 0 23674 0
[pid=22167] ppid=22165 vsize=77576 CPUtime=488.12 cores=0,2,4,6
/proc/22167/stat : 22167 (march_hi) R 22165 22167 22167 0 -1 4202496 16011 0 0 0 48806 6 0 0 25 0 1 0 390774294 79437824 15732 33554432000 4194304 4955518 140736360017472 18446744073709551615 4209733 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/22167/statm: 19394 15732 104 186 0 19207 0
[pid=22186] ppid=21834 vsize=361552 CPUtime=0.49 cores=0,2,4,6
/proc/22186/stat : 22186 (python) S 21834 21834 21227 0 -1 4202560 1012 0 0 0 25 24 0 0 15 0 1 0 390810498 370229248 5838 33554432000 4194304 5683644 140735545757248 18446744073709551615 268660682310 0 0 16781312 514 6884508494435680514 0 0 17 2 0 0 0
/proc/22186/statm: 90388 5838 377 364 0 26282 0
[pid=22188] ppid=22186 vsize=16208 CPUtime=80.51 cores=0,2,4,6
/proc/22188/stat : 22188 (march_hi) R 22186 22188 22188 0 -1 4202496 3555 0 0 0 8051 0 0 0 25 0 1 0 390810498 16596992 3289 33554432000 4194304 4955518 140737458375600 18446744073709551615 4209643 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/22188/statm: 4052 3289 104 186 0 3865 0
[pid=21834/tid=21841] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21841/stat : 21841 (python) S 21832 21834 21227 0 -1 4202560 42 936 0 0 0 0 0 0 25 0 10 0 390709969 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21843] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21843/stat : 21843 (python) S 21832 21834 21227 0 -1 4202560 35 936 0 0 0 0 0 0 25 0 10 0 390709970 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21846] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21846/stat : 21846 (python) S 21832 21834 21227 0 -1 4202560 36 936 0 0 0 0 0 0 25 0 10 0 390709971 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21849] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21849/stat : 21849 (python) S 21832 21834 21227 0 -1 4202560 20 936 0 0 0 0 0 0 25 0 10 0 390709972 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21864] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21864/stat : 21864 (python) S 21832 21834 21227 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 10 0 390724811 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21870] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21870/stat : 21870 (python) S 21832 21834 21227 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 10 0 390729540 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21891] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/21891/stat : 21891 (python) S 21832 21834 21227 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 10 0 390759433 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=22166] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/22166/stat : 22166 (python) S 21832 21834 21227 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 10 0 390774294 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=22187] ppid=21832 vsize=371796 CPUtime=0 cores=0,2,4,6
/proc/21834/task/22187/stat : 22187 (python) S 21832 21834 21227 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 10 0 390810498 380719104 7349 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 2 18446604441973581952 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3472
Current children cumulated vsize (KiB) 5959592

[startup+1167.32 s]
/proc/loadavg: 9.16 9.42 9.18 10/424 22193
/proc/meminfo: memFree=24326240/32951124 swapFree=46575252/67111528
[pid=21834] ppid=21832 vsize=371696 CPUtime=3473.55 cores=0,2,4,6
/proc/21834/stat : 21834 (python) R 21832 21834 21227 0 -1 4202496 18696 100104 1 0 45 10 346594 706 17 0 10 0 390709911 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 5114052 0 0 16781312 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21834/statm: 92924 7334 1869 364 0 28843 0
[pid=21834/tid=21841] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21841/stat : 21841 (python) S 21832 21834 21227 0 -1 4202560 44 100104 0 0 0 0 346594 706 18 0 10 0 390709969 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 0 0 0 0
[pid=21834/tid=21843] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21843/stat : 21843 (python) S 21832 21834 21227 0 -1 4202560 39 100104 0 0 0 0 346594 706 18 0 10 0 390709970 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 0 0 0 0
[pid=21834/tid=21846] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21846/stat : 21846 (python) S 21832 21834 21227 0 -1 4202560 40 100104 0 0 0 0 346594 706 18 0 10 0 390709971 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 2 0 0 0
[pid=21834/tid=21849] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21849/stat : 21849 (python) S 21832 21834 21227 0 -1 4202560 24 100104 0 0 0 0 346594 706 18 0 10 0 390709972 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 2 0 0 0
[pid=21834/tid=21864] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21864/stat : 21864 (python) S 21832 21834 21227 0 -1 4202560 32 100104 0 0 0 0 346594 706 18 0 10 0 390724811 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=21870] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21870/stat : 21870 (python) S 21832 21834 21227 0 -1 4202560 29 100104 0 0 0 0 346594 706 18 0 10 0 390729540 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 0 0 0 0
[pid=21834/tid=21891] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/21891/stat : 21891 (python) S 21832 21834 21227 0 -1 4202560 31 100104 0 0 0 0 346594 706 18 0 10 0 390759433 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 0 0 0 0
[pid=21834/tid=22166] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/22166/stat : 22166 (python) S 21832 21834 21227 0 -1 4202560 27 100104 0 0 0 0 346594 706 18 0 10 0 390774294 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 6 0 0 0
[pid=21834/tid=22187] ppid=21832 vsize=371696 CPUtime=3473 cores=0,2,4,6
/proc/21834/task/22187/stat : 22187 (python) S 21832 21834 21227 0 -1 4202560 27 100104 0 0 0 0 346594 706 18 0 10 0 390810498 380616704 7334 33554432000 4194304 5683644 140735545757248 18446744073709551615 268672485809 0 0 16781312 0 18446604441973581952 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3473.55
Current children cumulated vsize (KiB) 371696

Child status: 0
Real time (s): 1167.33
CPU time (s): 3473.59
CPU user time (s): 3466.41
CPU system time (s): 7.17491
CPU usage (%): 297.566
Max. virtual memory (cumulated for all children) (KiB): 5959596

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3466.41
system time used= 7.17491
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 121017
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7642
involuntary context switches= 37527

runsolver used 7.05893 second user time and 11.9772 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-15 15:13:06
IDJOB=3378382
IDBENCH=82791
IDSOLVER=1859
FILE ID=node107/3378382-1305465186
RUNJOBID= node107-1305463373-21245
PBS_JOBID= 13324996
Free space on /tmp= 73428 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v550-c2343-S1663503036-045.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378382-1305465186/watcher-3378382-1305465186 -o /tmp/evaluation-result-3378382-1305465186/solver-3378382-1305465186 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378382-1305465186.cnf -seed 106280113 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8c477af50318fe10cbec1ad215bff97c
RANDOM SEED=106280113

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25060424 kB
Buffers:        186088 kB
Cached:        1464800 kB
SwapCached:       9832 kB
Active:        7061964 kB
Inactive:       527724 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25060424 kB
SwapTotal:    67111528 kB
SwapFree:     46575252 kB
Dirty:           34008 kB
Writeback:           0 kB
AnonPages:     5936376 kB
Mapped:          31036 kB
Slab:           153016 kB
PageTables:      81588 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 92412424 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73396 MiB
End job on node107 at 2011-05-15 15:32:34