Trace number 3378637

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 4772.71 1198.42

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v650-c2769-S2065143904-035.UNKNOWN.cnf
MD5SUM8f7328be14ce714cebc5a66418b5924c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables650
Number of clauses2769
Sum of the clauses size8307
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32769
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.30/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.46	c INFO: solving HOME/instance-3378637-1305466532.cnf
0.30/0.46	c INFO: computing task features
0.30/0.47	c INFO: computed variable-clause graph statistics
0.30/0.48	c INFO: computed clause balance statistics
0.38/0.52	c INFO: computed variable balance statistics
0.38/0.56	c INFO: computed small-clause counts
0.38/0.56	c INFO: computed Horn-clause counts
0.38/0.57	c INFO: computed variable graph statistics
0.38/0.57	c INFO: collected features for HOME/instance-3378637-1305466532.cnf in 0.10 s
0.38/0.57	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.58)
0.38/0.58	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.28)
0.38/0.59	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.47/0.60	c INFO: running march_hi@0 for 679 with 1199 remaining (b = 0.37)
396.02/99.90	c INFO: running march_hi@0 for 679 with 1100 remaining (b = 0.33)
400.01/100.85	c INFO: running march_hi@0 for 679 with 1099 remaining (b = 0.29)
1581.07/397.08	c INFO: running kcnfs-2006@0 for 582 with 803 remaining (b = 0.25)
2762.43/693.33	c INFO: running march_hi@0 for 507 with 507 remaining (b = 0.20)
3122.45/783.61	c INFO: running march_hi@0 for 416 with 416 remaining (b = 0.15)
3127.21/784.88	c INFO: running march_hi@0 for 291 with 415 remaining (b = 0.10)
3918.41/983.74	c INFO: running march_hi@393 for 97 with 216 remaining (b = 0.04)
4278.83/1074.32	c INFO: running march_hi@586 for 126 with 126 remaining (b = 0.04)
4313.45/1083.04	c INFO: running kcnfs-2006@683 for 117 with 117 remaining (b = 0.04)
4772.67/1198.40	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378637-1305466532/watcher-3378637-1305466532 -o /tmp/evaluation-result-3378637-1305466532/solver-3378637-1305466532 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378637-1305466532.cnf -seed 108187587 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.18 8.02 6.12 6/385 6252
/proc/meminfo: memFree=23664448/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=15780 CPUtime=0 cores=0,2,4,6
/proc/6252/stat : 6252 (python) R 6250 6252 5751 0 -1 4202496 897 0 0 0 0 0 0 0 25 0 1 0 658253654 16158720 779 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446988423 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6252/statm: 3945 779 343 364 0 432 0

[startup+0.0739111 s]
/proc/loadavg: 8.18 8.02 6.12 6/385 6252
/proc/meminfo: memFree=23664448/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=84528 CPUtime=0.05 cores=0,2,4,6
/proc/6252/stat : 6252 (python) R 6250 6252 5751 0 -1 4202496 2249 0 0 0 4 1 0 0 25 0 1 0 658253654 86556672 1998 33554432000 4194304 5683644 140735013828944 18446744073709551615 244441976794 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6252/statm: 21132 1999 618 364 0 1336 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 84528

[startup+0.100889 s]
/proc/loadavg: 8.18 8.02 6.12 6/385 6252
/proc/meminfo: memFree=23664448/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=104228 CPUtime=0.06 cores=0,2,4,6
/proc/6252/stat : 6252 (python) R 6250 6252 5751 0 -1 4202496 3067 0 0 0 5 1 0 0 25 0 1 0 658253654 106729472 2749 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446943232 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6252/statm: 26057 2751 847 364 0 1805 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104228

[startup+0.300868 s]
/proc/loadavg: 8.18 8.02 6.12 6/385 6252
/proc/meminfo: memFree=23664448/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=187744 CPUtime=0.22 cores=0,2,4,6
/proc/6252/stat : 6252 (python) R 6250 6252 5751 0 -1 4202496 6798 937 1 0 17 5 0 0 17 0 1 0 658253654 192249856 4863 33554432000 4194304 5683644 140735013828944 18446744073709551615 244441982869 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6252/statm: 46936 4864 1383 364 0 3724 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 187744

[startup+0.700752 s]
/proc/loadavg: 8.18 8.02 6.12 6/385 6252
/proc/meminfo: memFree=23664448/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=320264 CPUtime=0.49 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 12204 937 1 0 41 8 0 0 24 0 5 0 658253654 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6252/statm: 80066 7254 1873 364 0 15975 0
Current children cumulated CPU time (s) 0.49
Current children cumulated vsize (KiB) 320264

[startup+1.5006 s]
/proc/loadavg: 8.18 8.02 6.12 10/398 6269
/proc/meminfo: memFree=23624308/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=320264 CPUtime=0.49 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 12204 937 1 0 41 8 0 0 24 0 5 0 658253654 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6252/statm: 80066 7254 1873 364 0 15975 0
[pid=6258] ppid=6252 vsize=278644 CPUtime=0 cores=0,2,4,6
/proc/6258/stat : 6258 (python) S 6252 6252 5751 0 -1 4202560 726 0 0 0 0 0 0 0 18 0 1 0 658253712 285331456 5567 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965263 0 0 16781312 514 3022198203626815487 0 0 17 0 0 0 0
/proc/6258/statm: 69661 5567 354 364 0 5579 0
[pid=6260] ppid=6258 vsize=1017940 CPUtime=0.91 cores=0,2,4,6
/proc/6260/stat : 6260 (gnovelty+2) R 6258 6260 6260 0 -1 4202496 453 0 0 0 91 0 0 0 21 0 1 0 658253712 1042370560 191 33554432000 134512640 135663704 4286767808 18446744073709551615 134544259 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6260/statm: 254485 191 113 282 0 254201 0
[pid=6261] ppid=6252 vsize=289408 CPUtime=0 cores=0,2,4,6
/proc/6261/stat : 6261 (python) S 6252 6252 5751 0 -1 4202560 751 0 0 0 0 0 0 0 15 0 1 0 658253713 296353792 5709 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 6055516313806143745 0 0 17 0 0 0 0
/proc/6261/statm: 72352 5709 348 364 0 8267 0
[pid=6263] ppid=6261 vsize=2728 CPUtime=0.9 cores=0,2,4,6
/proc/6263/stat : 6263 (march_hi) R 6261 6263 6263 0 -1 4202496 751 0 0 0 90 0 0 0 23 0 1 0 658253713 2793472 485 33554432000 4194304 4955518 140735642963792 18446744073709551615 4209740 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6263/statm: 682 485 104 186 0 495 0
[pid=6264] ppid=6252 vsize=299156 CPUtime=0 cores=0,2,4,6
/proc/6264/stat : 6264 (python) S 6252 6252 5751 0 -1 4202560 744 0 0 0 0 0 0 0 21 0 1 0 658253714 306335744 5601 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 4606912767868272639 0 0 17 6 0 0 0
/proc/6264/statm: 74789 5601 348 364 0 10701 0
[pid=6266] ppid=6264 vsize=1017940 CPUtime=0.88 cores=0,2,4,6
/proc/6266/stat : 6266 (gnovelty+2) R 6264 6266 6266 0 -1 4202496 455 0 0 0 88 0 0 0 25 0 1 0 658253714 1042370560 192 33554432000 134512640 135663704 4288933488 18446744073709551615 134524711 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/6266/statm: 254485 192 113 282 0 254201 0
[pid=6267] ppid=6252 vsize=310020 CPUtime=0 cores=0,2,4,6
/proc/6267/stat : 6267 (python) S 6252 6252 5751 0 -1 4202560 757 0 0 0 0 0 0 0 17 0 1 0 658253715 317460480 5725 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 17304399875256844545 0 0 17 2 0 0 0
/proc/6267/statm: 77505 5725 348 364 0 13414 0
[pid=6269] ppid=6267 vsize=2728 CPUtime=0.88 cores=0,2,4,6
/proc/6269/stat : 6269 (march_hi) R 6267 6269 6269 0 -1 4202496 748 0 0 0 88 0 0 0 25 0 1 0 658253715 2793472 484 33554432000 4194304 4955518 140735569719696 18446744073709551615 4227671 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6269/statm: 682 484 104 186 0 495 0
[pid=6252/tid=6259] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6259/stat : 6259 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 19 0 5 0 658253712 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 6 0 0 0
[pid=6252/tid=6262] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6262/stat : 6262 (python) S 6250 6252 5751 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 5 0 658253713 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6265] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6265/stat : 6265 (python) S 6250 6252 5751 0 -1 4202560 36 937 0 0 0 0 0 0 22 0 5 0 658253714 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6268] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6268/stat : 6268 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 24 0 5 0 658253715 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.06
Current children cumulated vsize (KiB) 3538828

[startup+3.1013 s]
/proc/loadavg: 8.18 8.02 6.12 10/398 6269
/proc/meminfo: memFree=23622572/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=320264 CPUtime=0.49 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 12204 937 1 0 41 8 0 0 24 0 5 0 658253654 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6252/statm: 80066 7254 1873 364 0 15975 0
[pid=6258] ppid=6252 vsize=278644 CPUtime=0 cores=0,2,4,6
/proc/6258/stat : 6258 (python) S 6252 6252 5751 0 -1 4202560 726 0 0 0 0 0 0 0 18 0 1 0 658253712 285331456 5567 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965263 0 0 16781312 514 3022198203626782978 0 0 17 0 0 0 0
/proc/6258/statm: 69661 5567 354 364 0 5579 0
[pid=6260] ppid=6258 vsize=1017940 CPUtime=2.51 cores=0,2,4,6
/proc/6260/stat : 6260 (gnovelty+2) R 6258 6260 6260 0 -1 4202496 453 0 0 0 251 0 0 0 25 0 1 0 658253712 1042370560 191 33554432000 134512640 135663704 4286767808 18446744073709551615 134545786 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6260/statm: 254485 191 113 282 0 254201 0
[pid=6261] ppid=6252 vsize=289408 CPUtime=0 cores=0,2,4,6
/proc/6261/stat : 6261 (python) S 6252 6252 5751 0 -1 4202560 751 0 0 0 0 0 0 0 15 0 1 0 658253713 296353792 5709 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 6055516313806143745 0 0 17 0 0 0 0
/proc/6261/statm: 72352 5709 348 364 0 8267 0
[pid=6263] ppid=6261 vsize=3164 CPUtime=2.5 cores=0,2,4,6
/proc/6263/stat : 6263 (march_hi) R 6261 6263 6263 0 -1 4202496 820 0 0 0 250 0 0 0 25 0 1 0 658253713 3239936 554 33554432000 4194304 4955518 140735642963792 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6263/statm: 791 554 104 186 0 604 0
[pid=6264] ppid=6252 vsize=299156 CPUtime=0 cores=0,2,4,6
/proc/6264/stat : 6264 (python) S 6252 6252 5751 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 658253714 306335744 5601 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 4606912767868240129 0 0 17 6 0 0 0
/proc/6264/statm: 74789 5601 348 364 0 10701 0
[pid=6266] ppid=6264 vsize=1017940 CPUtime=2.47 cores=0,2,4,6
/proc/6266/stat : 6266 (gnovelty+2) R 6264 6266 6266 0 -1 4202496 455 0 0 0 247 0 0 0 25 0 1 0 658253714 1042370560 192 33554432000 134512640 135663704 4288933488 18446744073709551615 134525162 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/6266/statm: 254485 192 113 282 0 254201 0
[pid=6267] ppid=6252 vsize=310020 CPUtime=0 cores=0,2,4,6
/proc/6267/stat : 6267 (python) S 6252 6252 5751 0 -1 4202560 757 0 0 0 0 0 0 0 15 0 1 0 658253715 317460480 5725 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 17304399875256844545 0 0 17 2 0 0 0
/proc/6267/statm: 77505 5725 348 364 0 13414 0
[pid=6269] ppid=6267 vsize=3164 CPUtime=2.48 cores=0,2,4,6
/proc/6269/stat : 6269 (march_hi) R 6267 6269 6269 0 -1 4202496 817 0 0 0 248 0 0 0 25 0 1 0 658253715 3239936 553 33554432000 4194304 4955518 140735569719696 18446744073709551615 4224628 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6269/statm: 791 553 104 186 0 604 0
[pid=6252/tid=6259] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6259/stat : 6259 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 19 0 5 0 658253712 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 6 0 0 0
[pid=6252/tid=6262] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6262/stat : 6262 (python) S 6250 6252 5751 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 5 0 658253713 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6265] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6265/stat : 6265 (python) S 6250 6252 5751 0 -1 4202560 36 937 0 0 0 0 0 0 22 0 5 0 658253714 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6268] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6268/stat : 6268 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 24 0 5 0 658253715 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 10.45
Current children cumulated vsize (KiB) 3539700

[startup+6.30071 s]
/proc/loadavg: 8.25 8.04 6.14 10/398 6269
/proc/meminfo: memFree=23617768/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=320264 CPUtime=0.49 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 12204 937 1 0 41 8 0 0 24 0 5 0 658253654 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6252/statm: 80066 7254 1873 364 0 15975 0
[pid=6258] ppid=6252 vsize=278644 CPUtime=0.01 cores=0,2,4,6
/proc/6258/stat : 6258 (python) S 6252 6252 5751 0 -1 4202560 882 0 0 0 0 1 0 0 18 0 1 0 658253712 285331456 5589 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965263 0 0 16781312 514 10507383472413180162 0 0 17 0 0 0 0
/proc/6258/statm: 69661 5589 376 364 0 5579 0
[pid=6260] ppid=6258 vsize=1017940 CPUtime=5.7 cores=0,2,4,6
/proc/6260/stat : 6260 (gnovelty+2) R 6258 6260 6260 0 -1 4202496 453 0 0 0 570 0 0 0 25 0 1 0 658253712 1042370560 191 33554432000 134512640 135663704 4286767808 18446744073709551615 134525506 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6260/statm: 254485 191 113 282 0 254201 0
[pid=6261] ppid=6252 vsize=289408 CPUtime=0.01 cores=0,2,4,6
/proc/6261/stat : 6261 (python) S 6252 6252 5751 0 -1 4202560 917 0 0 0 1 0 0 0 15 0 1 0 658253713 296353792 5732 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 6055516313806143745 0 0 17 0 0 0 0
/proc/6261/statm: 72352 5732 371 364 0 8267 0
[pid=6263] ppid=6261 vsize=3876 CPUtime=5.67 cores=0,2,4,6
/proc/6263/stat : 6263 (march_hi) R 6261 6263 6263 0 -1 4202496 973 0 0 0 567 0 0 0 25 0 1 0 658253713 3969024 707 33554432000 4194304 4955518 140735642963792 18446744073709551615 4227635 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6263/statm: 969 707 104 186 0 782 0
[pid=6264] ppid=6252 vsize=299156 CPUtime=0 cores=0,2,4,6
/proc/6264/stat : 6264 (python) S 6252 6252 5751 0 -1 4202560 900 0 0 0 0 0 0 0 18 0 1 0 658253714 306335744 5624 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 4606912767868240129 0 0 17 6 0 0 0
/proc/6264/statm: 74789 5624 371 364 0 10701 0
[pid=6266] ppid=6264 vsize=1017940 CPUtime=5.61 cores=0,2,4,6
/proc/6266/stat : 6266 (gnovelty+2) R 6264 6266 6266 0 -1 4202496 455 0 0 0 561 0 0 0 25 0 1 0 658253714 1042370560 192 33554432000 134512640 135663704 4288933488 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/6266/statm: 254485 192 113 282 0 254201 0
[pid=6267] ppid=6252 vsize=310020 CPUtime=0.01 cores=0,2,4,6
/proc/6267/stat : 6267 (python) S 6252 6252 5751 0 -1 4202560 915 0 0 0 1 0 0 0 15 0 1 0 658253715 317460480 5748 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 17304399875256844545 0 0 17 2 0 0 0
/proc/6267/statm: 77505 5748 371 364 0 13414 0
[pid=6269] ppid=6267 vsize=3876 CPUtime=5.66 cores=0,2,4,6
/proc/6269/stat : 6269 (march_hi) R 6267 6269 6269 0 -1 4202496 970 0 0 0 566 0 0 0 25 0 1 0 658253715 3969024 706 33554432000 4194304 4955518 140735569719696 18446744073709551615 4238727 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6269/statm: 969 706 104 186 0 782 0
[pid=6252/tid=6259] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6259/stat : 6259 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 19 0 5 0 658253712 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 6 0 0 0
[pid=6252/tid=6262] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6262/stat : 6262 (python) S 6250 6252 5751 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 5 0 658253713 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6265] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6265/stat : 6265 (python) S 6250 6252 5751 0 -1 4202560 36 937 0 0 0 0 0 0 22 0 5 0 658253714 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6268] ppid=6250 vsize=320264 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6268/stat : 6268 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 24 0 5 0 658253715 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 23.16
Current children cumulated vsize (KiB) 3541124

[startup+12.7005 s]
/proc/loadavg: 8.31 8.06 6.15 10/398 6269
/proc/meminfo: memFree=23614340/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=320264 CPUtime=0.49 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 12204 937 1 0 41 8 0 0 24 0 5 0 658253654 327950336 7254 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6252/statm: 80066 7254 1873 364 0 15975 0
[pid=6258] ppid=6252 vsize=278644 CPUtime=0.03 cores=0,2,4,6
/proc/6258/stat : 6258 (python) S 6252 6252 5751 0 -1 4202560 890 0 0 0 1 2 0 0 18 0 1 0 658253712 285331456 5590 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965263 0 0 16781312 514 2628800360617050370 0 0 17 0 0 0 0
/proc/6258/statm: 69661 5590 377 364 0 5579 0
[pid=6260] ppid=6258 vsize=1017940 CPUtime=12.1 cores=0,2,4,6
/proc/6260/stat : 6260 (gnovelty+2) R 6258 6260 6260 0 -1 4202496 453 0 0 0 1210 0 0 0 25 0 1 0 658253712 1042370560 191 33554432000 134512640 135663704 4286767808 18446744073709551615 134545619 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6260/statm: 254485 191 113 282 0 254201 0
[pid=6261] ppid=6252 vsize=289408 CPUtime=0.03 cores=0,2,4,6

################
# More data... #
################

/proc/6276/stat : 6276 (march_hi) T 6274 6276 6276 0 -1 4202496 24568 0 0 0 68277 6 0 0 25 0 1 0 658263644 112611328 24305 33554432000 4194304 4955518 140733338133808 18446744073709551615 4224738 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6276/statm: 27493 24305 104 186 0 27306 0
[pid=6274/tid=6323] ppid=6252 vsize=320276 CPUtime=0 cores=0,2,4,6
/proc/6274/task/6323/stat : 6323 (python) S 6252 6252 5751 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658332014 327962624 5782 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604446908994880 0 0 -1 2 0 0 0
[pid=6278] ppid=6252 vsize=330008 CPUtime=2.29 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6252 6252 5751 0 -1 4202560 1154 0 0 0 121 108 0 0 15 0 2 0 658263739 337928192 5687 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6278/statm: 82502 5687 400 364 0 18405 0
[pid=6280] ppid=6278 vsize=128700 CPUtime=779 cores=0,2,4,6
/proc/6280/stat : 6280 (march_hi) T 6278 6280 6280 0 -1 4202496 27784 0 0 0 77895 5 0 0 25 0 1 0 658263739 131788800 27523 33554432000 4194304 4955518 140735737335472 18446744073709551615 4263652 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/6280/statm: 32175 27523 104 186 0 31988 0
[pid=6278/tid=6328] ppid=6252 vsize=330008 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6328/stat : 6328 (python) S 6252 6252 5751 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 658332142 337928192 5687 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604446908995776 0 0 -1 6 0 0 0
[pid=6298] ppid=6252 vsize=340888 CPUtime=1.98 cores=0,2,4,6
/proc/6298/stat : 6298 (python) S 6252 6252 5751 0 -1 4202560 1165 0 0 0 105 93 0 0 18 0 2 0 658293362 349069312 5824 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6298/statm: 85222 5824 409 364 0 21122 0
[pid=6300] ppid=6298 vsize=182580 CPUtime=701.53 cores=0,2,4,6
/proc/6300/stat : 6300 (kcnfs-2006) R 6298 6300 6300 0 -1 4202496 646 0 0 0 70153 0 0 0 25 0 1 0 658293363 186961920 386 33554432000 134512640 135163436 4287355088 18446744073709551615 134569074 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6300/statm: 45645 386 77 159 0 45485 0
[pid=6298/tid=6335] ppid=6252 vsize=340888 CPUtime=0 cores=0,2,4,6
/proc/6298/task/6335/stat : 6335 (python) S 6252 6252 5751 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658352028 349069312 5824 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604440784728512 0 0 -1 2 0 0 0
[pid=6317] ppid=6252 vsize=351144 CPUtime=1.49 cores=0,2,4,6
/proc/6317/stat : 6317 (python) S 6252 6252 5751 0 -1 4202560 1051 0 0 0 79 70 0 0 15 0 1 0 658322987 359571456 5817 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 2628800360617050369 0 0 17 2 0 0 0
/proc/6317/statm: 87786 5817 377 364 0 23683 0
[pid=6319] ppid=6317 vsize=80504 CPUtime=502.46 cores=0,2,4,6
/proc/6319/stat : 6319 (march_hi) R 6317 6319 6319 0 -1 4202496 18632 0 0 0 50239 7 0 0 25 0 1 0 658322987 82436096 18363 33554432000 4194304 4955518 140735651863968 18446744073709551615 4238756 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6319/statm: 20126 18363 104 186 0 19939 0
[pid=6324] ppid=6252 vsize=361588 CPUtime=1.23 cores=0,2,4,6
/proc/6324/stat : 6324 (python) S 6252 6252 5751 0 -1 4202560 1029 0 0 0 68 55 0 0 15 0 1 0 658332015 370266112 5846 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 15903428547423731969 0 0 17 2 0 0 0
/proc/6324/statm: 90397 5846 377 364 0 26291 0
[pid=6326] ppid=6324 vsize=80504 CPUtime=403.94 cores=0,2,4,6
/proc/6326/stat : 6326 (march_hi) R 6324 6326 6326 0 -1 4202496 15310 0 0 0 40392 2 0 0 25 0 1 0 658332015 82436096 15044 33554432000 4194304 4955518 140737098842240 18446744073709551615 4209776 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/6326/statm: 20126 15044 104 186 0 19939 0
[pid=6252/tid=6259] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6259/stat : 6259 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 19 0 10 0 658253712 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 6 0 0 0
[pid=6252/tid=6262] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6262/stat : 6262 (python) S 6250 6252 5751 0 -1 4202560 27 937 0 0 0 0 0 0 18 0 10 0 658253713 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6265] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6265/stat : 6265 (python) S 6250 6252 5751 0 -1 4202560 36 937 0 0 0 0 0 0 22 0 10 0 658253714 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6268] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6268/stat : 6268 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 24 0 10 0 658253715 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6275] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6275/stat : 6275 (python) S 6250 6252 5751 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 10 0 658263644 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6279] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6279/stat : 6279 (python) S 6250 6252 5751 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 10 0 658263739 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6299] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6299/stat : 6299 (python) S 6250 6252 5751 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 10 0 658293362 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6318] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6318/stat : 6318 (python) S 6250 6252 5751 0 -1 4202560 25 937 0 0 0 0 0 0 20 0 10 0 658322987 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6325] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6325/stat : 6325 (python) S 6250 6252 5751 0 -1 4202560 26 937 0 0 0 0 0 0 20 0 10 0 658332015 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4770.6
Current children cumulated vsize (KiB) 6164384

[startup+1198.3 s]
/proc/loadavg: 7.88 8.10 7.57 5/401 6360
/proc/meminfo: memFree=23124732/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=371996 CPUtime=0.56 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 17002 937 1 0 47 9 0 0 18 0 10 0 658253654 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6252/statm: 92999 7398 1884 364 0 28893 0
[pid=6258] ppid=6252 vsize=288888 CPUtime=0.26 cores=0,2,4,6
/proc/6258/stat : 6258 (python) S 6252 6252 5751 0 -1 4202560 1155 0 0 0 14 12 0 0 19 0 2 0 658253712 295821312 5634 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6258/statm: 72222 5634 416 364 0 8140 0
[pid=6260] ppid=6258 vsize=1017940 CPUtime=100.23 cores=0,2,4,6
/proc/6260/stat : 6260 (gnovelty+2) T 6258 6260 6260 0 -1 4202496 453 0 0 0 10023 0 0 0 25 0 1 0 658253712 1042370560 191 33554432000 134512640 135663704 4286767808 18446744073709551615 134525678 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/6260/statm: 254485 191 113 282 0 254201 0
[pid=6258/tid=6277] ppid=6252 vsize=288888 CPUtime=0 cores=0,2,4,6
/proc/6258/task/6277/stat : 6277 (python) S 6252 6252 5751 0 -1 4202560 43 0 0 0 0 0 0 0 19 0 2 0 658263738 295821312 5634 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604449321386304 0 0 -1 0 0 0 0
[pid=6261] ppid=6252 vsize=289408 CPUtime=2.37 cores=0,2,4,6
/proc/6261/stat : 6261 (python) S 6252 6252 5751 0 -1 4202560 1149 0 0 0 132 105 0 0 15 0 2 0 658253713 296353792 5762 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6261/statm: 72352 5762 400 364 0 8267 0
[pid=6263] ppid=6261 vsize=172900 CPUtime=805.39 cores=0,2,4,6
/proc/6263/stat : 6263 (march_hi) R 6261 6263 6263 0 -1 4202496 28748 0 0 0 80532 7 0 0 25 0 1 0 658253713 177049600 28482 33554432000 4194304 4955518 140735642963792 18446744073709551615 4209664 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6263/statm: 43225 28482 104 186 0 43038 0
[pid=6261/tid=6297] ppid=6252 vsize=289408 CPUtime=0 cores=0,2,4,6
/proc/6261/task/6297/stat : 6297 (python) S 6252 6252 5751 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 658293361 296353792 5762 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604440784731200 0 0 -1 0 0 0 0
[pid=6264] ppid=6252 vsize=299156 CPUtime=0.27 cores=0,2,4,6
/proc/6264/stat : 6264 (python) S 6252 6252 5751 0 -1 4202560 1091 0 0 0 14 13 0 0 19 0 2 0 658253714 306335744 5654 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6264/statm: 74789 5654 400 364 0 10701 0
[pid=6266] ppid=6264 vsize=1017940 CPUtime=97.84 cores=0,2,4,6
/proc/6266/stat : 6266 (gnovelty+2) T 6264 6266 6266 0 -1 4202496 456 0 0 0 9784 0 0 0 25 0 1 0 658253714 1042370560 193 33554432000 134512640 135663704 4288933488 18446744073709551615 134525286 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/6266/statm: 254485 193 113 282 0 254201 0
[pid=6264/tid=6273] ppid=6252 vsize=299156 CPUtime=0 cores=0,2,4,6
/proc/6264/task/6273/stat : 6273 (python) S 6252 6252 5751 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658263643 306335744 5654 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604453985323712 0 0 -1 6 0 0 0
[pid=6267] ppid=6252 vsize=310020 CPUtime=2.01 cores=0,2,4,6
/proc/6267/stat : 6267 (python) S 6252 6252 5751 0 -1 4202560 1160 0 0 0 104 97 0 0 16 0 2 0 658253715 317460480 5778 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6267/statm: 77505 5778 400 364 0 13414 0
[pid=6269] ppid=6267 vsize=109972 CPUtime=683.3 cores=0,2,4,6
/proc/6269/stat : 6269 (march_hi) T 6267 6269 6269 0 -1 4202496 24213 0 0 0 68325 5 0 0 25 0 1 0 658253715 112611328 23949 33554432000 4194304 4955518 140735569719696 18446744073709551615 4224628 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/6269/statm: 27493 23949 104 186 0 27306 0
[pid=6267/tid=6316] ppid=6252 vsize=310020 CPUtime=0 cores=0,2,4,6
/proc/6267/task/6316/stat : 6316 (python) S 6252 6252 5751 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658322986 317460480 5778 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604442565800256 0 0 -1 6 0 0 0
[pid=6274] ppid=6252 vsize=320276 CPUtime=2 cores=0,2,4,6
/proc/6274/stat : 6274 (python) S 6252 6252 5751 0 -1 4202560 1155 0 0 0 107 93 0 0 16 0 2 0 658263644 327962624 5782 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6274/statm: 80069 5782 400 364 0 15975 0
[pid=6276] ppid=6274 vsize=109972 CPUtime=682.83 cores=0,2,4,6
/proc/6276/stat : 6276 (march_hi) T 6274 6276 6276 0 -1 4202496 24568 0 0 0 68277 6 0 0 25 0 1 0 658263644 112611328 24305 33554432000 4194304 4955518 140733338133808 18446744073709551615 4224738 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6276/statm: 27493 24305 104 186 0 27306 0
[pid=6274/tid=6323] ppid=6252 vsize=320276 CPUtime=0 cores=0,2,4,6
/proc/6274/task/6323/stat : 6323 (python) S 6252 6252 5751 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658332014 327962624 5782 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604446908994880 0 0 -1 2 0 0 0
[pid=6278] ppid=6252 vsize=330008 CPUtime=2.29 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6252 6252 5751 0 -1 4202560 1154 0 0 0 121 108 0 0 15 0 2 0 658263739 337928192 5687 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458771915 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6278/statm: 82502 5687 400 364 0 18405 0
[pid=6280] ppid=6278 vsize=128700 CPUtime=779 cores=0,2,4,6
/proc/6280/stat : 6280 (march_hi) T 6278 6280 6280 0 -1 4202496 27784 0 0 0 77895 5 0 0 25 0 1 0 658263739 131788800 27523 33554432000 4194304 4955518 140735737335472 18446744073709551615 4263652 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/6280/statm: 32175 27523 104 186 0 31988 0
[pid=6278/tid=6328] ppid=6252 vsize=330008 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6328/stat : 6328 (python) S 6252 6252 5751 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 658332142 337928192 5687 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604446908995776 0 0 -1 6 0 0 0
[pid=6298] ppid=6252 vsize=340888 CPUtime=1.98 cores=0,2,4,6
/proc/6298/stat : 6298 (python) S 6252 6252 5751 0 -1 4202560 1165 0 0 0 105 93 0 0 18 0 2 0 658293362 349069312 5824 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6298/statm: 85222 5824 409 364 0 21122 0
[pid=6300] ppid=6298 vsize=182580 CPUtime=701.92 cores=0,2,4,6
/proc/6300/stat : 6300 (kcnfs-2006) R 6298 6300 6300 0 -1 4202496 646 0 0 0 70192 0 0 0 25 0 1 0 658293363 186961920 386 33554432000 134512640 135163436 4287355088 18446744073709551615 134559218 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6300/statm: 45645 386 77 159 0 45485 0
[pid=6298/tid=6335] ppid=6252 vsize=340888 CPUtime=0 cores=0,2,4,6
/proc/6298/task/6335/stat : 6335 (python) S 6252 6252 5751 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658352028 349069312 5824 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 514 18446604440784728512 0 0 -1 2 0 0 0
[pid=6317] ppid=6252 vsize=351144 CPUtime=1.49 cores=0,2,4,6
/proc/6317/stat : 6317 (python) S 6252 6252 5751 0 -1 4202560 1051 0 0 0 79 70 0 0 15 0 1 0 658322987 359571456 5817 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 2628800360617050369 0 0 17 2 0 0 0
/proc/6317/statm: 87786 5817 377 364 0 23683 0
[pid=6319] ppid=6317 vsize=80504 CPUtime=502.85 cores=0,2,4,6
/proc/6319/stat : 6319 (march_hi) R 6317 6319 6319 0 -1 4202496 18646 0 0 0 50278 7 0 0 25 0 1 0 658322987 82436096 18377 33554432000 4194304 4955518 140735651863968 18446744073709551615 4227917 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6319/statm: 20126 18377 104 186 0 19939 0
[pid=6324] ppid=6252 vsize=361588 CPUtime=1.23 cores=0,2,4,6
/proc/6324/stat : 6324 (python) S 6252 6252 5751 0 -1 4202560 1029 0 0 0 68 55 0 0 15 0 1 0 658332015 370266112 5846 33554432000 4194304 5683644 140735013828944 18446744073709551615 244446965318 0 0 16781312 514 15903428547423731969 0 0 17 2 0 0 0
/proc/6324/statm: 90397 5846 377 364 0 26291 0
[pid=6326] ppid=6324 vsize=80504 CPUtime=404.32 cores=0,2,4,6
/proc/6326/stat : 6326 (march_hi) R 6324 6326 6326 0 -1 4202496 15324 0 0 0 40430 2 0 0 25 0 1 0 658332015 82436096 15058 33554432000 4194304 4955518 140737098842240 18446744073709551615 4227640 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/6326/statm: 20126 15058 104 186 0 19939 0
[pid=6252/tid=6259] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6259/stat : 6259 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 19 0 10 0 658253712 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 6 0 0 0
[pid=6252/tid=6262] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6262/stat : 6262 (python) S 6250 6252 5751 0 -1 4202560 27 937 0 0 0 0 0 0 18 0 10 0 658253713 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6265] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6265/stat : 6265 (python) S 6250 6252 5751 0 -1 4202560 36 937 0 0 0 0 0 0 22 0 10 0 658253714 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6268] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6268/stat : 6268 (python) S 6250 6252 5751 0 -1 4202560 24 937 0 0 0 0 0 0 24 0 10 0 658253715 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6275] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6275/stat : 6275 (python) S 6250 6252 5751 0 -1 4202560 23 937 0 0 0 0 0 0 20 0 10 0 658263644 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6279] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6279/stat : 6279 (python) S 6250 6252 5751 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 10 0 658263739 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6299] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6299/stat : 6299 (python) S 6250 6252 5751 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 10 0 658293362 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6318] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6318/stat : 6318 (python) S 6250 6252 5751 0 -1 4202560 25 937 0 0 0 0 0 0 20 0 10 0 658322987 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6325] ppid=6250 vsize=371996 CPUtime=0 cores=0,2,4,6
/proc/6252/task/6325/stat : 6325 (python) S 6250 6252 5751 0 -1 4202560 26 937 0 0 0 0 0 0 20 0 10 0 658332015 380923904 7398 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4772.14
Current children cumulated vsize (KiB) 6164384

[startup+1198.4 s]
/proc/loadavg: 7.88 8.10 7.57 5/401 6360
/proc/meminfo: memFree=23124732/32951124 swapFree=44485764/67111528
[pid=6252] ppid=6250 vsize=370908 CPUtime=4772.67 cores=0,2,4,6
/proc/6252/stat : 6252 (python) S 6250 6252 5751 0 -1 4202496 17143 152179 1 0 48 9 476520 690 17 0 8 0 658253654 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/6252/statm: 92727 7169 1867 364 0 28649 0
[pid=6252/tid=6259] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6259/stat : 6259 (python) R 6250 6252 5751 0 -1 4202560 29 152179 0 0 0 0 476520 690 18 0 8 0 658253712 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 0 0 0 -1 6 0 0 0
[pid=6252/tid=6262] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6262/stat : 6262 (python) S 6250 6252 5751 0 -1 4202560 27 152179 0 0 0 0 476520 690 18 0 8 0 658253713 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6265] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6265/stat : 6265 (python) S 6250 6252 5751 0 -1 4202560 38 152179 0 0 0 0 476520 690 18 0 8 0 658253714 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6268] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6268/stat : 6268 (python) S 6250 6252 5751 0 -1 4202560 27 152179 0 0 0 0 476520 690 18 0 8 0 658253715 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6275] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6275/stat : 6275 (python) S 6250 6252 5751 0 -1 4202560 29 152179 0 0 0 0 476520 690 18 0 8 0 658263644 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6279] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6279/stat : 6279 (python) S 6250 6252 5751 0 -1 4202560 31 152179 0 0 0 0 476520 690 18 0 8 0 658263739 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
[pid=6252/tid=6299] ppid=6250 vsize=370908 CPUtime=4772.1 cores=0,2,4,6
/proc/6252/task/6299/stat : 6299 (python) S 6250 6252 5751 0 -1 4202560 32 152179 0 0 0 0 476520 690 18 0 8 0 658293362 379809792 7169 33554432000 4194304 5683644 140735013828944 18446744073709551615 244458768817 0 0 16781312 2 18446604450939845696 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4772.67
Current children cumulated vsize (KiB) 370908

Child status: 0
Real time (s): 1198.42
CPU time (s): 4772.71
CPU user time (s): 4765.7
CPU system time (s): 7.00593
CPU usage (%): 398.25
Max. virtual memory (cumulated for all children) (KiB): 6164388

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4765.7
system time used= 7.00593
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 173242
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7822
involuntary context switches= 28124

runsolver used 6.33204 second user time and 12.5191 second system time

The end

Launcher Data 

Begin job on node105 at 2011-05-15 15:35:32
IDJOB=3378637
IDBENCH=83046
IDSOLVER=1859
FILE ID=node105/3378637-1305466532
RUNJOBID= node105-1305465320-5769
PBS_JOBID= 13325059
Free space on /tmp= 73548 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v650-c2769-S2065143904-035.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378637-1305466532/watcher-3378637-1305466532 -o /tmp/evaluation-result-3378637-1305466532/solver-3378637-1305466532 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378637-1305466532.cnf -seed 108187587 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8f7328be14ce714cebc5a66418b5924c
RANDOM SEED=108187587

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23664604 kB
Buffers:        308760 kB
Cached:        6815892 kB
SwapCached:      39656 kB
Active:        1663556 kB
Inactive:      7336800 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23664604 kB
SwapTotal:    67111528 kB
SwapFree:     44485764 kB
Dirty:           65164 kB
Writeback:           0 kB
AnonPages:     1844216 kB
Mapped:          22088 kB
Slab:           144560 kB
PageTables:      75912 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 93607660 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73548 MiB
End job on node105 at 2011-05-15 15:55:31