Trace number 3378613

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 3080.61 1158.15

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S689381858-022.UNKNOWN.cnf
MD5SUMe5ccab3a9294cbb9eed5df8c5c797376
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.32/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.45	c INFO: solving HOME/instance-3378613-1305466368.cnf
0.32/0.45	c INFO: computing task features
0.32/0.46	c INFO: computed variable-clause graph statistics
0.32/0.46	c INFO: computed clause balance statistics
0.41/0.54	c INFO: computed variable balance statistics
0.41/0.57	c INFO: computed small-clause counts
0.41/0.57	c INFO: computed Horn-clause counts
0.41/0.58	c INFO: computed variable graph statistics
0.41/0.58	c INFO: collected features for HOME/instance-3378613-1305466368.cnf in 0.13 s
0.41/0.59	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.63)
0.41/0.59	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.52/0.60	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.29)
0.52/0.62	c INFO: running march_hi@0 for 679 with 1199 remaining (b = 0.36)
460.07/185.25	c INFO: running march_hi@0 for 679 with 1015 remaining (b = 0.32)
495.87/197.20	c INFO: running march_hi@0 for 388 with 1003 remaining (b = 0.21)
1995.28/719.29	c INFO: running march_hi@98 for 481 with 481 remaining (b = 0.18)
2032.10/734.08	c INFO: running march_hi@0 for 466 with 466 remaining (b = 0.18)
2163.43/786.71	c INFO: running march_hi@0 for 413 with 413 remaining (b = 0.18)
3080.43/1158.12	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378613-1305466368/watcher-3378613-1305466368 -o /tmp/evaluation-result-3378613-1305466368/solver-3378613-1305466368 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378613-1305466368.cnf -seed 1782851059 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.32 8.23 8.04 7/459 29903
/proc/meminfo: memFree=20772556/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=16240 CPUtime=0 cores=1,3,5,7
/proc/29903/stat : 29903 (python) R 29900 29903 28586 0 -1 4202496 1038 0 0 0 0 0 0 0 25 0 1 0 736867496 16629760 911 33554432000 4194304 5683644 140734246027200 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/29903/statm: 4060 911 380 364 0 547 0

[startup+0.0332119 s]
/proc/loadavg: 8.32 8.23 8.04 7/459 29903
/proc/meminfo: memFree=20772556/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=18884 CPUtime=0.02 cores=1,3,5,7
/proc/29903/stat : 29903 (python) R 29900 29903 28586 0 -1 4202496 1283 0 0 0 2 0 0 0 25 0 1 0 736867496 19337216 1137 33554432000 4194304 5683644 140734246027200 18446744073709551615 4591900 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/29903/statm: 4721 1137 421 364 0 690 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18884

[startup+0.100203 s]
/proc/loadavg: 8.32 8.23 8.04 7/459 29903
/proc/meminfo: memFree=20772556/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=104616 CPUtime=0.07 cores=1,3,5,7
/proc/29903/stat : 29903 (python) R 29900 29903 28586 0 -1 4202496 3142 0 0 0 5 2 0 0 25 0 1 0 736867496 107126784 2822 33554432000 4194304 5683644 140734246027200 18446744073709551615 4623632 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/29903/statm: 26154 2822 847 364 0 1902 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104616

[startup+0.300167 s]
/proc/loadavg: 8.32 8.23 8.04 7/459 29903
/proc/meminfo: memFree=20772556/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=198920 CPUtime=0.24 cores=1,3,5,7
/proc/29903/stat : 29903 (python) R 29900 29903 28586 0 -1 4202496 7021 934 1 0 17 7 0 0 25 0 1 0 736867496 203694080 5054 33554432000 4194304 5683644 140734246027200 18446744073709551615 260908802720 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/29903/statm: 49730 5054 1466 364 0 3757 0
Current children cumulated CPU time (s) 0.24
Current children cumulated vsize (KiB) 198920

[startup+0.700093 s]
/proc/loadavg: 8.32 8.23 8.04 7/459 29903
/proc/meminfo: memFree=20772556/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=320100 CPUtime=0.53 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 12247 934 1 0 44 9 0 0 25 0 5 0 736867496 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29903/statm: 80025 7239 1881 364 0 15934 0
Current children cumulated CPU time (s) 0.53
Current children cumulated vsize (KiB) 320100

[startup+1.50094 s]
/proc/loadavg: 8.32 8.23 8.04 11/472 29924
/proc/meminfo: memFree=20732012/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=320100 CPUtime=0.53 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 12247 934 1 0 44 9 0 0 25 0 5 0 736867496 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29903/statm: 80025 7239 1881 364 0 15934 0
[pid=29910] ppid=29903 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/29910/stat : 29910 (python) S 29903 29903 28586 0 -1 4202560 735 0 0 0 0 0 0 0 25 0 1 0 736867555 285237248 5558 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414159 0 0 16781312 514 16178469456966582271 0 0 17 3 0 0 0
/proc/29910/statm: 69638 5558 354 364 0 5556 0
[pid=29913] ppid=29910 vsize=1017800 CPUtime=0.6 cores=1,3,5,7
/proc/29913/stat : 29913 (gnovelty+2) R 29910 29913 29913 0 -1 4202496 442 0 0 0 60 0 0 0 25 0 1 0 736867556 1042227200 180 33554432000 134512640 135663704 4291853600 18446744073709551615 134524470 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29913/statm: 254450 180 113 282 0 254166 0
[pid=29914] ppid=29903 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/29914/stat : 29914 (python) S 29903 29903 28586 0 -1 4202560 735 0 0 0 0 0 0 0 25 0 1 0 736867556 296288256 5677 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17389705484897878015 0 0 17 7 0 0 0
/proc/29914/statm: 72336 5677 348 364 0 8251 0
[pid=29919] ppid=29914 vsize=1017800 CPUtime=0.85 cores=1,3,5,7
/proc/29919/stat : 29919 (gnovelty+2) R 29914 29919 29919 0 -1 4202496 443 0 0 0 85 0 0 0 25 0 1 0 736867558 1042227200 180 33554432000 134512640 135663704 4289361088 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29919/statm: 254450 180 113 282 0 254166 0
[pid=29917] ppid=29903 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/29917/stat : 29917 (python) S 29903 29903 28586 0 -1 4202560 749 0 0 0 0 0 0 0 18 0 1 0 736867557 306790400 5698 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 16020003168247054599 0 0 17 3 0 0 0
/proc/29917/statm: 74900 5698 348 364 0 10812 0
[pid=29920] ppid=29917 vsize=1860 CPUtime=0.3 cores=1,3,5,7
/proc/29920/stat : 29920 (march_hi) R 29917 29920 29920 0 -1 4202496 536 0 0 0 30 0 0 0 25 0 1 0 736867558 1904640 274 33554432000 4194304 4955518 140733227520304 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29920/statm: 465 274 106 186 0 278 0
[pid=29921] ppid=29903 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/29921/stat : 29921 (python) S 29903 29903 28586 0 -1 4202560 755 0 0 0 0 0 0 0 18 0 1 0 736867559 317292544 5702 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 5459726147086418182 0 0 17 7 0 0 0
/proc/29921/statm: 77464 5702 348 364 0 13373 0
[pid=29924] ppid=29921 vsize=1860 CPUtime=0.41 cores=1,3,5,7
/proc/29924/stat : 29924 (march_hi) R 29921 29924 29924 0 -1 4202496 543 0 0 0 41 0 0 0 25 0 1 0 736867559 1904640 283 33554432000 4194304 4955518 140737051804912 18446744073709551615 4244951 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29924/statm: 465 283 106 186 0 278 0
[pid=29903/tid=29911] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29911/stat : 29911 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 736867555 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 3 0 0 0
[pid=29903/tid=29915] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29915/stat : 29915 (python) S 29900 29903 28586 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 5 0 736867556 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29918] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29918/stat : 29918 (python) S 29900 29903 28586 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 736867557 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29923] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29923/stat : 29923 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 736867559 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2.69
Current children cumulated vsize (KiB) 3536772

[startup+3.10065 s]
/proc/loadavg: 8.45 8.26 8.05 11/472 29924
/proc/meminfo: memFree=20730028/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=320100 CPUtime=0.53 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 12247 934 1 0 44 9 0 0 25 0 5 0 736867496 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29903/statm: 80025 7239 1881 364 0 15934 0
[pid=29910] ppid=29903 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/29910/stat : 29910 (python) S 29903 29903 28586 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 736867555 285237248 5558 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414159 0 0 16781312 514 16178469456966549764 0 0 17 3 0 0 0
/proc/29910/statm: 69638 5558 354 364 0 5556 0
[pid=29913] ppid=29910 vsize=1017800 CPUtime=1.4 cores=1,3,5,7
/proc/29913/stat : 29913 (gnovelty+2) R 29910 29913 29913 0 -1 4202496 442 0 0 0 140 0 0 0 25 0 1 0 736867556 1042227200 180 33554432000 134512640 135663704 4291853600 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29913/statm: 254450 180 113 282 0 254166 0
[pid=29914] ppid=29903 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/29914/stat : 29914 (python) S 29903 29903 28586 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 736867556 296288256 5677 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17389705484897845512 0 0 17 7 0 0 0
/proc/29914/statm: 72336 5677 348 364 0 8251 0
[pid=29919] ppid=29914 vsize=1017800 CPUtime=2.43 cores=1,3,5,7
/proc/29919/stat : 29919 (gnovelty+2) R 29914 29919 29919 0 -1 4202496 443 0 0 0 243 0 0 0 25 0 1 0 736867558 1042227200 180 33554432000 134512640 135663704 4289361088 18446744073709551615 134525533 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29919/statm: 254450 180 113 282 0 254166 0
[pid=29917] ppid=29903 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/29917/stat : 29917 (python) S 29903 29903 28586 0 -1 4202560 757 0 0 0 0 0 0 0 15 0 1 0 736867557 306790400 5698 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 16020003168247054599 0 0 17 3 0 0 0
/proc/29917/statm: 74900 5698 348 364 0 10812 0
[pid=29920] ppid=29917 vsize=2072 CPUtime=1.1 cores=1,3,5,7
/proc/29920/stat : 29920 (march_hi) R 29917 29920 29920 0 -1 4202496 635 0 0 0 110 0 0 0 25 0 1 0 736867558 2121728 354 33554432000 4194304 4955518 140733227520304 18446744073709551615 4297378 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29920/statm: 518 354 106 186 0 331 0
[pid=29921] ppid=29903 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/29921/stat : 29921 (python) S 29903 29903 28586 0 -1 4202560 760 0 0 0 0 0 0 0 15 0 1 0 736867559 317292544 5702 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 5459726147086418182 0 0 17 7 0 0 0
/proc/29921/statm: 77464 5702 348 364 0 13373 0
[pid=29924] ppid=29921 vsize=2224 CPUtime=1.21 cores=1,3,5,7
/proc/29924/stat : 29924 (march_hi) R 29921 29924 29924 0 -1 4202496 642 0 0 0 121 0 0 0 25 0 1 0 736867559 2277376 363 33554432000 4194304 4955518 140737051804912 18446744073709551615 4210508 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29924/statm: 556 363 106 186 0 369 0
[pid=29903/tid=29911] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29911/stat : 29911 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 736867555 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 3 0 0 0
[pid=29903/tid=29915] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29915/stat : 29915 (python) S 29900 29903 28586 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 5 0 736867556 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29918] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29918/stat : 29918 (python) S 29900 29903 28586 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 736867557 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29923] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29923/stat : 29923 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 736867559 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.67
Current children cumulated vsize (KiB) 3537348

[startup+6.30106 s]
/proc/loadavg: 8.45 8.26 8.05 11/472 29924
/proc/meminfo: memFree=20724664/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=320100 CPUtime=0.53 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 12247 934 1 0 44 9 0 0 25 0 5 0 736867496 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29903/statm: 80025 7239 1881 364 0 15934 0
[pid=29910] ppid=29903 vsize=278552 CPUtime=0.01 cores=1,3,5,7
/proc/29910/stat : 29910 (python) S 29903 29903 28586 0 -1 4202560 895 0 0 0 1 0 0 0 18 0 1 0 736867555 285237248 5580 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414159 0 0 16781312 514 16178469456966549764 0 0 17 3 0 0 0
/proc/29910/statm: 69638 5580 376 364 0 5556 0
[pid=29913] ppid=29910 vsize=1017800 CPUtime=2.95 cores=1,3,5,7
/proc/29913/stat : 29913 (gnovelty+2) R 29910 29913 29913 0 -1 4202496 442 0 0 0 295 0 0 0 25 0 1 0 736867556 1042227200 180 33554432000 134512640 135663704 4291853600 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29913/statm: 254450 180 113 282 0 254166 0
[pid=29914] ppid=29903 vsize=289344 CPUtime=0.01 cores=1,3,5,7
/proc/29914/stat : 29914 (python) S 29903 29903 28586 0 -1 4202560 914 0 0 0 1 0 0 0 18 0 1 0 736867556 296288256 5700 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17389705484897845512 0 0 17 7 0 0 0
/proc/29914/statm: 72336 5700 371 364 0 8251 0
[pid=29919] ppid=29914 vsize=1017800 CPUtime=5.55 cores=1,3,5,7
/proc/29919/stat : 29919 (gnovelty+2) R 29914 29919 29919 0 -1 4202496 443 0 0 0 555 0 0 0 25 0 1 0 736867558 1042227200 180 33554432000 134512640 135663704 4289361088 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29919/statm: 254450 180 113 282 0 254166 0
[pid=29917] ppid=29903 vsize=299600 CPUtime=0.02 cores=1,3,5,7
/proc/29917/stat : 29917 (python) S 29903 29903 28586 0 -1 4202560 916 0 0 0 1 1 0 0 15 0 1 0 736867557 306790400 5721 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 16020003168247054599 0 0 17 3 0 0 0
/proc/29917/statm: 74900 5721 371 364 0 10812 0
[pid=29920] ppid=29917 vsize=2452 CPUtime=2.7 cores=1,3,5,7
/proc/29920/stat : 29920 (march_hi) R 29917 29920 29920 0 -1 4202496 733 0 0 0 270 0 0 0 25 0 1 0 736867558 2510848 452 33554432000 4194304 4955518 140733227520304 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29920/statm: 613 452 106 186 0 426 0
[pid=29921] ppid=29903 vsize=309856 CPUtime=0.01 cores=1,3,5,7
/proc/29921/stat : 29921 (python) S 29903 29903 28586 0 -1 4202560 915 0 0 0 0 1 0 0 15 0 1 0 736867559 317292544 5724 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 5459726147086418182 0 0 17 7 0 0 0
/proc/29921/statm: 77464 5724 370 364 0 13373 0
[pid=29924] ppid=29921 vsize=2792 CPUtime=2.81 cores=1,3,5,7
/proc/29924/stat : 29924 (march_hi) R 29921 29924 29924 0 -1 4202496 739 0 0 0 281 0 0 0 25 0 1 0 736867559 2859008 460 33554432000 4194304 4955518 140737051804912 18446744073709551615 4224468 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29924/statm: 698 460 106 186 0 511 0
[pid=29903/tid=29911] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29911/stat : 29911 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 736867555 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 3 0 0 0
[pid=29903/tid=29915] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29915/stat : 29915 (python) S 29900 29903 28586 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 5 0 736867556 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29918] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29918/stat : 29918 (python) S 29900 29903 28586 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 736867557 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29923] ppid=29900 vsize=320100 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29923/stat : 29923 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 5 0 736867559 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 14.59
Current children cumulated vsize (KiB) 3538296

[startup+12.7009 s]
/proc/loadavg: 8.69 8.31 8.07 11/472 29924
/proc/meminfo: memFree=20719716/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=320100 CPUtime=0.53 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 12247 934 1 0 44 9 0 0 25 0 5 0 736867496 327782400 7239 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29903/statm: 80025 7239 1881 364 0 15934 0
[pid=29910] ppid=29903 vsize=278552 CPUtime=0.03 cores=1,3,5,7
/proc/29910/stat : 29910 (python) S 29903 29903 28586 0 -1 4202560 898 0 0 0 2 1 0 0 18 0 1 0 736867555 285237248 5580 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414159 0 0 16781312 514 17795428569787105540 0 0 17 3 0 0 0
/proc/29910/statm: 69638 5580 376 364 0 5556 0
[pid=29913] ppid=29910 vsize=1017800 CPUtime=6.12 cores=1,3,5,7
/proc/29913/stat : 29913 (gnovelty+2) R 29910 29913 29913 0 -1 4202496 442 0 0 0 612 0 0 0 25 0 1 0 736867556 1042227200 180 33554432000 134512640 135663704 4291853600 18446744073709551615 134525253 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29913/statm: 254450 180 113 282 0 254166 0
[pid=29914] ppid=29903 vsize=289344 CPUtime=0.03 cores=1,3,5,7

################
# More data... #
################

[pid=29921] ppid=29903 vsize=309856 CPUtime=3.89 cores=1,3,5,7
/proc/29921/stat : 29921 (python) S 29903 29903 28586 0 -1 4202560 1157 0 0 0 206 183 0 0 15 0 2 0 736867559 317292544 5755 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29921/statm: 77464 5755 400 364 0 13373 0
[pid=29924] ppid=29921 vsize=213800 CPUtime=941.92 cores=1,3,5,7
/proc/29924/stat : 29924 (march_hi) R 29921 29924 29924 0 -1 4202496 40917 0 0 0 94183 9 0 0 25 0 1 0 736867559 218931200 40638 33554432000 4194304 4955518 140737051804912 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29924/statm: 53450 40638 106 186 0 53263 0
[pid=29921/tid=30219] ppid=29903 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/29921/task/30219/stat : 30219 (python) S 29903 29903 28586 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 736939425 317292544 5755 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604464836123392 0 0 -1 5 0 0 0
[pid=29934] ppid=29903 vsize=320112 CPUtime=3.33 cores=1,3,5,7
/proc/29934/stat : 29934 (python) S 29903 29903 28586 0 -1 4202560 999 0 0 0 174 159 0 0 15 0 1 0 736886022 327794688 5729 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17795428569787105542 0 0 17 7 0 0 0
/proc/29934/statm: 80028 5729 371 364 0 15934 0
[pid=29937] ppid=29934 vsize=140936 CPUtime=679.42 cores=1,3,5,7
/proc/29937/stat : 29937 (march_hi) R 29934 29937 29937 0 -1 4202496 30186 0 0 0 67936 6 0 0 25 0 1 0 736886022 144318464 29908 33554432000 4194304 4955518 140735454342000 18446744073709551615 4263698 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29937/statm: 35234 29908 106 186 0 35047 0
[pid=29939] ppid=29903 vsize=330512 CPUtime=1.99 cores=1,3,5,7
/proc/29939/stat : 29939 (python) S 29903 29903 28586 0 -1 4202560 1148 0 0 0 104 95 0 0 16 0 2 0 736887217 338444288 5782 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29939/statm: 82628 5782 400 364 0 18531 0
[pid=29941] ppid=29939 vsize=93456 CPUtime=392.38 cores=1,3,5,7
/proc/29941/stat : 29941 (march_hi) T 29939 29941 29941 0 -1 4202496 18519 0 0 0 39234 4 0 0 25 0 1 0 736887217 95698944 18241 33554432000 4194304 4955518 140737241208352 18446744073709551615 4297477 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/29941/statm: 23364 18241 106 186 0 23177 0
[pid=29939/tid=30223] ppid=29903 vsize=330512 CPUtime=0 cores=1,3,5,7
/proc/29939/task/30223/stat : 30223 (python) S 29903 29903 28586 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 736946167 338444288 5782 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604454462878784 0 0 -1 3 0 0 0
[pid=30224] ppid=29903 vsize=340264 CPUtime=1.24 cores=1,3,5,7
/proc/30224/stat : 30224 (python) S 29903 29903 28586 0 -1 4202560 1002 0 0 0 67 57 0 0 15 0 1 0 736946168 348430336 5662 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17795428569787105541 0 0 17 7 0 0 0
/proc/30224/statm: 85066 5662 371 364 0 20966 0
[pid=30226] ppid=30224 vsize=63988 CPUtime=246 cores=1,3,5,7
/proc/30226/stat : 30226 (march_hi) R 30224 30226 30226 0 -1 4202496 12180 0 0 0 24598 2 0 0 25 0 1 0 736946178 65523712 11900 33554432000 4194304 4955518 140735377106960 18446744073709551615 4245413 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/30226/statm: 15997 11900 106 186 0 15810 0
[pid=29903/tid=29911] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29911/stat : 29911 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 8 0 736867555 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 3 0 0 0
[pid=29903/tid=29915] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29915/stat : 29915 (python) S 29900 29903 28586 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 8 0 736867556 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29918] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29918/stat : 29918 (python) S 29900 29903 28586 0 -1 4202560 22 934 0 0 0 0 0 0 18 0 8 0 736867557 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29923] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29923/stat : 29923 (python) S 29900 29903 28586 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 8 0 736867559 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29936] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29936/stat : 29936 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 8 0 736886022 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29940] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29940/stat : 29940 (python) S 29900 29903 28586 0 -1 4202560 21 934 0 0 0 0 0 0 20 0 8 0 736887217 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=30225] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/30225/stat : 30225 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 8 0 736946178 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3078.87
Current children cumulated vsize (KiB) 5217708

[startup+1157.9 s]
/proc/loadavg: 10.31 10.18 9.57 11/478 30245
/proc/meminfo: memFree=20069016/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=350508 CPUtime=0.57 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 14951 934 1 0 47 10 0 0 19 0 8 0 736867496 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29903/statm: 87627 7183 1888 364 0 23527 0
[pid=29910] ppid=29903 vsize=288796 CPUtime=0.62 cores=1,3,5,7
/proc/29910/stat : 29910 (python) S 29903 29903 28586 0 -1 4202560 1197 0 0 0 32 30 0 0 19 0 2 0 736867555 295727104 5625 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29910/statm: 72199 5625 416 364 0 8117 0
[pid=29913] ppid=29910 vsize=1017800 CPUtime=102.61 cores=1,3,5,7
/proc/29913/stat : 29913 (gnovelty+2) T 29910 29913 29913 0 -1 4202496 442 0 0 0 10261 0 0 0 25 0 1 0 736867556 1042227200 180 33554432000 134512640 135663704 4291853600 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/29913/statm: 254450 180 113 282 0 254166 0
[pid=29910/tid=29938] ppid=29903 vsize=288796 CPUtime=0 cores=1,3,5,7
/proc/29910/task/29938/stat : 29938 (python) S 29903 29903 28586 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 736887216 295727104 5625 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604469564438592 0 0 -1 3 0 0 0
[pid=29914] ppid=29903 vsize=289344 CPUtime=0.59 cores=1,3,5,7
/proc/29914/stat : 29914 (python) S 29903 29903 28586 0 -1 4202560 1119 0 0 0 32 27 0 0 19 0 2 0 736867556 296288256 5730 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/29914/statm: 72336 5730 400 364 0 8251 0
[pid=29919] ppid=29914 vsize=1017800 CPUtime=98.49 cores=1,3,5,7
/proc/29919/stat : 29919 (gnovelty+2) T 29914 29919 29919 0 -1 4202496 443 0 0 0 9849 0 0 0 25 0 1 0 736867558 1042227200 180 33554432000 134512640 135663704 4289361088 18446744073709551615 134524503 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/29919/statm: 254450 180 113 282 0 254166 0
[pid=29914/tid=29933] ppid=29903 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/29914/task/29933/stat : 29933 (python) S 29903 29903 28586 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 736886021 296288256 5730 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604456961549568 0 0 -1 7 0 0 0
[pid=29917] ppid=29903 vsize=299600 CPUtime=3.86 cores=1,3,5,7
/proc/29917/stat : 29917 (python) S 29903 29903 28586 0 -1 4202560 1152 0 0 0 201 185 0 0 15 0 2 0 736867557 306790400 5751 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29917/statm: 74900 5751 400 364 0 10812 0
[pid=29920] ppid=29917 vsize=140936 CPUtime=602.16 cores=1,3,5,7
/proc/29920/stat : 29920 (march_hi) R 29917 29920 29920 0 -1 4202496 27205 0 0 0 60211 5 0 0 25 0 1 0 736867558 144318464 26924 33554432000 4194304 4955518 140733227520304 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29920/statm: 35234 26924 106 186 0 35047 0
[pid=29917/tid=30221] ppid=29903 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/29917/task/30221/stat : 30221 (python) S 29903 29903 28586 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 736940904 306790400 5751 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604468944577664 0 0 -1 3 0 0 0
[pid=29921] ppid=29903 vsize=309856 CPUtime=3.89 cores=1,3,5,7
/proc/29921/stat : 29921 (python) S 29903 29903 28586 0 -1 4202560 1157 0 0 0 206 183 0 0 15 0 2 0 736867559 317292544 5755 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29921/statm: 77464 5755 400 364 0 13373 0
[pid=29924] ppid=29921 vsize=213800 CPUtime=942.32 cores=1,3,5,7
/proc/29924/stat : 29924 (march_hi) R 29921 29924 29924 0 -1 4202496 40932 0 0 0 94223 9 0 0 25 0 1 0 736867559 218931200 40653 33554432000 4194304 4955518 140737051804912 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29924/statm: 53450 40653 106 186 0 53263 0
[pid=29921/tid=30219] ppid=29903 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/29921/task/30219/stat : 30219 (python) S 29903 29903 28586 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 736939425 317292544 5755 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604464836123392 0 0 -1 5 0 0 0
[pid=29934] ppid=29903 vsize=320112 CPUtime=3.33 cores=1,3,5,7
/proc/29934/stat : 29934 (python) S 29903 29903 28586 0 -1 4202560 999 0 0 0 174 159 0 0 15 0 1 0 736886022 327794688 5729 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17795428569787105542 0 0 17 7 0 0 0
/proc/29934/statm: 80028 5729 371 364 0 15934 0
[pid=29937] ppid=29934 vsize=140936 CPUtime=679.62 cores=1,3,5,7
/proc/29937/stat : 29937 (march_hi) R 29934 29937 29937 0 -1 4202496 30195 0 0 0 67956 6 0 0 25 0 1 0 736886022 144318464 29917 33554432000 4194304 4955518 140735454342000 18446744073709551615 4237750 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29937/statm: 35234 29917 106 186 0 35047 0
[pid=29939] ppid=29903 vsize=330512 CPUtime=1.99 cores=1,3,5,7
/proc/29939/stat : 29939 (python) S 29903 29903 28586 0 -1 4202560 1148 0 0 0 104 95 0 0 16 0 2 0 736887217 338444288 5782 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29939/statm: 82628 5782 400 364 0 18531 0
[pid=29941] ppid=29939 vsize=93456 CPUtime=392.38 cores=1,3,5,7
/proc/29941/stat : 29941 (march_hi) T 29939 29941 29941 0 -1 4202496 18519 0 0 0 39234 4 0 0 25 0 1 0 736887217 95698944 18241 33554432000 4194304 4955518 140737241208352 18446744073709551615 4297477 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/29941/statm: 23364 18241 106 186 0 23177 0
[pid=29939/tid=30223] ppid=29903 vsize=330512 CPUtime=0 cores=1,3,5,7
/proc/29939/task/30223/stat : 30223 (python) S 29903 29903 28586 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 736946167 338444288 5782 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604454462878784 0 0 -1 3 0 0 0
[pid=30224] ppid=29903 vsize=340264 CPUtime=1.24 cores=1,3,5,7
/proc/30224/stat : 30224 (python) S 29903 29903 28586 0 -1 4202560 1002 0 0 0 67 57 0 0 15 0 1 0 736946168 348430336 5662 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17795428569787105541 0 0 17 7 0 0 0
/proc/30224/statm: 85066 5662 371 364 0 20966 0
[pid=30226] ppid=30224 vsize=63988 CPUtime=246.2 cores=1,3,5,7
/proc/30226/stat : 30226 (march_hi) R 30224 30226 30226 0 -1 4202496 12188 0 0 0 24618 2 0 0 25 0 1 0 736946178 65523712 11908 33554432000 4194304 4955518 140735377106960 18446744073709551615 4208667 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/30226/statm: 15997 11908 106 186 0 15810 0
[pid=29903/tid=29911] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29911/stat : 29911 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 8 0 736867555 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 3 0 0 0
[pid=29903/tid=29915] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29915/stat : 29915 (python) S 29900 29903 28586 0 -1 4202560 37 934 0 0 0 0 0 0 25 0 8 0 736867556 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29918] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29918/stat : 29918 (python) S 29900 29903 28586 0 -1 4202560 22 934 0 0 0 0 0 0 18 0 8 0 736867557 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29923] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29923/stat : 29923 (python) S 29900 29903 28586 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 8 0 736867559 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29936] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29936/stat : 29936 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 8 0 736886022 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29940] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/29940/stat : 29940 (python) S 29900 29903 28586 0 -1 4202560 21 934 0 0 0 0 0 0 20 0 8 0 736887217 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=30225] ppid=29900 vsize=350508 CPUtime=0 cores=1,3,5,7
/proc/29903/task/30225/stat : 30225 (python) S 29900 29903 28586 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 8 0 736946178 358920192 7183 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3079.87
Current children cumulated vsize (KiB) 5217708

[startup+1158.1 s]
/proc/loadavg: 10.31 10.18 9.57 11/478 30245
/proc/meminfo: memFree=20069016/32950928 swapFree=65730864/67111528
[pid=29903] ppid=29900 vsize=351140 CPUtime=597.33 cores=1,3,5,7
/proc/29903/stat : 29903 (python) S 29900 29903 28586 0 -1 4202496 15370 23871 1 0 48 10 59516 159 17 0 8 0 736867496 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917224255 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/29903/statm: 87785 7309 1888 364 0 23685 0
[pid=29917] ppid=29903 vsize=299600 CPUtime=3.86 cores=1,3,5,7
/proc/29917/stat : 29917 (python) S 29903 29903 28586 0 -1 4202560 1152 0 0 0 201 185 0 0 15 0 2 0 736867557 306790400 5751 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29917/statm: 74900 5751 400 364 0 10812 0
[pid=29920] ppid=29917 vsize=140936 CPUtime=602.26 cores=1,3,5,7
/proc/29920/stat : 29920 (march_hi) R 29917 29920 29920 0 -1 4202496 27209 0 0 0 60221 5 0 0 25 0 1 0 736867558 144318464 26928 33554432000 4194304 4955518 140733227520304 18446744073709551615 4224409 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29920/statm: 35234 26928 106 186 0 35047 0
[pid=29917/tid=30221] ppid=29903 vsize=299600 CPUtime=0 cores=1,3,5,7
/proc/29917/task/30221/stat : 30221 (python) S 29903 29903 28586 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 736940904 306790400 5751 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604468944577664 0 0 -1 3 0 0 0
[pid=29921] ppid=29903 vsize=309856 CPUtime=3.89 cores=1,3,5,7
/proc/29921/stat : 29921 (python) S 29903 29903 28586 0 -1 4202560 1157 0 0 0 206 183 0 0 15 0 2 0 736867559 317292544 5755 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29921/statm: 77464 5755 400 364 0 13373 0
[pid=29924] ppid=29921 vsize=213800 CPUtime=942.52 cores=1,3,5,7
/proc/29924/stat : 29924 (march_hi) R 29921 29924 29924 0 -1 4202496 40937 0 0 0 94243 9 0 0 25 0 1 0 736867559 218931200 40658 33554432000 4194304 4955518 140737051804912 18446744073709551615 4208656 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29924/statm: 53450 40658 106 186 0 53263 0
[pid=29921/tid=30219] ppid=29903 vsize=309856 CPUtime=0 cores=1,3,5,7
/proc/29921/task/30219/stat : 30219 (python) S 29903 29903 28586 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 736939425 317292544 5755 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 514 18446604464836123392 0 0 -1 5 0 0 0
[pid=29934] ppid=29903 vsize=320112 CPUtime=3.34 cores=1,3,5,7
/proc/29934/stat : 29934 (python) S 29903 29903 28586 0 -1 4202560 1155 0 0 0 175 159 0 0 16 0 2 0 736886022 327794688 5759 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917224255 0 0 16781824 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29934/statm: 80028 5759 400 364 0 15934 0
[pid=29937] ppid=29934 vsize=140936 CPUtime=679.69 cores=1,3,5,7
/proc/29937/stat : 29937 (march_hi) R 29934 29937 29937 0 -1 4202496 30199 0 0 0 67963 6 0 0 25 0 1 0 736886022 144318464 29921 33554432000 4194304 4955518 140735454342000 18446744073709551615 4208667 256 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29937/statm: 35234 29921 106 186 0 35047 0
[pid=30224] ppid=29903 vsize=340264 CPUtime=1.24 cores=1,3,5,7
/proc/30224/stat : 30224 (python) S 29903 29903 28586 0 -1 4202560 1002 0 0 0 67 57 0 0 15 0 1 0 736946168 348430336 5662 33554432000 4194304 5683644 140734246027200 18446744073709551615 260905414214 0 0 16781312 514 17795428569787105541 0 0 17 7 0 0 0
/proc/30224/statm: 85066 5662 371 364 0 20966 0
[pid=30226] ppid=30224 vsize=63988 CPUtime=246.3 cores=1,3,5,7
/proc/30226/stat : 30226 (march_hi) R 30224 30226 30226 0 -1 4202496 12192 0 0 0 24628 2 0 0 25 0 1 0 736946178 65523712 11912 33554432000 4194304 4955518 140735377106960 18446744073709551615 4296952 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/30226/statm: 15997 11912 106 186 0 15810 0
[pid=29903/tid=29911] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/29911/stat : 29911 (python) S 29900 29903 28586 0 -1 4202560 26 23871 0 0 0 0 59516 159 25 0 8 0 736867555 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 3 0 0 0
[pid=29903/tid=29915] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/29915/stat : 29915 (python) S 29900 29903 28586 0 -1 4202560 37 23871 0 0 0 0 59516 159 25 0 8 0 736867556 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29918] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/29918/stat : 29918 (python) S 29900 29903 28586 0 -1 4202560 22 23871 0 0 0 0 59516 159 18 0 8 0 736867557 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29923] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/29923/stat : 29923 (python) S 29900 29903 28586 0 -1 4202560 32 23871 0 0 0 0 59516 159 18 0 8 0 736867559 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29936] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/29936/stat : 29936 (python) S 29900 29903 28586 0 -1 4202560 26 23871 0 0 0 0 59516 159 20 0 8 0 736886022 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=29940] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/29940/stat : 29940 (python) S 29900 29903 28586 0 -1 4202560 21 23871 0 0 0 0 59516 159 20 0 8 0 736887217 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
[pid=29903/tid=30225] ppid=29900 vsize=351140 CPUtime=596.75 cores=1,3,5,7
/proc/29903/task/30225/stat : 30225 (python) S 29900 29903 28586 0 -1 4202560 26 23871 0 0 0 0 59516 159 20 0 8 0 736946178 359567360 7309 33554432000 4194304 5683644 140734246027200 18446744073709551615 260917217713 0 0 16781312 2 18446604465368003648 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3080.43
Current children cumulated vsize (KiB) 2180632

Child status: 0
Real time (s): 1158.15
CPU time (s): 3080.61
CPU user time (s): 3072.78
CPU system time (s): 7.82881
CPU usage (%): 265.995
Max. virtual memory (cumulated for all children) (KiB): 5217712

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3072.78
system time used= 7.82881
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 158545
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7829
involuntary context switches= 40225

runsolver used 5.14322 second user time and 11.5952 second system time

The end

Launcher Data 

Begin job on node116 at 2011-05-15 15:32:48
IDJOB=3378613
IDBENCH=83022
IDSOLVER=1859
FILE ID=node116/3378613-1305466368
RUNJOBID= node116-1305463137-28605
PBS_JOBID= 13324982
Free space on /tmp= 73104 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S689381858-022.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378613-1305466368/watcher-3378613-1305466368 -o /tmp/evaluation-result-3378613-1305466368/solver-3378613-1305466368 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378613-1305466368.cnf -seed 1782851059 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e5ccab3a9294cbb9eed5df8c5c797376
RANDOM SEED=1782851059

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      20772588 kB
Buffers:        127028 kB
Cached:        1337936 kB
SwapCached:      12356 kB
Active:       11016508 kB
Inactive:       920528 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      20772588 kB
SwapTotal:    67111528 kB
SwapFree:     65730864 kB
Dirty:           33028 kB
Writeback:           0 kB
AnonPages:    10469868 kB
Mapped:          27336 kB
Slab:           113988 kB
PageTables:      59884 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 116846572 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73076 MiB
End job on node116 at 2011-05-15 15:52:06