Trace number 3378547

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 4719.19 1200.21

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v650-c2769-S510739670-068.UNKNOWN.cnf
MD5SUM12dd7af6f21831da932d15cd8d1054f3
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables650
Number of clauses2769
Sum of the clauses size8307
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32769
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.30/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.47	c INFO: solving HOME/instance-3378547-1305466131.cnf
0.30/0.47	c INFO: computing task features
0.30/0.48	c INFO: computed variable-clause graph statistics
0.30/0.48	c INFO: computed clause balance statistics
0.39/0.52	c INFO: computed variable balance statistics
0.39/0.56	c INFO: computed small-clause counts
0.39/0.56	c INFO: computed Horn-clause counts
0.39/0.57	c INFO: computed variable graph statistics
0.39/0.57	c INFO: collected features for HOME/instance-3378547-1305466131.cnf in 0.10 s
0.39/0.58	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.58)
0.39/0.59	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.28)
0.39/0.60	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.48/0.61	c INFO: running march_hi@0 for 679 with 1199 remaining (b = 0.37)
338.29/97.88	c INFO: running march_hi@0 for 679 with 1102 remaining (b = 0.33)
490.60/136.03	c INFO: running march_hi@0 for 679 with 1064 remaining (b = 0.29)
1551.99/402.24	c INFO: running kcnfs-2006@0 for 582 with 798 remaining (b = 0.25)
2719.48/695.11	c INFO: running march_hi@0 for 505 with 505 remaining (b = 0.20)
3061.99/780.92	c INFO: running march_hi@0 for 419 with 419 remaining (b = 0.15)
3242.14/826.21	c INFO: running march_hi@0 for 374 with 374 remaining (b = 0.10)
3878.89/989.08	c INFO: running march_hi@392 for 97 with 211 remaining (b = 0.04)
4289.45/1092.27	c INFO: running kcnfs-2006@682 for 108 with 108 remaining (b = 0.04)
4718.70/1200.19	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378547-1305466131/watcher-3378547-1305466131 -o /tmp/evaluation-result-3378547-1305466131/solver-3378547-1305466131 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378547-1305466131.cnf -seed 1521998644 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.42 7.00 7.41 3/385 28220
/proc/meminfo: memFree=20620076/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=15912 CPUtime=0 cores=1,3,5,7
/proc/28220/stat : 28220 (python) R 28218 28220 27281 0 -1 4202496 958 0 0 0 0 0 0 0 17 0 1 0 658212804 16293888 834 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647189639 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/28220/statm: 3977 834 357 364 0 464 0

[startup+0.055578 s]
/proc/loadavg: 6.42 7.00 7.41 3/385 28220
/proc/meminfo: memFree=20620076/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=34424 CPUtime=0.03 cores=1,3,5,7
/proc/28220/stat : 28220 (python) R 28218 28220 27281 0 -1 4202496 1552 0 0 0 3 0 0 0 17 0 1 0 658212804 35250176 1388 33554432000 4194304 5683644 140734266678624 18446744073709551615 210642222727 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/28220/statm: 9127 1388 448 364 0 944 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 34424

[startup+0.100524 s]
/proc/loadavg: 6.42 7.00 7.41 3/385 28220
/proc/meminfo: memFree=20620076/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=104224 CPUtime=0.06 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 3050 0 0 0 5 1 0 0 17 0 1 0 658212804 106725376 2734 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 5 0 0 0
/proc/28220/statm: 26056 2734 847 364 0 1804 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104224

[startup+0.300537 s]
/proc/loadavg: 6.42 7.00 7.41 3/385 28220
/proc/meminfo: memFree=20620076/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=170756 CPUtime=0.22 cores=1,3,5,7
/proc/28220/stat : 28220 (python) R 28218 28220 27281 0 -1 4202496 6733 934 1 0 17 5 0 0 17 0 1 0 658212804 174854144 4807 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647144448 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/28220/statm: 42689 4807 1336 364 0 3454 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 170756

[startup+0.700424 s]
/proc/loadavg: 6.42 7.00 7.41 3/385 28220
/proc/meminfo: memFree=20620076/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=320248 CPUtime=0.49 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 12196 934 1 0 41 8 0 0 24 0 5 0 658212804 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/28220/statm: 80062 7251 1873 364 0 15971 0
Current children cumulated CPU time (s) 0.49
Current children cumulated vsize (KiB) 320248

[startup+1.50129 s]
/proc/loadavg: 6.42 7.00 7.41 7/398 28237
/proc/meminfo: memFree=20580136/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=320248 CPUtime=0.49 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 12196 934 1 0 41 8 0 0 24 0 5 0 658212804 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/28220/statm: 80062 7251 1873 364 0 15971 0
[pid=28226] ppid=28220 vsize=278628 CPUtime=0 cores=1,3,5,7
/proc/28226/stat : 28226 (python) S 28220 28220 27281 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 658212862 285315072 5564 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166479 0 0 16781312 514 11468852136876441599 0 0 17 7 0 0 0
/proc/28226/statm: 69657 5564 354 364 0 5575 0
[pid=28228] ppid=28226 vsize=1017940 CPUtime=0.91 cores=1,3,5,7
/proc/28228/stat : 28228 (gnovelty+2) R 28226 28228 28228 0 -1 4202496 458 0 0 0 91 0 0 0 21 0 1 0 658212862 1042370560 191 33554432000 134512640 135663704 4289975040 18446744073709551615 134547346 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28228/statm: 254485 191 113 282 0 254201 0
[pid=28229] ppid=28220 vsize=289392 CPUtime=0 cores=1,3,5,7
/proc/28229/stat : 28229 (python) S 28220 28220 27281 0 -1 4202560 750 0 0 0 0 0 0 0 15 0 1 0 658212863 296337408 5705 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11651361996116295942 0 0 17 7 0 0 0
/proc/28229/statm: 72348 5705 348 364 0 8263 0
[pid=28231] ppid=28229 vsize=2744 CPUtime=0.88 cores=1,3,5,7
/proc/28231/stat : 28231 (march_hi) R 28229 28231 28231 0 -1 4202496 744 0 0 0 88 0 0 0 25 0 1 0 658212863 2809856 477 33554432000 4194304 4955518 140733199528784 18446744073709551615 4227925 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28231/statm: 686 477 104 186 0 499 0
[pid=28232] ppid=28220 vsize=299140 CPUtime=0 cores=1,3,5,7
/proc/28232/stat : 28232 (python) S 28220 28220 27281 0 -1 4202560 734 0 0 0 0 0 0 0 22 0 1 0 658212864 306319360 5598 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11468848288585744383 0 0 17 5 0 0 0
/proc/28232/statm: 74785 5598 348 364 0 10697 0
[pid=28236] ppid=28232 vsize=1017940 CPUtime=0.88 cores=1,3,5,7
/proc/28236/stat : 28236 (gnovelty+2) R 28232 28236 28236 0 -1 4202496 451 0 0 0 88 0 0 0 25 0 1 0 658212865 1042370560 191 33554432000 134512640 135663704 4290339568 18446744073709551615 134545534 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28236/statm: 254485 191 113 282 0 254201 0
[pid=28234] ppid=28220 vsize=310004 CPUtime=0 cores=1,3,5,7
/proc/28234/stat : 28234 (python) S 28220 28220 27281 0 -1 4202560 757 0 0 0 0 0 0 0 17 0 1 0 658212865 317444096 5722 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 8903959515234271495 0 0 17 5 0 0 0
/proc/28234/statm: 77501 5722 348 364 0 13410 0
[pid=28237] ppid=28234 vsize=2540 CPUtime=0.47 cores=1,3,5,7
/proc/28237/stat : 28237 (march_hi) R 28234 28237 28237 0 -1 4202496 701 0 0 0 47 0 0 0 25 0 1 0 658212865 2600960 438 33554432000 4194304 4955518 140737279268896 18446744073709551615 4244449 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28237/statm: 635 438 104 186 0 448 0
[pid=28220/tid=28227] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28227/stat : 28227 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 5 0 658212862 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28230] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28230/stat : 28230 (python) S 28218 28220 27281 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 658212863 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28233] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28233/stat : 28233 (python) S 28218 28220 27281 0 -1 4202560 36 934 0 0 0 0 0 0 23 0 5 0 658212864 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28235] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28235/stat : 28235 (python) S 28218 28220 27281 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 658212865 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3.63
Current children cumulated vsize (KiB) 3538576

[startup+3.10102 s]
/proc/loadavg: 6.38 6.99 7.40 7/398 28237
/proc/meminfo: memFree=20578524/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=320248 CPUtime=0.49 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 12196 934 1 0 41 8 0 0 24 0 5 0 658212804 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/28220/statm: 80062 7251 1873 364 0 15971 0
[pid=28226] ppid=28220 vsize=278628 CPUtime=0 cores=1,3,5,7
/proc/28226/stat : 28226 (python) S 28220 28220 27281 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 658212862 285315072 5564 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166479 0 0 16781312 514 11468852136876409096 0 0 17 7 0 0 0
/proc/28226/statm: 69657 5564 354 364 0 5575 0
[pid=28228] ppid=28226 vsize=1017940 CPUtime=2.51 cores=1,3,5,7
/proc/28228/stat : 28228 (gnovelty+2) R 28226 28228 28228 0 -1 4202496 458 0 0 0 251 0 0 0 25 0 1 0 658212862 1042370560 191 33554432000 134512640 135663704 4289975040 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28228/statm: 254485 191 113 282 0 254201 0
[pid=28229] ppid=28220 vsize=289392 CPUtime=0 cores=1,3,5,7
/proc/28229/stat : 28229 (python) S 28220 28220 27281 0 -1 4202560 750 0 0 0 0 0 0 0 15 0 1 0 658212863 296337408 5705 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11651361996116295942 0 0 17 7 0 0 0
/proc/28229/statm: 72348 5705 348 364 0 8263 0
[pid=28231] ppid=28229 vsize=3176 CPUtime=2.47 cores=1,3,5,7
/proc/28231/stat : 28231 (march_hi) R 28229 28231 28231 0 -1 4202496 815 0 0 0 247 0 0 0 25 0 1 0 658212863 3252224 548 33554432000 4194304 4955518 140733199528784 18446744073709551615 4224615 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28231/statm: 794 548 104 186 0 607 0
[pid=28232] ppid=28220 vsize=299140 CPUtime=0 cores=1,3,5,7
/proc/28232/stat : 28232 (python) S 28220 28220 27281 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 658212864 306319360 5598 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11468848288585711876 0 0 17 5 0 0 0
/proc/28232/statm: 74785 5598 348 364 0 10697 0
[pid=28236] ppid=28232 vsize=1017940 CPUtime=2.48 cores=1,3,5,7
/proc/28236/stat : 28236 (gnovelty+2) R 28232 28236 28236 0 -1 4202496 451 0 0 0 248 0 0 0 25 0 1 0 658212865 1042370560 191 33554432000 134512640 135663704 4290339568 18446744073709551615 134525181 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28236/statm: 254485 191 113 282 0 254201 0
[pid=28234] ppid=28220 vsize=310004 CPUtime=0 cores=1,3,5,7
/proc/28234/stat : 28234 (python) S 28220 28220 27281 0 -1 4202560 757 0 0 0 0 0 0 0 15 0 1 0 658212865 317444096 5722 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 8903959515234271495 0 0 17 5 0 0 0
/proc/28234/statm: 77501 5722 348 364 0 13410 0
[pid=28237] ppid=28234 vsize=2744 CPUtime=1.27 cores=1,3,5,7
/proc/28237/stat : 28237 (march_hi) R 28234 28237 28237 0 -1 4202496 755 0 0 0 127 0 0 0 25 0 1 0 658212865 2809856 492 33554432000 4194304 4955518 140737279268896 18446744073709551615 4224745 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28237/statm: 686 492 104 186 0 499 0
[pid=28220/tid=28227] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28227/stat : 28227 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 5 0 658212862 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28230] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28230/stat : 28230 (python) S 28218 28220 27281 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 658212863 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28233] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28233/stat : 28233 (python) S 28218 28220 27281 0 -1 4202560 36 934 0 0 0 0 0 0 23 0 5 0 658212864 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28235] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28235/stat : 28235 (python) S 28218 28220 27281 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 658212865 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 9.22
Current children cumulated vsize (KiB) 3539212

[startup+6.30048 s]
/proc/loadavg: 6.38 6.99 7.40 7/398 28237
/proc/meminfo: memFree=20574988/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=320248 CPUtime=0.49 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 12196 934 1 0 41 8 0 0 24 0 5 0 658212804 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/28220/statm: 80062 7251 1873 364 0 15971 0
[pid=28226] ppid=28220 vsize=278628 CPUtime=0.01 cores=1,3,5,7
/proc/28226/stat : 28226 (python) S 28220 28220 27281 0 -1 4202560 898 0 0 0 0 1 0 0 18 0 1 0 658212862 285315072 5587 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166479 0 0 16781312 514 11468852136876409096 0 0 17 7 0 0 0
/proc/28226/statm: 69657 5587 377 364 0 5575 0
[pid=28228] ppid=28226 vsize=1017940 CPUtime=5.71 cores=1,3,5,7
/proc/28228/stat : 28228 (gnovelty+2) R 28226 28228 28228 0 -1 4202496 458 0 0 0 571 0 0 0 25 0 1 0 658212862 1042370560 191 33554432000 134512640 135663704 4289975040 18446744073709551615 134525544 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28228/statm: 254485 191 113 282 0 254201 0
[pid=28229] ppid=28220 vsize=289392 CPUtime=0.01 cores=1,3,5,7
/proc/28229/stat : 28229 (python) S 28220 28220 27281 0 -1 4202560 924 0 0 0 1 0 0 0 15 0 1 0 658212863 296337408 5728 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11651361996116295942 0 0 17 7 0 0 0
/proc/28229/statm: 72348 5728 371 364 0 8263 0
[pid=28231] ppid=28229 vsize=3892 CPUtime=4.89 cores=1,3,5,7
/proc/28231/stat : 28231 (march_hi) R 28229 28231 28231 0 -1 4202496 934 0 0 0 489 0 0 0 25 0 1 0 658212863 3985408 667 33554432000 4194304 4955518 140733199528784 18446744073709551615 4286726 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28231/statm: 973 667 104 186 0 786 0
[pid=28232] ppid=28220 vsize=299140 CPUtime=0.01 cores=1,3,5,7
/proc/28232/stat : 28232 (python) S 28220 28220 27281 0 -1 4202560 905 0 0 0 0 1 0 0 18 0 1 0 658212864 306319360 5621 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11468848288585711876 0 0 17 5 0 0 0
/proc/28232/statm: 74785 5621 371 364 0 10697 0
[pid=28236] ppid=28232 vsize=1017940 CPUtime=5.64 cores=1,3,5,7
/proc/28236/stat : 28236 (gnovelty+2) R 28232 28236 28236 0 -1 4202496 451 0 0 0 564 0 0 0 25 0 1 0 658212865 1042370560 191 33554432000 134512640 135663704 4290339568 18446744073709551615 134546128 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28236/statm: 254485 191 113 282 0 254201 0
[pid=28234] ppid=28220 vsize=310004 CPUtime=0.01 cores=1,3,5,7
/proc/28234/stat : 28234 (python) S 28220 28220 27281 0 -1 4202560 918 0 0 0 1 0 0 0 15 0 1 0 658212865 317444096 5745 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 11468844440295014663 0 0 17 5 0 0 0
/proc/28234/statm: 77501 5745 371 364 0 13410 0
[pid=28237] ppid=28234 vsize=3176 CPUtime=2.8 cores=1,3,5,7
/proc/28237/stat : 28237 (march_hi) R 28234 28237 28237 0 -1 4202496 826 0 0 0 280 0 0 0 25 0 1 0 658212865 3252224 563 33554432000 4194304 4955518 140737279268896 18446744073709551615 4224608 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28237/statm: 794 563 104 186 0 607 0
[pid=28220/tid=28227] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28227/stat : 28227 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 5 0 658212862 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28230] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28230/stat : 28230 (python) S 28218 28220 27281 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 658212863 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28233] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28233/stat : 28233 (python) S 28218 28220 27281 0 -1 4202560 36 934 0 0 0 0 0 0 23 0 5 0 658212864 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28235] ppid=28218 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28235/stat : 28235 (python) S 28218 28220 27281 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 658212865 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 19.57
Current children cumulated vsize (KiB) 3540360

[startup+12.7014 s]
/proc/loadavg: 6.56 7.00 7.40 7/398 28237
/proc/meminfo: memFree=20572108/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=320248 CPUtime=0.49 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 12196 934 1 0 41 8 0 0 24 0 5 0 658212804 327933952 7251 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/28220/statm: 80062 7251 1873 364 0 15971 0
[pid=28226] ppid=28220 vsize=278628 CPUtime=0.02 cores=1,3,5,7
/proc/28226/stat : 28226 (python) S 28220 28220 27281 0 -1 4202560 901 0 0 0 1 1 0 0 18 0 1 0 658212862 285315072 5587 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166479 0 0 16781312 514 15868925868249153800 0 0 17 7 0 0 0
/proc/28226/statm: 69657 5587 377 364 0 5575 0
[pid=28228] ppid=28226 vsize=1017940 CPUtime=12.11 cores=1,3,5,7
/proc/28228/stat : 28228 (gnovelty+2) R 28226 28228 28228 0 -1 4202496 458 0 0 0 1211 0 0 0 25 0 1 0 658212862 1042370560 191 33554432000 134512640 135663704 4289975040 18446744073709551615 134525367 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28228/statm: 254485 191 113 282 0 254201 0
[pid=28229] ppid=28220 vsize=289392 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

[pid=28234] ppid=28220 vsize=310004 CPUtime=2.05 cores=1,3,5,7
/proc/28234/stat : 28234 (python) S 28220 28220 27281 0 -1 4202560 1166 0 0 0 103 102 0 0 16 0 2 0 658212865 317444096 5775 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/28234/statm: 77501 5775 400 364 0 13410 0
[pid=28237] ppid=28234 vsize=110300 CPUtime=682.73 cores=1,3,5,7
/proc/28237/stat : 28237 (march_hi) T 28234 28237 28237 0 -1 4202496 24057 0 0 0 68265 8 0 0 25 0 1 0 658212865 112947200 23794 33554432000 4194304 4955518 140737279268896 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28237/statm: 27575 23794 104 186 0 27388 0
[pid=28234/tid=28270] ppid=28220 vsize=310004 CPUtime=0 cores=1,3,5,7
/proc/28234/task/28270/stat : 28270 (python) S 28220 28220 27281 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658282314 317444096 5775 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604459877958976 0 0 -1 3 0 0 0
[pid=28242] ppid=28220 vsize=320260 CPUtime=2.02 cores=1,3,5,7
/proc/28242/stat : 28242 (python) S 28220 28220 27281 0 -1 4202560 1160 0 0 0 98 104 0 0 17 0 2 0 658222593 327946240 5779 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28242/statm: 80065 5779 400 364 0 15971 0
[pid=28244] ppid=28242 vsize=110300 CPUtime=682.66 cores=1,3,5,7
/proc/28244/stat : 28244 (march_hi) T 28242 28244 28244 0 -1 4202496 23986 0 0 0 68261 5 0 0 25 0 1 0 658222593 112947200 23726 33554432000 4194304 4955518 140733285518688 18446744073709551615 4263652 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28244/statm: 27575 23726 104 186 0 27388 0
[pid=28242/tid=28276] ppid=28220 vsize=320260 CPUtime=0 cores=1,3,5,7
/proc/28242/task/28276/stat : 28276 (python) S 28220 28220 27281 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 658290896 327946240 5779 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604466419335616 0 0 -1 5 0 0 0
[pid=28246] ppid=28220 vsize=330156 CPUtime=2.36 cores=1,3,5,7
/proc/28246/stat : 28246 (python) S 28220 28220 27281 0 -1 4202560 1162 0 0 0 121 115 0 0 15 0 2 0 658226407 338079744 5693 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28246/statm: 82539 5693 400 364 0 18442 0
[pid=28248] ppid=28246 vsize=129208 CPUtime=780.76 cores=1,3,5,7
/proc/28248/stat : 28248 (march_hi) T 28246 28248 28248 0 -1 4202496 27080 0 0 0 78072 4 0 0 25 0 1 0 658226407 132308992 26821 33554432000 4194304 4955518 140737350165648 18446744073709551615 4346304 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/28248/statm: 32302 26821 104 186 0 32115 0
[pid=28246/tid=28281] ppid=28220 vsize=330156 CPUtime=0 cores=1,3,5,7
/proc/28246/task/28281/stat : 28281 (python) S 28220 28220 27281 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658295424 338079744 5693 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604470782243200 0 0 -1 7 0 0 0
[pid=28259] ppid=28220 vsize=340952 CPUtime=2 cores=1,3,5,7
/proc/28259/stat : 28259 (python) S 28220 28220 27281 0 -1 4202560 1163 0 0 0 102 98 0 0 18 0 2 0 658253029 349134848 5832 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28259/statm: 85238 5832 400 364 0 21138 0
[pid=28261] ppid=28259 vsize=182580 CPUtime=691.62 cores=1,3,5,7
/proc/28261/stat : 28261 (kcnfs-2006) R 28259 28261 28261 0 -1 4202496 651 0 0 0 69162 0 0 0 25 0 1 0 658253029 186961920 390 33554432000 134512640 135163436 4288437360 18446744073709551615 134642672 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28261/statm: 45645 390 77 159 0 45485 0
[pid=28259/tid=28374] ppid=28220 vsize=340952 CPUtime=0 cores=1,3,5,7
/proc/28259/task/28374/stat : 28374 (python) S 28220 28220 27281 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658311711 349134848 5832 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604454331778112 0 0 -1 5 0 0 0
[pid=28271] ppid=28220 vsize=351208 CPUtime=1.55 cores=1,3,5,7
/proc/28271/stat : 28271 (python) S 28220 28220 27281 0 -1 4202560 1064 0 0 0 82 73 0 0 15 0 1 0 658282315 359636992 5812 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 4299323565364183302 0 0 17 5 0 0 0
/proc/28271/statm: 87802 5812 377 364 0 23699 0
[pid=28273] ppid=28271 vsize=80832 CPUtime=492.88 cores=1,3,5,7
/proc/28273/stat : 28273 (march_hi) R 28271 28273 28273 0 -1 4202496 17401 0 0 0 49225 63 0 0 25 0 1 0 658282315 82771968 17128 33554432000 4194304 4955518 140733557822352 18446744073709551615 4227695 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28273/statm: 20208 17128 104 186 0 20021 0
[pid=28277] ppid=28220 vsize=361068 CPUtime=1.28 cores=1,3,5,7
/proc/28277/stat : 28277 (python) S 28220 28220 27281 0 -1 4202560 1040 0 0 0 63 65 0 0 15 0 1 0 658290897 369733632 5717 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 15868925868249153798 0 0 17 5 0 0 0
/proc/28277/statm: 90267 5717 377 364 0 26161 0
[pid=28279] ppid=28277 vsize=80832 CPUtime=410.71 cores=1,3,5,7
/proc/28279/stat : 28279 (march_hi) R 28277 28279 28279 0 -1 4202496 15089 0 0 0 41067 4 0 0 25 0 1 0 658290897 82771968 14825 33554432000 4194304 4955518 140736439299232 18446744073709551615 4209692 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28279/statm: 20208 14825 104 186 0 20021 0
[pid=28220/tid=28227] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28227/stat : 28227 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 10 0 658212862 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28230] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28230/stat : 28230 (python) S 28218 28220 27281 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 10 0 658212863 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28233] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28233/stat : 28233 (python) S 28218 28220 27281 0 -1 4202560 36 934 0 0 0 0 0 0 23 0 10 0 658212864 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28235] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28235/stat : 28235 (python) S 28218 28220 27281 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 10 0 658212865 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28243] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28243/stat : 28243 (python) S 28218 28220 27281 0 -1 4202560 29 934 0 0 0 0 0 0 20 0 10 0 658222593 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28247] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28247/stat : 28247 (python) S 28218 28220 27281 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 10 0 658226407 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28260] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28260/stat : 28260 (python) S 28218 28220 27281 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 10 0 658253029 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28272] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28272/stat : 28272 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 10 0 658282315 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28278] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28278/stat : 28278 (python) S 28218 28220 27281 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 10 0 658290897 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4717.88
Current children cumulated vsize (KiB) 6122268

[startup+1200.12 s]
/proc/loadavg: 8.31 6.86 6.27 9/436 28388
/proc/meminfo: memFree=19797004/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=372076 CPUtime=0.57 cores=1,3,5,7
/proc/28220/stat : 28220 (python) S 28218 28220 27281 0 -1 4202496 17045 934 1 0 48 9 0 0 18 0 10 0 658212804 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/28220/statm: 93019 7427 1892 364 0 28913 0
[pid=28226] ppid=28220 vsize=288872 CPUtime=0.28 cores=1,3,5,7
/proc/28226/stat : 28226 (python) S 28220 28220 27281 0 -1 4202560 1165 0 0 0 16 12 0 0 19 0 2 0 658212862 295804928 5631 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28226/statm: 72218 5631 416 364 0 8136 0
[pid=28228] ppid=28226 vsize=1017940 CPUtime=97.29 cores=1,3,5,7
/proc/28228/stat : 28228 (gnovelty+2) T 28226 28228 28228 0 -1 4202496 458 0 0 0 9729 0 0 0 25 0 1 0 658212862 1042370560 191 33554432000 134512640 135663704 4289975040 18446744073709551615 134525187 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28228/statm: 254485 191 113 282 0 254201 0
[pid=28226/tid=28241] ppid=28220 vsize=288872 CPUtime=0 cores=1,3,5,7
/proc/28226/task/28241/stat : 28241 (python) S 28220 28220 27281 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 658222592 295804928 5631 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604459877960768 0 0 -1 7 0 0 0
[pid=28229] ppid=28220 vsize=289392 CPUtime=2.32 cores=1,3,5,7
/proc/28229/stat : 28229 (python) S 28220 28220 27281 0 -1 4202560 1161 0 0 0 120 112 0 0 15 0 2 0 658212863 296337408 5758 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/28229/statm: 72348 5758 400 364 0 8263 0
[pid=28231] ppid=28229 vsize=129208 CPUtime=763.78 cores=1,3,5,7
/proc/28231/stat : 28231 (march_hi) R 28229 28231 28231 0 -1 4202496 26685 0 0 0 76372 6 0 0 25 0 1 0 658212863 132308992 26418 33554432000 4194304 4955518 140733199528784 18446744073709551615 4227044 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/28231/statm: 32302 26418 104 186 0 32115 0
[pid=28229/tid=28258] ppid=28220 vsize=289392 CPUtime=0 cores=1,3,5,7
/proc/28229/task/28258/stat : 28258 (python) S 28220 28220 27281 0 -1 4202560 46 0 0 0 0 0 0 0 17 0 2 0 658253028 296337408 5758 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604465687817024 0 0 -1 7 0 0 0
[pid=28232] ppid=28220 vsize=299140 CPUtime=0.38 cores=1,3,5,7
/proc/28232/stat : 28232 (python) S 28220 28220 27281 0 -1 4202560 1098 0 0 0 20 18 0 0 19 0 2 0 658212864 306319360 5651 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28232/statm: 74785 5651 400 364 0 10697 0
[pid=28236] ppid=28232 vsize=1017940 CPUtime=100.85 cores=1,3,5,7
/proc/28236/stat : 28236 (gnovelty+2) T 28232 28236 28236 0 -1 4202496 451 0 0 0 10085 0 0 0 25 0 1 0 658212865 1042370560 191 33554432000 134512640 135663704 4290339568 18446744073709551615 134545525 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28236/statm: 254485 191 113 282 0 254201 0
[pid=28232/tid=28245] ppid=28220 vsize=299140 CPUtime=0 cores=1,3,5,7
/proc/28232/task/28245/stat : 28245 (python) S 28220 28220 27281 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 658226406 306319360 5651 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604459877959872 0 0 -1 1 0 0 0
[pid=28234] ppid=28220 vsize=310004 CPUtime=2.05 cores=1,3,5,7
/proc/28234/stat : 28234 (python) S 28220 28220 27281 0 -1 4202560 1166 0 0 0 103 102 0 0 16 0 2 0 658212865 317444096 5775 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/28234/statm: 77501 5775 400 364 0 13410 0
[pid=28237] ppid=28234 vsize=110300 CPUtime=682.73 cores=1,3,5,7
/proc/28237/stat : 28237 (march_hi) T 28234 28237 28237 0 -1 4202496 24057 0 0 0 68265 8 0 0 25 0 1 0 658212865 112947200 23794 33554432000 4194304 4955518 140737279268896 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/28237/statm: 27575 23794 104 186 0 27388 0
[pid=28234/tid=28270] ppid=28220 vsize=310004 CPUtime=0 cores=1,3,5,7
/proc/28234/task/28270/stat : 28270 (python) S 28220 28220 27281 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658282314 317444096 5775 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604459877958976 0 0 -1 3 0 0 0
[pid=28242] ppid=28220 vsize=320260 CPUtime=2.02 cores=1,3,5,7
/proc/28242/stat : 28242 (python) S 28220 28220 27281 0 -1 4202560 1160 0 0 0 98 104 0 0 17 0 2 0 658222593 327946240 5779 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28242/statm: 80065 5779 400 364 0 15971 0
[pid=28244] ppid=28242 vsize=110300 CPUtime=682.66 cores=1,3,5,7
/proc/28244/stat : 28244 (march_hi) T 28242 28244 28244 0 -1 4202496 23986 0 0 0 68261 5 0 0 25 0 1 0 658222593 112947200 23726 33554432000 4194304 4955518 140733285518688 18446744073709551615 4263652 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/28244/statm: 27575 23726 104 186 0 27388 0
[pid=28242/tid=28276] ppid=28220 vsize=320260 CPUtime=0 cores=1,3,5,7
/proc/28242/task/28276/stat : 28276 (python) S 28220 28220 27281 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 658290896 327946240 5779 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604466419335616 0 0 -1 5 0 0 0
[pid=28246] ppid=28220 vsize=330156 CPUtime=2.36 cores=1,3,5,7
/proc/28246/stat : 28246 (python) S 28220 28220 27281 0 -1 4202560 1162 0 0 0 121 115 0 0 15 0 2 0 658226407 338079744 5693 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28246/statm: 82539 5693 400 364 0 18442 0
[pid=28248] ppid=28246 vsize=129208 CPUtime=780.76 cores=1,3,5,7
/proc/28248/stat : 28248 (march_hi) T 28246 28248 28248 0 -1 4202496 27080 0 0 0 78072 4 0 0 25 0 1 0 658226407 132308992 26821 33554432000 4194304 4955518 140737350165648 18446744073709551615 4346304 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/28248/statm: 32302 26821 104 186 0 32115 0
[pid=28246/tid=28281] ppid=28220 vsize=330156 CPUtime=0 cores=1,3,5,7
/proc/28246/task/28281/stat : 28281 (python) S 28220 28220 27281 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658295424 338079744 5693 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604470782243200 0 0 -1 7 0 0 0
[pid=28259] ppid=28220 vsize=340952 CPUtime=2 cores=1,3,5,7
/proc/28259/stat : 28259 (python) S 28220 28220 27281 0 -1 4202560 1163 0 0 0 102 98 0 0 18 0 2 0 658253029 349134848 5832 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/28259/statm: 85238 5832 400 364 0 21138 0
[pid=28261] ppid=28259 vsize=182580 CPUtime=691.83 cores=1,3,5,7
/proc/28261/stat : 28261 (kcnfs-2006) R 28259 28261 28261 0 -1 4202496 651 0 0 0 69183 0 0 0 25 0 1 0 658253029 186961920 390 33554432000 134512640 135163436 4288437360 18446744073709551615 134675119 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/28261/statm: 45645 390 77 159 0 45485 0
[pid=28259/tid=28374] ppid=28220 vsize=340952 CPUtime=0 cores=1,3,5,7
/proc/28259/task/28374/stat : 28374 (python) S 28220 28220 27281 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658311711 349134848 5832 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 514 18446604454331778112 0 0 -1 5 0 0 0
[pid=28271] ppid=28220 vsize=351208 CPUtime=1.55 cores=1,3,5,7
/proc/28271/stat : 28271 (python) S 28220 28220 27281 0 -1 4202560 1064 0 0 0 82 73 0 0 15 0 1 0 658282315 359636992 5812 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 4299323565364183302 0 0 17 5 0 0 0
/proc/28271/statm: 87802 5812 377 364 0 23699 0
[pid=28273] ppid=28271 vsize=80832 CPUtime=493.09 cores=1,3,5,7
/proc/28273/stat : 28273 (march_hi) R 28271 28273 28273 0 -1 4202496 17409 0 0 0 49246 63 0 0 25 0 1 0 658282315 82771968 17136 33554432000 4194304 4955518 140733557822352 18446744073709551615 4227565 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/28273/statm: 20208 17136 104 186 0 20021 0
[pid=28277] ppid=28220 vsize=361068 CPUtime=1.28 cores=1,3,5,7
/proc/28277/stat : 28277 (python) S 28220 28220 27281 0 -1 4202560 1040 0 0 0 63 65 0 0 15 0 1 0 658290897 369733632 5717 33554432000 4194304 5683644 140734266678624 18446744073709551615 210647166534 0 0 16781312 514 15868925868249153798 0 0 17 5 0 0 0
/proc/28277/statm: 90267 5717 377 364 0 26161 0
[pid=28279] ppid=28277 vsize=80832 CPUtime=410.9 cores=1,3,5,7
/proc/28279/stat : 28279 (march_hi) R 28277 28279 28279 0 -1 4202496 15094 0 0 0 41086 4 0 0 25 0 1 0 658290897 82771968 14830 33554432000 4194304 4955518 140736439299232 18446744073709551615 4244813 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/28279/statm: 20208 14830 104 186 0 20021 0
[pid=28220/tid=28227] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28227/stat : 28227 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 10 0 658212862 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28230] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28230/stat : 28230 (python) S 28218 28220 27281 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 10 0 658212863 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 7 0 0 0
[pid=28220/tid=28233] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28233/stat : 28233 (python) S 28218 28220 27281 0 -1 4202560 36 934 0 0 0 0 0 0 23 0 10 0 658212864 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28235] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28235/stat : 28235 (python) S 28218 28220 27281 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 10 0 658212865 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28243] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28243/stat : 28243 (python) S 28218 28220 27281 0 -1 4202560 29 934 0 0 0 0 0 0 20 0 10 0 658222593 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28247] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28247/stat : 28247 (python) S 28218 28220 27281 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 10 0 658226407 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28260] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28260/stat : 28260 (python) S 28218 28220 27281 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 10 0 658253029 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28272] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28272/stat : 28272 (python) S 28218 28220 27281 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 10 0 658282315 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
[pid=28220/tid=28278] ppid=28218 vsize=372076 CPUtime=0 cores=1,3,5,7
/proc/28220/task/28278/stat : 28278 (python) S 28218 28220 27281 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 10 0 658290897 381005824 7427 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 2 18446604459355959680 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4718.7
Current children cumulated vsize (KiB) 6122268

[startup+1200.2 s]
/proc/loadavg: 8.31 6.86 6.27 9/436 28388
/proc/meminfo: memFree=19797004/32951132 swapFree=64495980/67111528
[pid=28220] ppid=28218 vsize=320736 CPUtime=4719.17 cores=1,3,5,7
/proc/28220/stat : 28220 (python) R 28218 28220 27281 0 -1 4202496 19565 147266 1 0 49 10 471056 802 17 0 2 0 658212804 328433664 7389 33554432000 4194304 5683644 140734266678624 18446744073709551615 4619677 0 0 16781312 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28220/statm: 80184 7389 1874 364 0 16108 0
[pid=28220/tid=28272] ppid=28218 vsize=320736 CPUtime=4718.58 cores=1,3,5,7
/proc/28220/task/28272/stat : 28272 (python) S 28218 28220 27281 0 -1 4202560 27 147266 0 0 0 0 471056 802 18 0 2 0 658282315 328433664 7389 33554432000 4194304 5683644 140734266678624 18446744073709551615 210658970033 0 0 16781312 0 18446604459355959680 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4719.17
Current children cumulated vsize (KiB) 320736

Child status: 0
Real time (s): 1200.21
CPU time (s): 4719.19
CPU user time (s): 4711.06
CPU system time (s): 8.12476
CPU usage (%): 393.196
Max. virtual memory (cumulated for all children) (KiB): 6122268

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4711.06
system time used= 8.12476
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 168369
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7600
involuntary context switches= 25058

runsolver used 5.52616 second user time and 13.5279 second system time

The end

Launcher Data 

Begin job on node129 at 2011-05-15 15:28:52
IDJOB=3378547
IDBENCH=82956
IDSOLVER=1859
FILE ID=node129/3378547-1305466131
RUNJOBID= node129-1305463314-27300
PBS_JOBID= 13324993
Free space on /tmp= 73364 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v650-c2769-S510739670-068.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378547-1305466131/watcher-3378547-1305466131 -o /tmp/evaluation-result-3378547-1305466131/solver-3378547-1305466131 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378547-1305466131.cnf -seed 1521998644 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 12dd7af6f21831da932d15cd8d1054f3
RANDOM SEED=1521998644

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      20620108 kB
Buffers:        270432 kB
Cached:        6414176 kB
SwapCached:      52668 kB
Active:        3569884 kB
Inactive:      8519512 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      20620108 kB
SwapTotal:    67111528 kB
SwapFree:     64495980 kB
Dirty:           32172 kB
Writeback:           0 kB
AnonPages:     5360700 kB
Mapped:          26780 kB
Slab:           134376 kB
PageTables:      42092 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 93514800 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73332 MiB
End job on node129 at 2011-05-15 15:48:52