Trace number 3378435

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 4775.79 1199.13

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S121099030-067.UNKNOWN.cnf
MD5SUMb936bd7aa1624f527379bba2a9d6f8ed
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.36/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.67	c INFO: solving HOME/instance-3378435-1305465446.cnf
0.36/0.67	c INFO: computing task features
0.36/0.68	c INFO: computed variable-clause graph statistics
0.36/0.68	c INFO: computed clause balance statistics
0.44/0.75	c INFO: computed variable balance statistics
0.44/0.78	c INFO: computed small-clause counts
0.44/0.78	c INFO: computed Horn-clause counts
0.44/0.79	c INFO: computed variable graph statistics
0.44/0.79	c INFO: collected features for HOME/instance-3378435-1305465446.cnf in 0.12 s
0.55/0.80	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.63)
0.55/0.81	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.08)
0.55/0.82	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.29)
0.55/0.83	c INFO: running march_hi@0 for 679 with 1199 remaining (b = 0.36)
396.14/100.16	c INFO: running march_hi@0 for 679 with 1100 remaining (b = 0.32)
400.11/101.16	c INFO: running march_hi@0 for 388 with 1099 remaining (b = 0.21)
1559.81/392.17	c INFO: running march_hi@0 for 97 with 808 remaining (b = 0.06)
1948.78/489.71	c INFO: running march_hi@97 for 97 with 710 remaining (b = 0.06)
1987.85/499.55	c INFO: running march_hi@391 for 97 with 701 remaining (b = 0.06)
2331.41/585.76	c INFO: running march_hi@0 for 485 with 614 remaining (b = 0.21)
2377.26/597.26	c INFO: running kcnfs-2006@0 for 603 with 603 remaining (b = 0.21)
2741.52/688.64	c INFO: running march_hi@98 for 291 with 512 remaining (b = 0.13)
3110.09/781.14	c INFO: running march_hi@98 for 291 with 419 remaining (b = 0.13)
3897.48/978.73	c INFO: running march_hi@488 for 221 with 221 remaining (b = 0.10)
4289.95/1077.24	c INFO: running march_hi@386 for 123 with 123 remaining (b = 0.07)
4309.47/1082.14	c INFO: running march_hi@97 for 118 with 118 remaining (b = 0.04)
4775.69/1199.10	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378435-1305465446/watcher-3378435-1305465446 -o /tmp/evaluation-result-3378435-1305465446/solver-3378435-1305465446 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378435-1305465446.cnf -seed 1000761321 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.15 8.25 7.86 7/507 9205
/proc/meminfo: memFree=21187232/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/9205/stat : 9205 (python) R 9203 9205 8588 0 -1 4202496 891 0 0 0 0 0 0 0 25 0 1 0 658145002 16154624 774 33554432000 4194304 5683644 140734946911744 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/9205/statm: 3944 775 343 364 0 431 0

[startup+0.035332 s]
/proc/loadavg: 8.15 8.25 7.86 7/507 9205
/proc/meminfo: memFree=21187232/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=16812 CPUtime=0.01 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 1228 0 0 0 1 0 0 0 25 0 1 0 658145002 17215488 1087 33554432000 4194304 5683644 140734946911744 18446744073709551615 253635935879 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/9205/statm: 4203 1087 418 364 0 690 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16812

[startup+0.100343 s]
/proc/loadavg: 8.15 8.25 7.86 7/507 9205
/proc/meminfo: memFree=21187232/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=89768 CPUtime=0.06 cores=1,3,5,7
/proc/9205/stat : 9205 (python) R 9203 9205 8588 0 -1 4202496 2642 0 0 0 5 1 0 0 17 0 1 0 658145002 91922432 2359 33554432000 4194304 5683644 140734946911744 18446744073709551615 47768037609648 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/9205/statm: 22442 2363 725 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89768

[startup+0.300308 s]
/proc/loadavg: 8.15 8.25 7.86 7/507 9205
/proc/meminfo: memFree=21187232/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=135748 CPUtime=0.18 cores=1,3,5,7
/proc/9205/stat : 9205 (python) R 9203 9205 8588 0 -1 4202496 5966 935 1 0 13 5 0 0 17 0 1 0 658145002 139005952 4151 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640855557 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/9205/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.700228 s]
/proc/loadavg: 8.15 8.25 7.86 7/507 9205
/proc/meminfo: memFree=21187232/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=278524 CPUtime=0.44 cores=1,3,5,7
/proc/9205/stat : 9205 (python) R 9203 9205 8588 0 -1 4202496 9362 935 1 0 34 10 0 0 15 0 1 0 658145002 285208576 7029 33554432000 4194304 5683644 140734946911744 18446744073709551615 47768076799213 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/9205/statm: 69631 7029 1826 364 0 5555 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 278524

[startup+1.50103 s]
/proc/loadavg: 8.15 8.25 7.86 9/520 9222
/proc/meminfo: memFree=21147696/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=320096 CPUtime=0.57 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 12238 935 1 0 46 11 0 0 22 0 5 0 658145002 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9205/statm: 80024 7230 1873 364 0 15933 0
[pid=9211] ppid=9205 vsize=278548 CPUtime=0 cores=1,3,5,7
/proc/9211/stat : 9211 (python) S 9205 9205 8588 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 658145083 285233152 5557 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879631 0 0 16781312 514 11948702100982071295 0 0 17 7 0 0 0
/proc/9211/statm: 69637 5557 354 364 0 5555 0
[pid=9213] ppid=9211 vsize=1017800 CPUtime=0.69 cores=1,3,5,7
/proc/9213/stat : 9213 (gnovelty+2) R 9211 9213 9213 0 -1 4202496 443 0 0 0 69 0 0 0 21 0 1 0 658145083 1042227200 180 33554432000 134512640 135663704 4288278688 18446744073709551615 134524945 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9213/statm: 254450 180 113 282 0 254166 0
[pid=9214] ppid=9205 vsize=289340 CPUtime=0 cores=1,3,5,7
/proc/9214/stat : 9214 (python) S 9205 9205 8588 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 658145084 296284160 5693 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18304335606852878335 0 0 17 5 0 0 0
/proc/9214/statm: 72335 5693 348 364 0 8250 0
[pid=9216] ppid=9214 vsize=1017800 CPUtime=0.66 cores=1,3,5,7
/proc/9216/stat : 9216 (gnovelty+2) R 9214 9216 9216 0 -1 4202496 446 0 0 0 66 0 0 0 21 0 1 0 658145084 1042227200 180 33554432000 134512640 135663704 4287275680 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9216/statm: 254450 180 113 282 0 254166 0
[pid=9217] ppid=9205 vsize=299596 CPUtime=0 cores=1,3,5,7
/proc/9217/stat : 9217 (python) S 9205 9205 8588 0 -1 4202560 752 0 0 0 0 0 0 0 15 0 1 0 658145085 306786304 5697 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 16507794904840110342 0 0 17 1 0 0 0
/proc/9217/statm: 74899 5697 348 364 0 10811 0
[pid=9219] ppid=9217 vsize=1988 CPUtime=0.67 cores=1,3,5,7
/proc/9219/stat : 9219 (march_hi) R 9217 9219 9219 0 -1 4202496 594 0 0 0 67 0 0 0 22 0 1 0 658145085 2035712 330 33554432000 4194304 4955518 140734270727392 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9219/statm: 497 330 106 186 0 310 0
[pid=9220] ppid=9205 vsize=309852 CPUtime=0 cores=1,3,5,7
/proc/9220/stat : 9220 (python) S 9205 9205 8588 0 -1 4202560 762 0 0 0 0 0 0 0 18 0 1 0 658145086 317288448 5701 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 15704685223633453319 0 0 17 5 0 0 0
/proc/9220/statm: 77463 5701 348 364 0 13372 0
[pid=9222] ppid=9220 vsize=1988 CPUtime=0.66 cores=1,3,5,7
/proc/9222/stat : 9222 (march_hi) R 9220 9222 9222 0 -1 4202496 592 0 0 0 66 0 0 0 25 0 1 0 658145086 2035712 329 33554432000 4194304 4955518 140734430443904 18446744073709551615 4224471 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9222/statm: 497 329 106 186 0 310 0
[pid=9205/tid=9212] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9212/stat : 9212 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 5 0 658145083 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 7 0 0 0
[pid=9205/tid=9215] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9215/stat : 9215 (python) S 9203 9205 8588 0 -1 4202560 20 935 0 0 0 0 0 0 19 0 5 0 658145084 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9218] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9218/stat : 9218 (python) S 9203 9205 8588 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 5 0 658145085 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9221] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9221/stat : 9221 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 5 0 658145086 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3.25
Current children cumulated vsize (KiB) 3537008

[startup+3.10074 s]
/proc/loadavg: 8.15 8.25 7.86 9/520 9222
/proc/meminfo: memFree=21146084/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=320096 CPUtime=0.57 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 12238 935 1 0 46 11 0 0 22 0 5 0 658145002 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9205/statm: 80024 7230 1873 364 0 15933 0
[pid=9211] ppid=9205 vsize=278548 CPUtime=0 cores=1,3,5,7
/proc/9211/stat : 9211 (python) S 9205 9205 8588 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 658145083 285233152 5557 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879631 0 0 16781312 514 11948702100982038791 0 0 17 7 0 0 0
/proc/9211/statm: 69637 5557 354 364 0 5555 0
[pid=9213] ppid=9211 vsize=1017800 CPUtime=2.29 cores=1,3,5,7
/proc/9213/stat : 9213 (gnovelty+2) R 9211 9213 9213 0 -1 4202496 443 0 0 0 229 0 0 0 25 0 1 0 658145083 1042227200 180 33554432000 134512640 135663704 4288278688 18446744073709551615 134545727 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9213/statm: 254450 180 113 282 0 254166 0
[pid=9214] ppid=9205 vsize=289340 CPUtime=0 cores=1,3,5,7
/proc/9214/stat : 9214 (python) S 9205 9205 8588 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 658145084 296284160 5693 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18304335606852845830 0 0 17 5 0 0 0
/proc/9214/statm: 72335 5693 348 364 0 8250 0
[pid=9216] ppid=9214 vsize=1017800 CPUtime=2.24 cores=1,3,5,7
/proc/9216/stat : 9216 (gnovelty+2) R 9214 9216 9216 0 -1 4202496 446 0 0 0 224 0 0 0 25 0 1 0 658145084 1042227200 180 33554432000 134512640 135663704 4287275680 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9216/statm: 254450 180 113 282 0 254166 0
[pid=9217] ppid=9205 vsize=299596 CPUtime=0 cores=1,3,5,7
/proc/9217/stat : 9217 (python) S 9205 9205 8588 0 -1 4202560 754 0 0 0 0 0 0 0 15 0 1 0 658145085 306786304 5697 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 16507794904840110342 0 0 17 1 0 0 0
/proc/9217/statm: 74899 5697 348 364 0 10811 0
[pid=9219] ppid=9217 vsize=2468 CPUtime=2.27 cores=1,3,5,7
/proc/9219/stat : 9219 (march_hi) R 9217 9219 9219 0 -1 4202496 692 0 0 0 227 0 0 0 25 0 1 0 658145085 2527232 428 33554432000 4194304 4955518 140734270727392 18446744073709551615 4297231 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9219/statm: 617 428 106 186 0 430 0
[pid=9220] ppid=9205 vsize=309852 CPUtime=0 cores=1,3,5,7
/proc/9220/stat : 9220 (python) S 9205 9205 8588 0 -1 4202560 764 0 0 0 0 0 0 0 15 0 1 0 658145086 317288448 5701 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 15704685223633453319 0 0 17 5 0 0 0
/proc/9220/statm: 77463 5701 348 364 0 13372 0
[pid=9222] ppid=9220 vsize=2468 CPUtime=2.25 cores=1,3,5,7
/proc/9222/stat : 9222 (march_hi) R 9220 9222 9222 0 -1 4202496 690 0 0 0 225 0 0 0 25 0 1 0 658145086 2527232 427 33554432000 4194304 4955518 140734430443904 18446744073709551615 4297483 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9222/statm: 617 427 106 186 0 430 0
[pid=9205/tid=9212] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9212/stat : 9212 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 5 0 658145083 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 7 0 0 0
[pid=9205/tid=9215] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9215/stat : 9215 (python) S 9203 9205 8588 0 -1 4202560 20 935 0 0 0 0 0 0 19 0 5 0 658145084 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9218] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9218/stat : 9218 (python) S 9203 9205 8588 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 5 0 658145085 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9221] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9221/stat : 9221 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 5 0 658145086 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 9.62
Current children cumulated vsize (KiB) 3537968

[startup+6.30116 s]
/proc/loadavg: 8.30 8.28 7.87 9/520 9222
/proc/meminfo: memFree=21142600/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=320096 CPUtime=0.57 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 12238 935 1 0 46 11 0 0 22 0 5 0 658145002 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9205/statm: 80024 7230 1873 364 0 15933 0
[pid=9211] ppid=9205 vsize=278548 CPUtime=0.01 cores=1,3,5,7
/proc/9211/stat : 9211 (python) S 9205 9205 8588 0 -1 4202560 897 0 0 0 0 1 0 0 18 0 1 0 658145083 285233152 5580 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879631 0 0 16781312 514 11948702100982038791 0 0 17 7 0 0 0
/proc/9211/statm: 69637 5580 377 364 0 5555 0
[pid=9213] ppid=9211 vsize=1017800 CPUtime=5.47 cores=1,3,5,7
/proc/9213/stat : 9213 (gnovelty+2) R 9211 9213 9213 0 -1 4202496 443 0 0 0 547 0 0 0 25 0 1 0 658145083 1042227200 180 33554432000 134512640 135663704 4288278688 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9213/statm: 254450 180 113 282 0 254166 0
[pid=9214] ppid=9205 vsize=289340 CPUtime=0.02 cores=1,3,5,7
/proc/9214/stat : 9214 (python) S 9205 9205 8588 0 -1 4202560 905 0 0 0 1 1 0 0 18 0 1 0 658145084 296284160 5716 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18304335606852845830 0 0 17 5 0 0 0
/proc/9214/statm: 72335 5716 371 364 0 8250 0
[pid=9216] ppid=9214 vsize=1017800 CPUtime=5.37 cores=1,3,5,7
/proc/9216/stat : 9216 (gnovelty+2) R 9214 9216 9216 0 -1 4202496 446 0 0 0 537 0 0 0 25 0 1 0 658145084 1042227200 180 33554432000 134512640 135663704 4287275680 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9216/statm: 254450 180 113 282 0 254166 0
[pid=9217] ppid=9205 vsize=299596 CPUtime=0.01 cores=1,3,5,7
/proc/9217/stat : 9217 (python) S 9205 9205 8588 0 -1 4202560 917 0 0 0 0 1 0 0 15 0 1 0 658145085 306786304 5719 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 16507794904840110342 0 0 17 1 0 0 0
/proc/9217/statm: 74899 5719 370 364 0 10811 0
[pid=9219] ppid=9217 vsize=3320 CPUtime=5.47 cores=1,3,5,7
/proc/9219/stat : 9219 (march_hi) R 9217 9219 9219 0 -1 4202496 863 0 0 0 547 0 0 0 25 0 1 0 658145085 3399680 599 33554432000 4194304 4955518 140734270727392 18446744073709551615 4307259 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9219/statm: 830 599 106 186 0 643 0
[pid=9220] ppid=9205 vsize=309852 CPUtime=0.02 cores=1,3,5,7
/proc/9220/stat : 9220 (python) S 9205 9205 8588 0 -1 4202560 924 0 0 0 1 1 0 0 15 0 1 0 658145086 317288448 5724 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 15704685223633453319 0 0 17 5 0 0 0
/proc/9220/statm: 77463 5724 371 364 0 13372 0
[pid=9222] ppid=9220 vsize=3320 CPUtime=5.41 cores=1,3,5,7
/proc/9222/stat : 9222 (march_hi) R 9220 9222 9222 0 -1 4202496 859 0 0 0 541 0 0 0 25 0 1 0 658145086 3399680 596 33554432000 4194304 4955518 140734430443904 18446744073709551615 4208710 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9222/statm: 830 596 106 186 0 643 0
[pid=9205/tid=9212] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9212/stat : 9212 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 5 0 658145083 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 7 0 0 0
[pid=9205/tid=9215] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9215/stat : 9215 (python) S 9203 9205 8588 0 -1 4202560 20 935 0 0 0 0 0 0 19 0 5 0 658145084 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9218] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9218/stat : 9218 (python) S 9203 9205 8588 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 5 0 658145085 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9221] ppid=9203 vsize=320096 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9221/stat : 9221 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 5 0 658145086 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 22.35
Current children cumulated vsize (KiB) 3539672

[startup+12.701 s]
/proc/loadavg: 8.28 8.27 7.87 9/521 9223
/proc/meminfo: memFree=21138780/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=320096 CPUtime=0.57 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 12238 935 1 0 46 11 0 0 22 0 5 0 658145002 327778304 7230 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9205/statm: 80024 7230 1873 364 0 15933 0
[pid=9211] ppid=9205 vsize=278548 CPUtime=0.03 cores=1,3,5,7
/proc/9211/stat : 9211 (python) S 9205 9205 8588 0 -1 4202560 901 0 0 0 1 2 0 0 18 0 1 0 658145083 285233152 5580 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879631 0 0 16781312 514 16700526094754742535 0 0 17 7 0 0 0
/proc/9211/statm: 69637 5580 377 364 0 5555 0
[pid=9213] ppid=9211 vsize=1017800 CPUtime=11.84 cores=1,3,5,7
/proc/9213/stat : 9213 (gnovelty+2) R 9211 9213 9213 0 -1 4202496 443 0 0 0 1184 0 0 0 25 0 1 0 658145083 1042227200 180 33554432000 134512640 135663704 4288278688 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9213/statm: 254450 180 113 282 0 254166 0
[pid=9214] ppid=9205 vsize=289340 CPUtime=0.03 cores=1,3,5,7

################
# More data... #
################

/proc/9233/stat : 9233 (march_hi) R 9231 9233 9233 0 -1 4202496 46279 0 0 0 109526 11 0 0 25 0 1 0 658155020 218877952 46016 33554432000 4194304 4955518 140736535004544 18446744073709551615 4224450 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9233/statm: 53437 46016 106 186 0 53250 0
[pid=9231/tid=9947] ppid=9205 vsize=320244 CPUtime=0 cores=1,3,5,7
/proc/9231/task/9947/stat : 9947 (python) S 9205 9205 8588 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 658223116 327929856 5775 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604469284524992 0 0 -1 5 0 0 0
[pid=9235] ppid=9205 vsize=330500 CPUtime=1.78 cores=1,3,5,7
/proc/9235/stat : 9235 (python) S 9205 9205 8588 0 -1 4202560 1162 0 0 0 95 83 0 0 15 0 2 0 658155119 338432000 5781 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9235/statm: 82625 5781 400 364 0 18528 0
[pid=9237] ppid=9235 vsize=96720 CPUtime=488.2 cores=1,3,5,7
/proc/9237/stat : 9237 (march_hi) T 9235 9237 9237 0 -1 4202496 21886 0 0 0 48814 6 0 0 25 0 1 0 658155119 99041280 21625 33554432000 4194304 4955518 140736921910336 18446744073709551615 4238064 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/9237/statm: 24180 21625 106 186 0 23993 0
[pid=9235/tid=9389] ppid=9205 vsize=330500 CPUtime=0 cores=1,3,5,7
/proc/9235/task/9389/stat : 9389 (python) S 9205 9205 8588 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658194957 338432000 5781 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604465796436224 0 0 -1 7 0 0 0
[pid=9929] ppid=9205 vsize=340936 CPUtime=2.17 cores=1,3,5,7
/proc/9929/stat : 9929 (python) S 9205 9205 8588 0 -1 4202560 1173 0 0 0 114 103 0 0 15 0 2 0 658203579 349118464 5828 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9929/statm: 85234 5828 400 364 0 21134 0
[pid=9931] ppid=9929 vsize=140920 CPUtime=605.9 cores=1,3,5,7
/proc/9931/stat : 9931 (march_hi) R 9929 9931 9931 0 -1 4202496 27027 0 0 0 60585 5 0 0 25 0 1 0 658203579 144302080 26763 33554432000 4194304 4955518 140735034878816 18446744073709551615 4235599 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9931/statm: 35230 26763 106 186 0 35043 0
[pid=9929/tid=9968] ppid=9205 vsize=340936 CPUtime=0 cores=1,3,5,7
/proc/9929/task/9968/stat : 9968 (python) S 9205 9205 8588 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 658253216 349118464 5828 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604460364109056 0 0 -1 7 0 0 0
[pid=9933] ppid=9205 vsize=351192 CPUtime=2 cores=1,3,5,7
/proc/9933/stat : 9933 (python) S 9205 9205 8588 0 -1 4202560 1047 0 0 0 104 96 0 0 18 0 1 0 658204729 359620608 5808 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 16700526094754742535 0 0 17 5 0 0 0
/proc/9933/statm: 87798 5808 377 364 0 23695 0
[pid=9935] ppid=9933 vsize=181380 CPUtime=598.31 cores=1,3,5,7
/proc/9935/stat : 9935 (kcnfs-2006) R 9933 9935 9935 0 -1 4202496 685 0 0 0 59831 0 0 0 25 0 1 0 658204730 185733120 420 33554432000 134512640 135163436 4288985440 18446744073709551615 134564830 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9935/statm: 45345 420 76 159 0 45185 0
[pid=9205/tid=9212] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9212/stat : 9212 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 9 0 658145083 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 7 0 0 0
[pid=9205/tid=9215] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9215/stat : 9215 (python) S 9203 9205 8588 0 -1 4202560 20 935 0 0 0 0 0 0 19 0 9 0 658145084 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9218] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9218/stat : 9218 (python) S 9203 9205 8588 0 -1 4202560 36 935 0 0 0 0 0 0 18 0 9 0 658145085 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9221] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9221/stat : 9221 (python) S 9203 9205 8588 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 9 0 658145086 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9232] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9232/stat : 9232 (python) S 9203 9205 8588 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 9 0 658155019 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9236] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9236/stat : 9236 (python) S 9203 9205 8588 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 9 0 658155119 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9930] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9930/stat : 9930 (python) S 9203 9205 8588 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 658203579 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9934] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9934/stat : 9934 (python) S 9203 9205 8588 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 658204730 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4772.42
Current children cumulated vsize (KiB) 5914924

[startup+1198.7 s]
/proc/loadavg: 9.22 9.13 8.61 10/507 9971
/proc/meminfo: memFree=20438896/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=361436 CPUtime=0.67 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 16492 935 1 0 55 12 0 0 18 0 9 0 658145002 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/9205/statm: 90359 7343 1882 364 0 26256 0
[pid=9211] ppid=9205 vsize=288792 CPUtime=0.35 cores=1,3,5,7
/proc/9211/stat : 9211 (python) S 9205 9205 8588 0 -1 4202560 1173 0 0 0 17 18 0 0 19 0 2 0 658145083 295723008 5624 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9211/statm: 72198 5624 416 364 0 8116 0
[pid=9213] ppid=9211 vsize=1017800 CPUtime=98.98 cores=1,3,5,7
/proc/9213/stat : 9213 (gnovelty+2) T 9211 9213 9213 0 -1 4202496 443 0 0 0 9898 0 0 0 25 0 1 0 658145083 1042227200 180 33554432000 134512640 135663704 4288278688 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9213/statm: 254450 180 113 282 0 254166 0
[pid=9211/tid=9230] ppid=9205 vsize=288792 CPUtime=0 cores=1,3,5,7
/proc/9211/task/9230/stat : 9230 (python) S 9205 9205 8588 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658155018 295723008 5624 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604468579619136 0 0 -1 7 0 0 0
[pid=9214] ppid=9205 vsize=289340 CPUtime=0.35 cores=1,3,5,7
/proc/9214/stat : 9214 (python) S 9205 9205 8588 0 -1 4202560 1095 0 0 0 19 16 0 0 19 0 2 0 658145084 296284160 5746 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9214/statm: 72335 5746 400 364 0 8250 0
[pid=9216] ppid=9214 vsize=1017800 CPUtime=98.42 cores=1,3,5,7
/proc/9216/stat : 9216 (gnovelty+2) T 9214 9216 9216 0 -1 4202496 446 0 0 0 9842 0 0 0 25 0 1 0 658145084 1042227200 180 33554432000 134512640 135663704 4287275680 18446744073709551615 134545693 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/9216/statm: 254450 180 113 282 0 254166 0
[pid=9214/tid=9234] ppid=9205 vsize=289340 CPUtime=0 cores=1,3,5,7
/proc/9214/task/9234/stat : 9234 (python) S 9205 9205 8588 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 658155118 296284160 5746 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604461469470784 0 0 -1 7 0 0 0
[pid=9217] ppid=9205 vsize=299596 CPUtime=2.87 cores=1,3,5,7
/proc/9217/stat : 9217 (python) S 9205 9205 8588 0 -1 4202560 1162 0 0 0 151 136 0 0 15 0 2 0 658145085 306786304 5750 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9217/statm: 74899 5750 400 364 0 10811 0
[pid=9219] ppid=9217 vsize=140920 CPUtime=798.85 cores=1,3,5,7
/proc/9219/stat : 9219 (march_hi) R 9217 9219 9219 0 -1 4202496 34786 0 0 0 79871 14 0 0 25 0 1 0 658145085 144302080 34522 33554432000 4194304 4955518 140734270727392 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/9219/statm: 35230 34522 106 186 0 35043 0
[pid=9217/tid=9341] ppid=9205 vsize=299596 CPUtime=0 cores=1,3,5,7
/proc/9217/task/9341/stat : 9341 (python) S 9205 9205 8588 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 658184219 306786304 5750 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604459411162240 0 0 -1 1 0 0 0
[pid=9220] ppid=9205 vsize=309852 CPUtime=3.43 cores=1,3,5,7
/proc/9220/stat : 9220 (python) S 9205 9205 8588 0 -1 4202560 1165 0 0 0 181 162 0 0 15 0 2 0 658145086 317288448 5754 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9220/statm: 77463 5754 400 364 0 13372 0
[pid=9222] ppid=9220 vsize=213748 CPUtime=971.32 cores=1,3,5,7
/proc/9222/stat : 9222 (march_hi) T 9220 9222 9222 0 -1 4202496 41583 0 0 0 97122 10 0 0 25 0 1 0 658145086 218877952 41320 33554432000 4194304 4955518 140734430443904 18446744073709551615 4297477 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/9222/statm: 53437 41320 106 186 0 53250 0
[pid=9220/tid=9942] ppid=9205 vsize=309852 CPUtime=0 cores=1,3,5,7
/proc/9220/task/9942/stat : 9942 (python) S 9205 9205 8588 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 658213866 317288448 5754 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604455880734592 0 0 -1 1 0 0 0
[pid=9231] ppid=9205 vsize=320244 CPUtime=3.85 cores=1,3,5,7
/proc/9231/stat : 9231 (python) S 9205 9205 8588 0 -1 4202560 1165 0 0 0 210 175 0 0 15 0 2 0 658155019 327929856 5775 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9231/statm: 80061 5775 400 364 0 15967 0
[pid=9233] ppid=9231 vsize=213748 CPUtime=1095.76 cores=1,3,5,7
/proc/9233/stat : 9233 (march_hi) R 9231 9233 9233 0 -1 4202496 46299 0 0 0 109565 11 0 0 25 0 1 0 658155020 218877952 46036 33554432000 4194304 4955518 140736535004544 18446744073709551615 4237786 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/9233/statm: 53437 46036 106 186 0 53250 0
[pid=9231/tid=9947] ppid=9205 vsize=320244 CPUtime=0 cores=1,3,5,7
/proc/9231/task/9947/stat : 9947 (python) S 9205 9205 8588 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 658223116 327929856 5775 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604469284524992 0 0 -1 5 0 0 0
[pid=9235] ppid=9205 vsize=330500 CPUtime=1.78 cores=1,3,5,7
/proc/9235/stat : 9235 (python) S 9205 9205 8588 0 -1 4202560 1162 0 0 0 95 83 0 0 15 0 2 0 658155119 338432000 5781 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9235/statm: 82625 5781 400 364 0 18528 0
[pid=9237] ppid=9235 vsize=96720 CPUtime=488.2 cores=1,3,5,7
/proc/9237/stat : 9237 (march_hi) T 9235 9237 9237 0 -1 4202496 21886 0 0 0 48814 6 0 0 25 0 1 0 658155119 99041280 21625 33554432000 4194304 4955518 140736921910336 18446744073709551615 4238064 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/9237/statm: 24180 21625 106 186 0 23993 0
[pid=9235/tid=9389] ppid=9205 vsize=330500 CPUtime=0 cores=1,3,5,7
/proc/9235/task/9389/stat : 9389 (python) S 9205 9205 8588 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658194957 338432000 5781 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604465796436224 0 0 -1 7 0 0 0
[pid=9929] ppid=9205 vsize=340936 CPUtime=2.17 cores=1,3,5,7
/proc/9929/stat : 9929 (python) S 9205 9205 8588 0 -1 4202560 1173 0 0 0 114 103 0 0 15 0 2 0 658203579 349118464 5828 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9929/statm: 85234 5828 400 364 0 21134 0
[pid=9931] ppid=9929 vsize=140920 CPUtime=606.3 cores=1,3,5,7
/proc/9931/stat : 9931 (march_hi) R 9929 9931 9931 0 -1 4202496 27044 0 0 0 60625 5 0 0 25 0 1 0 658203579 144302080 26780 33554432000 4194304 4955518 140735034878816 18446744073709551615 4208802 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9931/statm: 35230 26780 106 186 0 35043 0
[pid=9929/tid=9968] ppid=9205 vsize=340936 CPUtime=0 cores=1,3,5,7
/proc/9929/task/9968/stat : 9968 (python) S 9205 9205 8588 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 658253216 349118464 5828 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 514 18446604460364109056 0 0 -1 7 0 0 0
[pid=9933] ppid=9205 vsize=351192 CPUtime=2 cores=1,3,5,7
/proc/9933/stat : 9933 (python) S 9205 9205 8588 0 -1 4202560 1047 0 0 0 104 96 0 0 18 0 1 0 658204729 359620608 5808 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 16700526094754742535 0 0 17 5 0 0 0
/proc/9933/statm: 87798 5808 377 364 0 23695 0
[pid=9935] ppid=9933 vsize=181380 CPUtime=598.71 cores=1,3,5,7
/proc/9935/stat : 9935 (kcnfs-2006) R 9933 9935 9935 0 -1 4202496 685 0 0 0 59871 0 0 0 25 0 1 0 658204730 185733120 420 33554432000 134512640 135163436 4288985440 18446744073709551615 134712495 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9935/statm: 45345 420 76 159 0 45185 0
[pid=9205/tid=9212] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9212/stat : 9212 (python) S 9203 9205 8588 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 9 0 658145083 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 7 0 0 0
[pid=9205/tid=9215] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9215/stat : 9215 (python) S 9203 9205 8588 0 -1 4202560 20 935 0 0 0 0 0 0 19 0 9 0 658145084 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9218] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9218/stat : 9218 (python) S 9203 9205 8588 0 -1 4202560 36 935 0 0 0 0 0 0 18 0 9 0 658145085 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9221] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9221/stat : 9221 (python) S 9203 9205 8588 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 9 0 658145086 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9232] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9232/stat : 9232 (python) S 9203 9205 8588 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 9 0 658155019 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9236] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9236/stat : 9236 (python) S 9203 9205 8588 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 9 0 658155119 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9930] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9930/stat : 9930 (python) S 9203 9205 8588 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 658203579 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9934] ppid=9203 vsize=361436 CPUtime=0 cores=1,3,5,7
/proc/9205/task/9934/stat : 9934 (python) S 9203 9205 8588 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 658204730 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4774.01
Current children cumulated vsize (KiB) 5914924

[startup+1199.1 s]
/proc/loadavg: 9.22 9.13 8.61 10/507 9971
/proc/meminfo: memFree=20437656/32951124 swapFree=56449232/67111528
[pid=9205] ppid=9203 vsize=361436 CPUtime=3074.57 cores=1,3,5,7
/proc/9205/stat : 9205 (python) S 9203 9205 8588 0 -1 4202496 16502 134279 1 0 55 12 306827 563 17 0 9 0 658145002 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652689727 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/9205/statm: 90359 7343 1882 364 0 26256 0
[pid=9231] ppid=9205 vsize=320244 CPUtime=3.85 cores=1,3,5,7
/proc/9231/stat : 9231 (python) S 9205 9205 8588 0 -1 4202560 1168 0 0 0 210 175 0 0 15 0 2 0 658155019 327929856 5775 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652689727 0 0 16781824 2 18446744073709551615 0 0 17 5 0 0 0
/proc/9231/statm: 80061 5775 400 364 0 15967 0
[pid=9233] ppid=9231 vsize=0 CPUtime=1096.16 cores=1,3,5,7
/proc/9233/stat : 9233 (march_hi) R 9231 9233 9233 0 -1 4203524 46317 0 0 0 109605 11 0 0 25 0 1 0 658155020 0 0 33554432000 0 0 0 0 0 0 0 16781312 0 18446744073709551615 0 0 17 5 0 0 0
/proc/9233/statm: 0 0 0 0 0 0 0
[pid=9231/tid=9947] ppid=9205 vsize=320244 CPUtime=0 cores=1,3,5,7
/proc/9231/task/9947/stat : 9947 (python) S 9205 9205 8588 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 658223116 327929856 5775 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781824 2 18446604469284524992 0 0 -1 5 0 0 0
[pid=9933] ppid=9205 vsize=351192 CPUtime=2 cores=1,3,5,7
/proc/9933/stat : 9933 (python) S 9205 9205 8588 0 -1 4202560 1047 0 0 0 104 96 0 0 18 0 1 0 658204729 359620608 5808 33554432000 4194304 5683644 140734946911744 18446744073709551615 253640879686 0 0 16781312 514 16700526094754742535 0 0 17 5 0 0 0
/proc/9933/statm: 87798 5808 377 364 0 23695 0
[pid=9935] ppid=9933 vsize=181380 CPUtime=599.11 cores=1,3,5,7
/proc/9935/stat : 9935 (kcnfs-2006) R 9933 9935 9935 0 -1 4202496 685 0 0 0 59911 0 0 0 25 0 1 0 658204730 185733120 420 33554432000 134512640 135163436 4288985440 18446744073709551615 134606328 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/9935/statm: 45345 420 76 159 0 45185 0
[pid=9205/tid=9212] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9212/stat : 9212 (python) S 9203 9205 8588 0 -1 4202560 27 134279 0 0 0 0 306827 563 18 0 9 0 658145083 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 7 0 0 0
[pid=9205/tid=9215] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9215/stat : 9215 (python) S 9203 9205 8588 0 -1 4202560 20 134279 0 0 0 0 306827 563 19 0 9 0 658145084 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9218] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9218/stat : 9218 (python) S 9203 9205 8588 0 -1 4202560 36 134279 0 0 0 0 306827 563 18 0 9 0 658145085 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9221] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9221/stat : 9221 (python) S 9203 9205 8588 0 -1 4202560 31 134279 0 0 0 0 306827 563 18 0 9 0 658145086 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9232] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9232/stat : 9232 (python) S 9203 9205 8588 0 -1 4202560 34 134279 0 0 0 0 306827 563 18 0 9 0 658155019 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9236] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9236/stat : 9236 (python) S 9203 9205 8588 0 -1 4202560 30 134279 0 0 0 0 306827 563 18 0 9 0 658155119 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9930] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9930/stat : 9930 (python) S 9203 9205 8588 0 -1 4202560 29 134279 0 0 0 0 306827 563 18 0 9 0 658203579 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
[pid=9205/tid=9934] ppid=9203 vsize=361436 CPUtime=3073.9 cores=1,3,5,7
/proc/9205/task/9934/stat : 9934 (python) S 9203 9205 8588 0 -1 4202560 28 134279 0 0 0 0 306827 563 20 0 9 0 658204730 370110464 7343 33554432000 4194304 5683644 140734946911744 18446744073709551615 253652683185 0 0 16781312 2 18446604469340748800 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4775.69
Current children cumulated vsize (KiB) 1214252

Child status: 0
Real time (s): 1199.13
CPU time (s): 4775.79
CPU user time (s): 4767.17
CPU system time (s): 8.61769
CPU usage (%): 398.272
Max. virtual memory (cumulated for all children) (KiB): 5914928

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4767.17
system time used= 8.61769
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 204090
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10207
involuntary context switches= 28528

runsolver used 5.76312 second user time and 12.2831 second system time

The end

Launcher Data 

Begin job on node106 at 2011-05-15 15:17:26
IDJOB=3378435
IDBENCH=82844
IDSOLVER=1859
FILE ID=node106/3378435-1305465446
RUNJOBID= node106-1305463093-8607
PBS_JOBID= 13324980
Free space on /tmp= 73352 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S121099030-067.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378435-1305465446/watcher-3378435-1305465446 -o /tmp/evaluation-result-3378435-1305465446/solver-3378435-1305465446 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378435-1305465446.cnf -seed 1000761321 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b936bd7aa1624f527379bba2a9d6f8ed
RANDOM SEED=1000761321

node106.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21187512 kB
Buffers:        299656 kB
Cached:        6292852 kB
SwapCached:      27536 kB
Active:        3153036 kB
Inactive:      8337272 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21187512 kB
SwapTotal:    67111528 kB
SwapFree:     56449232 kB
Dirty:           31856 kB
Writeback:           0 kB
AnonPages:     4878656 kB
Mapped:          26952 kB
Slab:           131704 kB
PageTables:      72292 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 131912952 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73320 MiB
End job on node106 at 2011-05-15 15:37:25