Trace number 3378344

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 3575.16 1125.22

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v600-c2556-S1734827769-099.UNKNOWN.cnf
MD5SUM2eb2fa74ab0dce335225200db81f6563
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses2556
Sum of the clauses size7668
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32556
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.30/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.30/0.46	c INFO: solving HOME/instance-3378344-1305464856.cnf
0.30/0.46	c INFO: computing task features
0.30/0.47	c INFO: computed variable-clause graph statistics
0.30/0.47	c INFO: computed clause balance statistics
0.39/0.51	c INFO: computed variable balance statistics
0.39/0.55	c INFO: computed small-clause counts
0.39/0.55	c INFO: computed Horn-clause counts
0.39/0.56	c INFO: computed variable graph statistics
0.39/0.56	c INFO: collected features for HOME/instance-3378344-1305464856.cnf in 0.10 s
0.39/0.57	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.58)
0.39/0.58	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.30)
0.39/0.58	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.39/0.59	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.29)
306.22/100.85	c INFO: running march_hi@0 for 679 with 1099 remaining (b = 0.35)
446.34/142.93	c INFO: running march_hi@0 for 679 with 1057 remaining (b = 0.31)
1446.05/429.49	c INFO: running kcnfs-2006@0 for 582 with 770 remaining (b = 0.26)
1799.34/533.96	c INFO: running march_hi@0 for 485 with 666 remaining (b = 0.22)
2627.96/810.13	c INFO: running march_hi@0 for 291 with 390 remaining (b = 0.12)
3503.62/1101.41	c INFO: running march_hi@100 for 99 with 99 remaining (b = 0.05)
3574.77/1125.19	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378344-1305464856/watcher-3378344-1305464856 -o /tmp/evaluation-result-3378344-1305464856/solver-3378344-1305464856 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378344-1305464856.cnf -seed 1927868539 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.01 8.12 7.35 6/363 2092
/proc/meminfo: memFree=23491496/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=15524 CPUtime=0 cores=0,2,4,6
/proc/2092/stat : 2092 (python) R 2090 2092 1220 0 -1 4202496 824 0 0 0 0 0 0 0 25 0 1 0 638900911 15896576 710 33554432000 4194304 5683644 140736177872368 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/2092/statm: 3946 713 334 364 0 433 0

[startup+0.0268789 s]
/proc/loadavg: 8.01 8.12 7.35 6/363 2092
/proc/meminfo: memFree=23491496/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=18884 CPUtime=0.01 cores=0,2,4,6
/proc/2092/stat : 2092 (python) R 2090 2092 1220 0 -1 4202496 1234 0 0 0 1 0 0 0 25 0 1 0 638900911 19337216 1091 33554432000 4194304 5683644 140736177872368 18446744073709551615 254256654080 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/2092/statm: 4721 1092 421 364 0 690 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 18884

[startup+0.100876 s]
/proc/loadavg: 8.01 8.12 7.35 6/363 2092
/proc/meminfo: memFree=23491496/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=104232 CPUtime=0.06 cores=0,2,4,6
/proc/2092/stat : 2092 (python) R 2090 2092 1220 0 -1 4202496 3087 0 0 0 5 1 0 0 17 0 1 0 638900911 106733568 2768 33554432000 4194304 5683644 140736177872368 18446744073709551615 5114052 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/2092/statm: 26058 2768 847 364 0 1806 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104232

[startup+0.300838 s]
/proc/loadavg: 8.01 8.12 7.35 6/363 2092
/proc/meminfo: memFree=23491496/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=194556 CPUtime=0.22 cores=0,2,4,6
/proc/2092/stat : 2092 (python) R 2090 2092 1220 0 -1 4202496 6910 934 1 0 16 6 0 0 17 0 1 0 638900911 199225344 4963 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261277729 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/2092/statm: 48638 4964 1431 364 0 3757 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 194556

[startup+0.700723 s]
/proc/loadavg: 8.01 8.12 7.35 6/363 2092
/proc/meminfo: memFree=23491496/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=320232 CPUtime=0.48 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 12184 934 1 0 39 9 0 0 24 0 5 0 638900911 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 80058 7247 1873 364 0 15967 0
Current children cumulated CPU time (s) 0.48
Current children cumulated vsize (KiB) 320232

[startup+1.50058 s]
/proc/loadavg: 8.01 8.12 7.35 10/376 2109
/proc/meminfo: memFree=23452084/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=320232 CPUtime=0.48 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 12184 934 1 0 39 9 0 0 24 0 5 0 638900911 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 80058 7247 1873 364 0 15967 0
[pid=2098] ppid=2092 vsize=278616 CPUtime=0 cores=0,2,4,6
/proc/2098/stat : 2098 (python) S 2092 2092 1220 0 -1 4202560 730 0 0 0 0 0 0 0 18 0 1 0 638900968 285302784 5561 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636623 0 0 16781312 514 15982237967489957887 0 0 17 2 0 0 0
/proc/2098/statm: 69654 5561 354 364 0 5572 0
[pid=2100] ppid=2098 vsize=1017932 CPUtime=0.91 cores=0,2,4,6
/proc/2100/stat : 2100 (gnovelty+2) R 2098 2100 2100 0 -1 4202496 448 0 0 0 91 0 0 0 21 0 1 0 638900969 1042362368 185 33554432000 134512640 135663704 4290127840 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2100/statm: 254483 185 113 282 0 254199 0
[pid=2101] ppid=2092 vsize=289376 CPUtime=0 cores=0,2,4,6
/proc/2101/stat : 2101 (python) S 2092 2092 1220 0 -1 4202560 746 0 0 0 0 0 0 0 15 0 1 0 638900969 296321024 5702 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 11159128232547025153 0 0 17 0 0 0 0
/proc/2101/statm: 72344 5702 348 364 0 8259 0
[pid=2103] ppid=2101 vsize=2612 CPUtime=0.6 cores=0,2,4,6
/proc/2103/stat : 2103 (march_hi) R 2101 2103 2103 0 -1 4202496 716 0 0 0 60 0 0 0 25 0 1 0 638900969 2674688 455 33554432000 4194304 4955518 140734198829296 18446744073709551615 4308569 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2103/statm: 653 455 104 186 0 466 0
[pid=2104] ppid=2092 vsize=299128 CPUtime=0 cores=0,2,4,6
/proc/2104/stat : 2104 (python) S 2092 2092 1220 0 -1 4202560 739 0 0 0 0 0 0 0 22 0 1 0 638900970 306307072 5595 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 4428969455296446463 0 0 17 2 0 0 0
/proc/2104/statm: 74782 5595 348 364 0 10694 0
[pid=2108] ppid=2104 vsize=1017932 CPUtime=0.89 cores=0,2,4,6
/proc/2108/stat : 2108 (gnovelty+2) R 2104 2108 2108 0 -1 4202496 445 0 0 0 89 0 0 0 25 0 1 0 638900971 1042362368 184 33554432000 134512640 135663704 4293095024 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2108/statm: 254483 184 113 282 0 254199 0
[pid=2106] ppid=2092 vsize=309988 CPUtime=0 cores=0,2,4,6
/proc/2106/stat : 2106 (python) S 2092 2092 1220 0 -1 4202560 743 0 0 0 0 0 0 0 17 0 1 0 638900971 317427712 5718 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 10988017834986012930 0 0 17 2 0 0 0
/proc/2106/statm: 77497 5718 348 364 0 13406 0
[pid=2109] ppid=2106 vsize=2476 CPUtime=0.31 cores=0,2,4,6
/proc/2109/stat : 2109 (march_hi) R 2106 2109 2109 0 -1 4202496 665 0 0 0 31 0 0 0 25 0 1 0 638900972 2535424 405 33554432000 4194304 4955518 140735423275248 18446744073709551615 4295413 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2109/statm: 619 405 104 186 0 432 0
[pid=2092/tid=2099] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2099/stat : 2099 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 5 0 638900968 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2102] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2102/stat : 2102 (python) S 2090 2092 1220 0 -1 4202560 23 934 0 0 0 0 0 0 21 0 5 0 638900969 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 0 0 0 0
[pid=2092/tid=2105] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2105/stat : 2105 (python) S 2090 2092 1220 0 -1 4202560 37 934 0 0 0 0 0 0 23 0 5 0 638900970 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2107] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2107/stat : 2107 (python) S 2090 2092 1220 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 638900971 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3.19
Current children cumulated vsize (KiB) 3538292

[startup+3.1013 s]
/proc/loadavg: 8.01 8.12 7.35 10/376 2109
/proc/meminfo: memFree=23450348/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=320232 CPUtime=0.48 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 12184 934 1 0 39 9 0 0 24 0 5 0 638900911 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 80058 7247 1873 364 0 15967 0
[pid=2098] ppid=2092 vsize=278616 CPUtime=0 cores=0,2,4,6
/proc/2098/stat : 2098 (python) S 2092 2092 1220 0 -1 4202560 730 0 0 0 0 0 0 0 18 0 1 0 638900968 285302784 5561 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636623 0 0 16781312 514 15982237967489925378 0 0 17 2 0 0 0
/proc/2098/statm: 69654 5561 354 364 0 5572 0
[pid=2100] ppid=2098 vsize=1017932 CPUtime=2.49 cores=0,2,4,6
/proc/2100/stat : 2100 (gnovelty+2) R 2098 2100 2100 0 -1 4202496 448 0 0 0 249 0 0 0 25 0 1 0 638900969 1042362368 185 33554432000 134512640 135663704 4290127840 18446744073709551615 134544293 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2100/statm: 254483 185 113 282 0 254199 0
[pid=2101] ppid=2092 vsize=289376 CPUtime=0 cores=0,2,4,6
/proc/2101/stat : 2101 (python) S 2092 2092 1220 0 -1 4202560 747 0 0 0 0 0 0 0 15 0 1 0 638900969 296321024 5702 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 4503611726293368831 0 0 17 0 0 0 0
/proc/2101/statm: 72344 5702 348 364 0 8259 0
[pid=2103] ppid=2101 vsize=2780 CPUtime=1.4 cores=0,2,4,6
/proc/2103/stat : 2103 (march_hi) R 2101 2103 2103 0 -1 4202496 749 0 0 0 140 0 0 0 25 0 1 0 638900969 2846720 488 33554432000 4194304 4955518 140734198829296 18446744073709551615 4209624 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2103/statm: 695 488 104 186 0 508 0
[pid=2104] ppid=2092 vsize=299128 CPUtime=0 cores=0,2,4,6
/proc/2104/stat : 2104 (python) S 2092 2092 1220 0 -1 4202560 739 0 0 0 0 0 0 0 18 0 1 0 638900970 306307072 5595 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 4428969455296413954 0 0 17 2 0 0 0
/proc/2104/statm: 74782 5595 348 364 0 10694 0
[pid=2108] ppid=2104 vsize=1017932 CPUtime=2.49 cores=0,2,4,6
/proc/2108/stat : 2108 (gnovelty+2) R 2104 2108 2108 0 -1 4202496 445 0 0 0 249 0 0 0 25 0 1 0 638900971 1042362368 184 33554432000 134512640 135663704 4293095024 18446744073709551615 134525544 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2108/statm: 254483 184 113 282 0 254199 0
[pid=2106] ppid=2092 vsize=309988 CPUtime=0 cores=0,2,4,6
/proc/2106/stat : 2106 (python) S 2092 2092 1220 0 -1 4202560 743 0 0 0 0 0 0 0 15 0 1 0 638900971 317427712 5718 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 10988017834986012930 0 0 17 2 0 0 0
/proc/2106/statm: 77497 5718 348 364 0 13406 0
[pid=2109] ppid=2106 vsize=2680 CPUtime=1.11 cores=0,2,4,6
/proc/2109/stat : 2109 (march_hi) R 2106 2109 2109 0 -1 4202496 738 0 0 0 111 0 0 0 25 0 1 0 638900972 2744320 478 33554432000 4194304 4955518 140735423275248 18446744073709551615 4224691 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2109/statm: 670 478 104 186 0 483 0
[pid=2092/tid=2099] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2099/stat : 2099 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 5 0 638900968 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2102] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2102/stat : 2102 (python) S 2090 2092 1220 0 -1 4202560 23 934 0 0 0 0 0 0 21 0 5 0 638900969 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 0 0 0 0
[pid=2092/tid=2105] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2105/stat : 2105 (python) S 2090 2092 1220 0 -1 4202560 37 934 0 0 0 0 0 0 23 0 5 0 638900970 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2107] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2107/stat : 2107 (python) S 2090 2092 1220 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 638900971 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 7.97
Current children cumulated vsize (KiB) 3538664

[startup+6.30073 s]
/proc/loadavg: 8.09 8.13 7.36 10/376 2109
/proc/meminfo: memFree=23448100/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=320232 CPUtime=0.48 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 12184 934 1 0 39 9 0 0 24 0 5 0 638900911 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 80058 7247 1873 364 0 15967 0
[pid=2098] ppid=2092 vsize=278616 CPUtime=0 cores=0,2,4,6
/proc/2098/stat : 2098 (python) S 2092 2092 1220 0 -1 4202560 892 0 0 0 0 0 0 0 18 0 1 0 638900968 285302784 5584 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636623 0 0 16781312 514 15982237967489925378 0 0 17 2 0 0 0
/proc/2098/statm: 69654 5584 377 364 0 5572 0
[pid=2100] ppid=2098 vsize=1017932 CPUtime=5.66 cores=0,2,4,6
/proc/2100/stat : 2100 (gnovelty+2) R 2098 2100 2100 0 -1 4202496 448 0 0 0 566 0 0 0 25 0 1 0 638900969 1042362368 185 33554432000 134512640 135663704 4290127840 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2100/statm: 254483 185 113 282 0 254199 0
[pid=2101] ppid=2092 vsize=289376 CPUtime=0.01 cores=0,2,4,6
/proc/2101/stat : 2101 (python) S 2092 2092 1220 0 -1 4202560 904 0 0 0 0 1 0 0 15 0 1 0 638900969 296321024 5725 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 11159128232547025153 0 0 17 0 0 0 0
/proc/2101/statm: 72344 5725 371 364 0 8259 0
[pid=2103] ppid=2101 vsize=2932 CPUtime=2.99 cores=0,2,4,6
/proc/2103/stat : 2103 (march_hi) R 2101 2103 2103 0 -1 4202496 808 0 0 0 299 0 0 0 25 0 1 0 638900969 3002368 547 33554432000 4194304 4955518 140734198829296 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2103/statm: 733 547 104 186 0 546 0
[pid=2104] ppid=2092 vsize=299128 CPUtime=0.01 cores=0,2,4,6
/proc/2104/stat : 2104 (python) S 2092 2092 1220 0 -1 4202560 899 0 0 0 0 1 0 0 18 0 1 0 638900970 306307072 5617 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 4428969455296413954 0 0 17 2 0 0 0
/proc/2104/statm: 74782 5617 370 364 0 10694 0
[pid=2108] ppid=2104 vsize=1017932 CPUtime=5.69 cores=0,2,4,6
/proc/2108/stat : 2108 (gnovelty+2) R 2104 2108 2108 0 -1 4202496 445 0 0 0 569 0 0 0 25 0 1 0 638900971 1042362368 184 33554432000 134512640 135663704 4293095024 18446744073709551615 134544380 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2108/statm: 254483 184 113 282 0 254199 0
[pid=2106] ppid=2092 vsize=309988 CPUtime=0.01 cores=0,2,4,6
/proc/2106/stat : 2106 (python) S 2092 2092 1220 0 -1 4202560 904 0 0 0 0 1 0 0 15 0 1 0 638900971 317427712 5741 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 10988017834986012930 0 0 17 2 0 0 0
/proc/2106/statm: 77497 5741 371 364 0 13406 0
[pid=2109] ppid=2106 vsize=2932 CPUtime=2.7 cores=0,2,4,6
/proc/2109/stat : 2109 (march_hi) R 2106 2109 2109 0 -1 4202496 795 0 0 0 270 0 0 0 25 0 1 0 638900972 3002368 535 33554432000 4194304 4955518 140735423275248 18446744073709551615 4238735 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2109/statm: 733 535 104 186 0 546 0
[pid=2092/tid=2099] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2099/stat : 2099 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 5 0 638900968 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2102] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2102/stat : 2102 (python) S 2090 2092 1220 0 -1 4202560 23 934 0 0 0 0 0 0 21 0 5 0 638900969 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 0 0 0 0
[pid=2092/tid=2105] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2105/stat : 2105 (python) S 2090 2092 1220 0 -1 4202560 37 934 0 0 0 0 0 0 23 0 5 0 638900970 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2107] ppid=2090 vsize=320232 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2107/stat : 2107 (python) S 2090 2092 1220 0 -1 4202560 18 934 0 0 0 0 0 0 25 0 5 0 638900971 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 17.55
Current children cumulated vsize (KiB) 3539068

[startup+12.7006 s]
/proc/loadavg: 8.16 8.15 7.37 10/376 2109
/proc/meminfo: memFree=23442552/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=320232 CPUtime=0.48 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 12184 934 1 0 39 9 0 0 24 0 5 0 638900911 327917568 7247 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 80058 7247 1873 364 0 15967 0
[pid=2098] ppid=2092 vsize=278616 CPUtime=0.02 cores=0,2,4,6
/proc/2098/stat : 2098 (python) S 2092 2092 1220 0 -1 4202560 895 0 0 0 1 1 0 0 18 0 1 0 638900968 285302784 5584 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636623 0 0 16781312 514 3797642793594093826 0 0 17 2 0 0 0
/proc/2098/statm: 69654 5584 377 364 0 5572 0
[pid=2100] ppid=2098 vsize=1017932 CPUtime=10.78 cores=0,2,4,6
/proc/2100/stat : 2100 (gnovelty+2) R 2098 2100 2100 0 -1 4202496 448 0 0 0 1078 0 0 0 25 0 1 0 638900969 1042362368 185 33554432000 134512640 135663704 4290127840 18446744073709551615 134546746 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2100/statm: 254483 185 113 282 0 254199 0
[pid=2101] ppid=2092 vsize=289376 CPUtime=0.03 cores=0,2,4,6

################
# More data... #
################

[pid=2141] ppid=2139 vsize=182580 CPUtime=462.58 cores=0,2,4,6
/proc/2141/stat : 2141 (kcnfs-2006) R 2139 2141 2141 0 -1 4202496 630 0 0 0 46258 0 0 0 25 0 1 0 638943861 186961920 370 33554432000 134512640 135163436 4293855296 18446744073709551615 134569635 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2141/statm: 45645 370 77 159 0 45485 0
[pid=2146] ppid=2092 vsize=351112 CPUtime=1.65 cores=0,2,4,6
/proc/2146/stat : 2146 (python) S 2092 2092 1220 0 -1 4202560 1055 0 0 0 84 81 0 0 15 0 1 0 638954308 359538688 5810 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 3797642793594093825 0 0 17 0 0 0 0
/proc/2146/statm: 87778 5810 377 364 0 23675 0
[pid=2148] ppid=2146 vsize=78976 CPUtime=449.56 cores=0,2,4,6
/proc/2148/stat : 2148 (march_hi) R 2146 2148 2148 0 -1 4202496 16230 0 0 0 44948 8 0 0 25 0 1 0 638954308 80871424 15964 33554432000 4194304 4955518 140734169176064 18446744073709551615 4307552 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2148/statm: 19744 15964 104 186 0 19557 0
[pid=2092/tid=2099] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2099/stat : 2099 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 9 0 638900968 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2102] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2102/stat : 2102 (python) S 2090 2092 1220 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 9 0 638900969 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 0 0 0 0
[pid=2092/tid=2105] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2105/stat : 2105 (python) S 2090 2092 1220 0 -1 4202560 37 934 0 0 0 0 0 0 23 0 9 0 638900970 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2107] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2107/stat : 2107 (python) S 2090 2092 1220 0 -1 4202560 23 934 0 0 0 0 0 0 18 0 9 0 638900971 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2123] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2123/stat : 2123 (python) S 2090 2092 1220 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 9 0 638910997 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2128] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2128/stat : 2128 (python) S 2090 2092 1220 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 638915204 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2140] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2140/stat : 2140 (python) S 2090 2092 1220 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 638943860 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2147] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2147/stat : 2147 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 9 0 638954308 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3571.12
Current children cumulated vsize (KiB) 5644424

[startup+1124.7 s]
/proc/loadavg: 9.43 9.26 8.64 10/417 2183
/proc/meminfo: memFree=22244300/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=361640 CPUtime=0.55 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 16062 934 1 0 44 11 0 0 18 0 9 0 638900911 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 90410 7402 1890 364 0 26307 0
[pid=2098] ppid=2092 vsize=288860 CPUtime=0.36 cores=0,2,4,6
/proc/2098/stat : 2098 (python) S 2092 2092 1220 0 -1 4202560 1174 0 0 0 18 18 0 0 19 0 2 0 638900968 295792640 5628 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2098/statm: 72215 5628 416 364 0 8133 0
[pid=2100] ppid=2098 vsize=1017932 CPUtime=102.05 cores=0,2,4,6
/proc/2100/stat : 2100 (gnovelty+2) T 2098 2100 2100 0 -1 4202496 448 0 0 0 10205 0 0 0 25 0 1 0 638900969 1042362368 185 33554432000 134512640 135663704 4290127840 18446744073709551615 134545583 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/2100/statm: 254483 185 113 282 0 254199 0
[pid=2098/tid=2126] ppid=2092 vsize=288860 CPUtime=0 cores=0,2,4,6
/proc/2098/task/2126/stat : 2126 (python) S 2092 2092 1220 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 638915203 295792640 5628 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 514 18446604443748946112 0 0 -1 2 0 0 0
[pid=2101] ppid=2092 vsize=289376 CPUtime=2.02 cores=0,2,4,6
/proc/2101/stat : 2101 (python) S 2092 2092 1220 0 -1 4202560 1151 0 0 0 103 99 0 0 15 0 2 0 638900969 296321024 5755 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/2101/statm: 72344 5755 400 364 0 8259 0
[pid=2103] ppid=2101 vsize=125616 CPUtime=680.71 cores=0,2,4,6
/proc/2103/stat : 2103 (march_hi) T 2101 2103 2103 0 -1 4202496 23654 0 0 0 68066 5 0 0 25 0 1 0 638900969 128630784 23393 33554432000 4194304 4955518 140734198829296 18446744073709551615 4238727 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2103/statm: 31404 23393 104 186 0 31217 0
[pid=2101/tid=2138] ppid=2092 vsize=289376 CPUtime=0 cores=0,2,4,6
/proc/2101/task/2138/stat : 2138 (python) S 2092 2092 1220 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 638943859 296321024 5755 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 514 18446604453961272064 0 0 -1 0 0 0 0
[pid=2104] ppid=2092 vsize=299128 CPUtime=0.25 cores=0,2,4,6
/proc/2104/stat : 2104 (python) S 2092 2092 1220 0 -1 4202560 1089 0 0 0 11 14 0 0 19 0 2 0 638900970 306307072 5648 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2104/statm: 74782 5648 400 364 0 10694 0
[pid=2108] ppid=2104 vsize=1017932 CPUtime=100.23 cores=0,2,4,6
/proc/2108/stat : 2108 (gnovelty+2) T 2104 2108 2108 0 -1 4202496 445 0 0 0 10023 0 0 0 25 0 1 0 638900971 1042362368 184 33554432000 134512640 135663704 4293095024 18446744073709551615 134525255 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2108/statm: 254483 184 113 282 0 254199 0
[pid=2104/tid=2121] ppid=2092 vsize=299128 CPUtime=0 cores=0,2,4,6
/proc/2104/task/2121/stat : 2121 (python) S 2092 2092 1220 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 638910996 306307072 5648 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 514 18446604453943893056 0 0 -1 2 0 0 0
[pid=2106] ppid=2092 vsize=309988 CPUtime=1.53 cores=0,2,4,6
/proc/2106/stat : 2106 (python) S 2092 2092 1220 0 -1 4202560 1158 0 0 0 73 80 0 0 15 0 2 0 638900971 317427712 5771 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2106/statm: 77497 5771 400 364 0 13406 0
[pid=2109] ppid=2106 vsize=78976 CPUtime=400.23 cores=0,2,4,6
/proc/2109/stat : 2109 (march_hi) R 2106 2109 2109 0 -1 4202496 14521 0 0 0 40020 3 0 0 25 0 1 0 638900972 80871424 14261 33554432000 4194304 4955518 140735423275248 18446744073709551615 4307087 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2109/statm: 19744 14261 104 186 0 19557 0
[pid=2106/tid=2145] ppid=2092 vsize=309988 CPUtime=0 cores=0,2,4,6
/proc/2106/task/2145/stat : 2145 (python) S 2092 2092 1220 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 638954307 317427712 5771 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 514 18446604446823407936 0 0 -1 2 0 0 0
[pid=2122] ppid=2092 vsize=320244 CPUtime=1.98 cores=0,2,4,6
/proc/2122/stat : 2122 (python) S 2092 2092 1220 0 -1 4202560 1146 0 0 0 105 93 0 0 16 0 2 0 638910997 327929856 5775 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/2122/statm: 80061 5775 400 364 0 15967 0
[pid=2124] ppid=2122 vsize=125616 CPUtime=683.81 cores=0,2,4,6
/proc/2124/stat : 2124 (march_hi) T 2122 2124 2124 0 -1 4202496 23646 0 0 0 68371 10 0 0 25 0 1 0 638910997 128630784 23387 33554432000 4194304 4955518 140735717353744 18446744073709551615 4224512 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/2124/statm: 31404 23387 104 186 0 31217 0
[pid=2122/tid=2163] ppid=2092 vsize=320244 CPUtime=0 cores=0,2,4,6
/proc/2122/task/2163/stat : 2163 (python) S 2092 2092 1220 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 638981924 327929856 5775 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 514 18446604445909460160 0 0 -1 0 0 0 0
[pid=2127] ppid=2092 vsize=329976 CPUtime=2.69 cores=0,2,4,6
/proc/2127/stat : 2127 (python) S 2092 2092 1220 0 -1 4202560 997 0 0 0 135 134 0 0 15 0 1 0 638915204 337895424 5650 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 3797642793594093825 0 0 17 4 0 0 0
/proc/2127/statm: 82494 5650 371 364 0 18397 0
[pid=2129] ppid=2127 vsize=125616 CPUtime=680.27 cores=0,2,4,6
/proc/2129/stat : 2129 (march_hi) R 2127 2129 2129 0 -1 4202496 23687 0 0 0 68021 6 0 0 25 0 1 0 638915205 128630784 23426 33554432000 4194304 4955518 140733605386992 18446744073709551615 4224768 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2129/statm: 31404 23426 104 186 0 31217 0
[pid=2139] ppid=2092 vsize=340856 CPUtime=1.85 cores=0,2,4,6
/proc/2139/stat : 2139 (python) S 2092 2092 1220 0 -1 4202560 994 0 0 0 89 96 0 0 18 0 1 0 638943860 349036544 5778 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 3797642793594093825 0 0 17 4 0 0 0
/proc/2139/statm: 85214 5778 371 364 0 21114 0
[pid=2141] ppid=2139 vsize=182580 CPUtime=462.98 cores=0,2,4,6
/proc/2141/stat : 2141 (kcnfs-2006) R 2139 2141 2141 0 -1 4202496 630 0 0 0 46298 0 0 0 25 0 1 0 638943861 186961920 370 33554432000 134512640 135163436 4293855296 18446744073709551615 134564834 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2141/statm: 45645 370 77 159 0 45485 0
[pid=2146] ppid=2092 vsize=351112 CPUtime=1.65 cores=0,2,4,6
/proc/2146/stat : 2146 (python) S 2092 2092 1220 0 -1 4202560 1055 0 0 0 84 81 0 0 15 0 1 0 638954308 359538688 5810 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 3797642793594093825 0 0 17 0 0 0 0
/proc/2146/statm: 87778 5810 377 364 0 23675 0
[pid=2148] ppid=2146 vsize=78976 CPUtime=450.36 cores=0,2,4,6
/proc/2148/stat : 2148 (march_hi) R 2146 2148 2148 0 -1 4202496 16254 0 0 0 45028 8 0 0 25 0 1 0 638954308 80871424 15988 33554432000 4194304 4955518 140734169176064 18446744073709551615 4224608 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2148/statm: 19744 15988 104 186 0 19557 0
[pid=2092/tid=2099] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2099/stat : 2099 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 9 0 638900968 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2102] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2102/stat : 2102 (python) S 2090 2092 1220 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 9 0 638900969 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 0 0 0 0
[pid=2092/tid=2105] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2105/stat : 2105 (python) S 2090 2092 1220 0 -1 4202560 37 934 0 0 0 0 0 0 23 0 9 0 638900970 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2107] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2107/stat : 2107 (python) S 2090 2092 1220 0 -1 4202560 23 934 0 0 0 0 0 0 18 0 9 0 638900971 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2123] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2123/stat : 2123 (python) S 2090 2092 1220 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 9 0 638910997 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2128] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2128/stat : 2128 (python) S 2090 2092 1220 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 638915204 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2140] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2140/stat : 2140 (python) S 2090 2092 1220 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 638943860 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2147] ppid=2090 vsize=361640 CPUtime=0 cores=0,2,4,6
/proc/2092/task/2147/stat : 2147 (python) S 2090 2092 1220 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 9 0 638954308 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3573.52
Current children cumulated vsize (KiB) 5644424

[startup+1125.1 s]
/proc/loadavg: 9.43 9.26 8.64 10/417 2183
/proc/meminfo: memFree=22244300/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=361640 CPUtime=1572.03 cores=0,2,4,6
/proc/2092/stat : 2092 (python) S 2090 2092 1220 0 -1 4202496 16101 53783 1 0 44 11 156905 243 17 0 9 0 638900911 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273446719 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/2092/statm: 90410 7402 1890 364 0 26307 0
[pid=2106] ppid=2092 vsize=309988 CPUtime=1.53 cores=0,2,4,6
/proc/2106/stat : 2106 (python) S 2092 2092 1220 0 -1 4202560 1158 0 0 0 73 80 0 0 15 0 2 0 638900971 317427712 5771 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2106/statm: 77497 5771 400 364 0 13406 0
[pid=2109] ppid=2106 vsize=78976 CPUtime=400.46 cores=0,2,4,6
/proc/2109/stat : 2109 (march_hi) R 2106 2109 2109 0 -1 4202496 14529 0 0 0 40043 3 0 0 25 0 1 0 638900972 80871424 14269 33554432000 4194304 4955518 140735423275248 18446744073709551615 4227640 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2109/statm: 19744 14269 104 186 0 19557 0
[pid=2106/tid=2145] ppid=2092 vsize=309988 CPUtime=0 cores=0,2,4,6
/proc/2106/task/2145/stat : 2145 (python) S 2092 2092 1220 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 638954307 317427712 5771 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 514 18446604446823407936 0 0 -1 2 0 0 0
[pid=2127] ppid=2092 vsize=329976 CPUtime=2.71 cores=0,2,4,6
/proc/2127/stat : 2127 (python) S 2092 2092 1220 0 -1 4202560 1151 0 0 0 136 135 0 0 17 0 2 0 638915204 337895424 5680 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273446719 0 0 16781824 2 18446744073709551615 0 0 17 4 0 0 0
/proc/2127/statm: 82494 5680 400 364 0 18397 0
[pid=2129] ppid=2127 vsize=125616 CPUtime=680.62 cores=0,2,4,6
/proc/2129/stat : 2129 (march_hi) R 2127 2129 2129 0 -1 4202496 23699 0 0 0 68056 6 0 0 25 0 1 0 638915205 128630784 23438 33554432000 4194304 4955518 140733605386992 18446744073709551615 4224665 256 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2129/statm: 31404 23438 104 186 0 31217 0
[pid=2139] ppid=2092 vsize=340856 CPUtime=1.85 cores=0,2,4,6
/proc/2139/stat : 2139 (python) S 2092 2092 1220 0 -1 4202560 994 0 0 0 89 96 0 0 18 0 1 0 638943860 349036544 5778 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 3797642793594093825 0 0 17 4 0 0 0
/proc/2139/statm: 85214 5778 371 364 0 21114 0
[pid=2141] ppid=2139 vsize=182580 CPUtime=463.17 cores=0,2,4,6
/proc/2141/stat : 2141 (kcnfs-2006) R 2139 2141 2141 0 -1 4202496 630 0 0 0 46317 0 0 0 25 0 1 0 638943861 186961920 370 33554432000 134512640 135163436 4293855296 18446744073709551615 134561489 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2141/statm: 45645 370 77 159 0 45485 0
[pid=2146] ppid=2092 vsize=351112 CPUtime=1.65 cores=0,2,4,6
/proc/2146/stat : 2146 (python) S 2092 2092 1220 0 -1 4202560 1055 0 0 0 84 81 0 0 15 0 1 0 638954308 359538688 5810 33554432000 4194304 5683644 140736177872368 18446744073709551615 254261636678 0 0 16781312 514 3797642793594093825 0 0 17 0 0 0 0
/proc/2146/statm: 87778 5810 377 364 0 23675 0
[pid=2148] ppid=2146 vsize=78976 CPUtime=450.75 cores=0,2,4,6
/proc/2148/stat : 2148 (march_hi) R 2146 2148 2148 0 -1 4202496 16265 0 0 0 45067 8 0 0 25 0 1 0 638954308 80871424 15999 33554432000 4194304 4955518 140734169176064 18446744073709551615 4227095 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2148/statm: 19744 15999 104 186 0 19557 0
[pid=2092/tid=2099] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2099/stat : 2099 (python) S 2090 2092 1220 0 -1 4202560 26 53783 0 0 0 0 156905 243 18 0 9 0 638900968 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2102] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2102/stat : 2102 (python) S 2090 2092 1220 0 -1 4202560 29 53783 0 0 0 0 156905 243 18 0 9 0 638900969 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 0 0 0 0
[pid=2092/tid=2105] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2105/stat : 2105 (python) S 2090 2092 1220 0 -1 4202560 37 53783 0 0 0 0 156905 243 23 0 9 0 638900970 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2107] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2107/stat : 2107 (python) S 2090 2092 1220 0 -1 4202560 23 53783 0 0 0 0 156905 243 18 0 9 0 638900971 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2123] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2123/stat : 2123 (python) S 2090 2092 1220 0 -1 4202560 27 53783 0 0 0 0 156905 243 20 0 9 0 638910997 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2128] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2128/stat : 2128 (python) S 2090 2092 1220 0 -1 4202560 25 53783 0 0 0 0 156905 243 20 0 9 0 638915204 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2140] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2140/stat : 2140 (python) S 2090 2092 1220 0 -1 4202560 25 53783 0 0 0 0 156905 243 20 0 9 0 638943860 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
[pid=2092/tid=2147] ppid=2090 vsize=361640 CPUtime=1571.48 cores=0,2,4,6
/proc/2092/task/2147/stat : 2147 (python) S 2090 2092 1220 0 -1 4202560 26 53783 0 0 0 0 156905 243 20 0 9 0 638954308 370319360 7402 33554432000 4194304 5683644 140736177872368 18446744073709551615 254273440177 0 0 16781312 2 18446604453956682944 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3574.77
Current children cumulated vsize (KiB) 2159720

[startup+1125.2 s]
/proc/loadavg: 9.43 9.26 8.64 7/388 2184
/proc/meminfo: memFree=22680764/32951124 swapFree=67057896/67111528
[pid=2092] ppid=2090 vsize=310312 CPUtime=3575.14 cores=0,2,4,6
/proc/2092/stat : 2092 (python) R 2090 2092 1220 0 -1 4202496 19616 113414 1 0 46 11 356799 658 17 0 1 0 638900911 317759488 7366 33554432000 4194304 5683644 140736177872368 18446744073709551615 4501813 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2092/statm: 77578 7366 1874 364 0 13502 0
Current children cumulated CPU time (s) 3575.14
Current children cumulated vsize (KiB) 310312

Child status: 0
Real time (s): 1125.22
CPU time (s): 3575.16
CPU user time (s): 3568.47
CPU system time (s): 6.69498
CPU usage (%): 317.731
Max. virtual memory (cumulated for all children) (KiB): 5644428

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3568.47
system time used= 6.69498
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 133581
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7192
involuntary context switches= 24917

runsolver used 4.73828 second user time and 9.46456 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-15 15:07:36
IDJOB=3378344
IDBENCH=82753
IDSOLVER=1859
FILE ID=node140/3378344-1305464856
RUNJOBID= node140-1305463229-1238
PBS_JOBID= 13324987
Free space on /tmp= 73436 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v600-c2556-S1734827769-099.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378344-1305464856/watcher-3378344-1305464856 -o /tmp/evaluation-result-3378344-1305464856/solver-3378344-1305464856 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378344-1305464856.cnf -seed 1927868539 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 2eb2fa74ab0dce335225200db81f6563
RANDOM SEED=1927868539

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23491528 kB
Buffers:        320428 kB
Cached:        4423672 kB
SwapCached:       8088 kB
Active:        5951476 kB
Inactive:      3175740 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23491528 kB
SwapTotal:    67111528 kB
SwapFree:     67057896 kB
Dirty:           64844 kB
Writeback:           0 kB
AnonPages:     4380116 kB
Mapped:          34272 kB
Slab:           229716 kB
PageTables:      35492 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 84262672 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73404 MiB
End job on node140 at 2011-05-15 15:26:21