Trace number 3385736

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)UNSAT 2775.9 485.584

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v500-c2130-S247862791-097.UNKNOWN.cnf
MD5SUM6b69f72e1046222baee8c8d817a4adac
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark444.867
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.45	c INFO: solving HOME/instance-3385736-1305572665.cnf
0.31/0.45	c INFO: computing task features
0.31/0.46	c INFO: computed variable-clause graph statistics
0.31/0.46	c INFO: computed clause balance statistics
0.39/0.50	c INFO: computed variable balance statistics
0.39/0.53	c INFO: computed small-clause counts
0.39/0.53	c INFO: computed Horn-clause counts
0.39/0.53	c INFO: computed variable graph statistics
0.39/0.53	c INFO: collected features for HOME/instance-3385736-1305572665.cnf in 0.08 s
0.39/0.56	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.57)
0.39/0.59	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.35)
0.49/0.62	c INFO: running march_hi@0 for 388 with 4999 remaining (b = 0.30)
0.49/0.65	c INFO: running march_hi@0 for 97 with 4999 remaining (b = 0.09)
0.49/0.69	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.07)
0.59/0.72	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.26)
0.59/0.76	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.23)
0.59/0.79	c INFO: running march_hi@0 for 485 with 4999 remaining (b = 0.21)
842.60/148.15	c INFO: running march_hi@0 for 485 with 4852 remaining (b = 0.18)
1135.87/197.03	c INFO: running kcnfs-2006@0 for 582 with 4803 remaining (b = 0.18)
1153.91/200.01	c INFO: running march_hi@0 for 97 with 4800 remaining (b = 0.04)
2235.86/392.83	c INFO: running march_hi@96 for 97 with 4607 remaining (b = 0.04)
2583.87/450.98	c INFO: running march_hi@0 for 97 with 4549 remaining (b = 0.04)
2775.35/485.56	s UNSATISFIABLE

Verifier Data 

No possible verification on an UNSAT instance

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385736-1305572665/watcher-3385736-1305572665 -o /tmp/evaluation-result-3385736-1305572665/solver-3385736-1305572665 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385736-1305572665.cnf -seed 2093793560 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.61 9.02 9.05 5/324 20048
/proc/meminfo: memFree=25426956/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=16240 CPUtime=0 cores=0-7
/proc/20048/stat : 20048 (python) R 20046 20048 19445 0 -1 4202496 996 0 0 0 0 0 0 0 25 0 1 0 747497128 16629760 869 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020056773 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/20048/statm: 4060 869 368 364 0 547 0

[startup+0.02583 s]
/proc/loadavg: 8.61 9.02 9.05 5/324 20048
/proc/meminfo: memFree=25426956/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=16400 CPUtime=0.01 cores=0-7
/proc/20048/stat : 20048 (python) R 20046 20048 19445 0 -1 4202496 1135 0 0 0 1 0 0 0 25 0 1 0 747497128 16793600 1002 33554432000 4194304 5683644 140735968644192 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/20048/statm: 4100 1004 413 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.100815 s]
/proc/loadavg: 8.61 9.02 9.05 5/324 20048
/proc/meminfo: memFree=25426956/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=104228 CPUtime=0.06 cores=0-7
/proc/20048/stat : 20048 (python) R 20046 20048 19445 0 -1 4202496 3070 0 0 0 5 1 0 0 25 0 1 0 747497128 106729472 2751 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020393984 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/20048/statm: 26057 2751 847 364 0 1805 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104228

[startup+0.300745 s]
/proc/loadavg: 8.61 9.02 9.05 5/324 20048
/proc/meminfo: memFree=25426956/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=198920 CPUtime=0.22 cores=0-7
/proc/20048/stat : 20048 (python) R 20046 20048 19445 0 -1 4202496 7019 931 1 0 18 4 0 0 25 0 1 0 747497128 203694080 5053 33554432000 4194304 5683644 140735968644192 18446744073709551615 48008046880464 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/20048/statm: 49730 5053 1466 364 0 3757 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 198920

[startup+0.700671 s]
/proc/loadavg: 8.61 9.02 9.05 5/324 20048
/proc/meminfo: memFree=25426956/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=330448 CPUtime=0.59 cores=0-7
/proc/20048/stat : 20048 (python) R 20046 20048 19445 0 -1 4202496 13570 931 1 0 52 7 0 0 25 0 6 0 747497128 338378752 7248 33554432000 4194304 5683644 140735968644192 18446744073709551615 4643548 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/20048/statm: 82612 7248 1881 364 0 18518 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 330448

[startup+1.50152 s]
/proc/loadavg: 8.61 9.02 9.05 13/349 20077
/proc/meminfo: memFree=25367228/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=361352 CPUtime=0.67 cores=0-7
/proc/20048/stat : 20048 (python) S 20046 20048 19445 0 -1 4202496 15744 931 1 0 60 7 0 0 25 0 9 0 747497128 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032222667 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/20048/statm: 90338 7302 1889 364 0 26235 0
[pid=20054] ppid=20048 vsize=278676 CPUtime=0 cores=0-7
/proc/20054/stat : 20054 (python) S 20048 20048 19445 0 -1 4202560 732 0 0 0 0 0 0 0 25 0 1 0 747497185 285364224 5575 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416015 0 0 16781312 514 16449749761864499199 0 0 17 5 0 0 0
/proc/20054/statm: 69669 5575 354 364 0 5587 0
[pid=20056] ppid=20054 vsize=1017800 CPUtime=0.61 cores=0-7
/proc/20056/stat : 20056 (gnovelty+2) R 20054 20056 20056 0 -1 4202496 439 0 0 0 61 0 0 0 25 0 1 0 747497186 1042227200 178 33554432000 134512640 135663704 4291797328 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/20056/statm: 254450 178 113 282 0 254166 0
[pid=20057] ppid=20048 vsize=289336 CPUtime=0 cores=0-7
/proc/20057/stat : 20057 (python) S 20048 20048 19445 0 -1 4202560 752 0 0 0 0 0 0 0 18 0 1 0 747497188 296280064 5691 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 5607391911611433219 0 0 17 6 0 0 0
/proc/20057/statm: 72334 5691 348 364 0 8249 0
[pid=20059] ppid=20057 vsize=2172 CPUtime=0.89 cores=0-7
/proc/20059/stat : 20059 (march_hi) R 20057 20059 20059 0 -1 4202496 619 0 0 0 89 0 0 0 25 0 1 0 747497188 2224128 355 33554432000 4194304 4955518 140733935430288 18446744073709551615 4224656 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/20059/statm: 543 355 104 186 0 356 0
[pid=20060] ppid=20048 vsize=299240 CPUtime=0 cores=0-7
/proc/20060/stat : 20060 (python) S 20048 20048 19445 0 -1 4202560 747 0 0 0 0 0 0 0 22 0 1 0 747497191 306421760 5611 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 6540616445934403844 0 0 17 5 0 0 0
/proc/20060/statm: 74810 5611 348 364 0 10722 0
[pid=20062] ppid=20060 vsize=2172 CPUtime=0.55 cores=0-7
/proc/20062/stat : 20062 (march_hi) R 20060 20062 20062 0 -1 4202496 616 0 0 0 55 0 0 0 25 0 1 0 747497192 2224128 351 33554432000 4194304 4955518 140734957602736 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/20062/statm: 543 351 104 186 0 356 0
[pid=20063] ppid=20048 vsize=309948 CPUtime=0 cores=0-7
/proc/20063/stat : 20063 (python) S 20048 20048 19445 0 -1 4202560 753 0 0 0 0 0 0 0 19 0 1 0 747497194 317386752 5707 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 3631150349164708098 0 0 17 2 0 0 0
/proc/20063/statm: 77487 5707 348 364 0 13396 0
[pid=20065] ppid=20063 vsize=2172 CPUtime=0.31 cores=0-7
/proc/20065/stat : 20065 (march_hi) R 20063 20065 20065 0 -1 4202496 603 0 0 0 31 0 0 0 25 0 1 0 747497197 2224128 338 33554432000 4194304 4955518 140733714518400 18446744073709551615 4209688 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/20065/statm: 543 338 104 186 0 356 0
[pid=20066] ppid=20048 vsize=320204 CPUtime=0 cores=0-7
/proc/20066/stat : 20066 (python) S 20048 20048 19445 0 -1 4202560 753 0 0 0 0 0 0 0 25 0 1 0 747497198 327888896 5711 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 17370001411894411263 0 0 17 2 0 0 0
/proc/20066/statm: 80051 5711 348 364 0 15957 0
[pid=20068] ppid=20066 vsize=1017800 CPUtime=0.72 cores=0-7
/proc/20068/stat : 20068 (gnovelty+2) R 20066 20068 20068 0 -1 4202496 441 0 0 0 72 0 0 0 25 0 1 0 747497200 1042227200 177 33554432000 134512640 135663704 4292995664 18446744073709551615 134525493 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/20068/statm: 254450 177 113 282 0 254166 0
[pid=20069] ppid=20048 vsize=330460 CPUtime=0 cores=0-7
/proc/20069/stat : 20069 (python) S 20048 20048 19445 0 -1 4202560 769 0 0 0 0 0 0 0 18 0 1 0 747497201 338391040 5715 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 5607391911611433219 0 0 17 2 0 0 0
/proc/20069/statm: 82615 5715 348 364 0 18518 0
[pid=20071] ppid=20069 vsize=2172 CPUtime=0.29 cores=0-7
/proc/20071/stat : 20071 (march_hi) R 20069 20071 20071 0 -1 4202496 605 0 0 0 29 0 0 0 25 0 1 0 747497204 2224128 339 33554432000 4194304 4955518 140735692014512 18446744073709551615 4224608 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/20071/statm: 543 339 104 186 0 356 0
[pid=20072] ppid=20048 vsize=340716 CPUtime=0 cores=0-7
/proc/20072/stat : 20072 (python) S 20048 20048 19445 0 -1 4202560 774 0 0 0 0 0 0 0 20 0 1 0 747497204 348893184 5736 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 206730180549378307 0 0 17 2 0 0 0
/proc/20072/statm: 85179 5736 348 364 0 21079 0
[pid=20074] ppid=20072 vsize=2172 CPUtime=0.32 cores=0-7
/proc/20074/stat : 20074 (march_hi) R 20072 20074 20074 0 -1 4202496 605 0 0 0 32 0 0 0 25 0 1 0 747497207 2224128 338 33554432000 4194304 4955518 140736667254528 18446744073709551615 4224642 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/20074/statm: 543 338 104 186 0 356 0
[pid=20075] ppid=20048 vsize=351108 CPUtime=0 cores=0-7
/proc/20075/stat : 20075 (python) S 20048 20048 19445 0 -1 4202560 813 0 0 0 0 0 0 0 21 0 1 0 747497208 359534592 5763 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 5990077779767427330 0 0 17 7 0 0 0
/proc/20075/statm: 87777 5763 354 364 0 23674 0
[pid=20077] ppid=20075 vsize=2172 CPUtime=0.6 cores=0-7
/proc/20077/stat : 20077 (march_hi) R 20075 20077 20077 0 -1 4202496 623 0 0 0 60 0 0 0 25 0 1 0 747497208 2224128 350 33554432000 4194304 4955518 140733288599696 18446744073709551615 4238968 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/20077/statm: 543 350 104 186 0 356 0
[pid=20048/tid=20055] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20055/stat : 20055 (python) S 20046 20048 19445 0 -1 4202560 27 931 0 0 0 0 0 0 25 0 9 0 747497185 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20058] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20058/stat : 20058 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 9 0 747497188 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 6 0 0 0
[pid=20048/tid=20061] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20061/stat : 20061 (python) S 20046 20048 19445 0 -1 4202560 20 931 0 0 0 0 0 0 25 0 9 0 747497191 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
[pid=20048/tid=20064] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20064/stat : 20064 (python) S 20046 20048 19445 0 -1 4202560 24 931 0 0 0 0 0 0 25 0 9 0 747497194 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20067] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20067/stat : 20067 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 9 0 747497198 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20070] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20070/stat : 20070 (python) S 20046 20048 19445 0 -1 4202560 26 931 0 0 0 0 0 0 25 0 9 0 747497201 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20073] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20073/stat : 20073 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 9 0 747497205 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20076] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20076/stat : 20076 (python) S 20046 20048 19445 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 9 0 747497208 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.96
Current children cumulated vsize (KiB) 4929672

[startup+3.10123 s]
/proc/loadavg: 8.61 9.02 9.05 13/349 20077
/proc/meminfo: memFree=25366236/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=361352 CPUtime=0.67 cores=0-7
/proc/20048/stat : 20048 (python) S 20046 20048 19445 0 -1 4202496 15744 931 1 0 60 7 0 0 25 0 9 0 747497128 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032222667 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/20048/statm: 90338 7302 1889 364 0 26235 0
[pid=20054] ppid=20048 vsize=278676 CPUtime=0 cores=0-7
/proc/20054/stat : 20054 (python) S 20048 20048 19445 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 747497185 285364224 5575 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416015 0 0 16781312 514 16449749761864466690 0 0 17 5 0 0 0
/proc/20054/statm: 69669 5575 354 364 0 5587 0
[pid=20056] ppid=20054 vsize=1017800 CPUtime=1.41 cores=0-7
/proc/20056/stat : 20056 (gnovelty+2) R 20054 20056 20056 0 -1 4202496 439 0 0 0 141 0 0 0 25 0 1 0 747497186 1042227200 178 33554432000 134512640 135663704 4291797328 18446744073709551615 134525506 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/20056/statm: 254450 178 113 282 0 254166 0
[pid=20057] ppid=20048 vsize=289336 CPUtime=0 cores=0-7
/proc/20057/stat : 20057 (python) S 20048 20048 19445 0 -1 4202560 752 0 0 0 0 0 0 0 15 0 1 0 747497188 296280064 5691 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14960232288745390339 0 0 17 6 0 0 0
/proc/20057/statm: 72334 5691 348 364 0 8249 0
[pid=20059] ppid=20057 vsize=2304 CPUtime=2.49 cores=0-7
/proc/20059/stat : 20059 (march_hi) R 20057 20059 20059 0 -1 4202496 635 0 0 0 249 0 0 0 25 0 1 0 747497188 2359296 371 33554432000 4194304 4955518 140733935430288 18446744073709551615 4314843 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/20059/statm: 576 371 104 186 0 389 0
[pid=20060] ppid=20048 vsize=299240 CPUtime=0 cores=0-7
/proc/20060/stat : 20060 (python) S 20048 20048 19445 0 -1 4202560 752 0 0 0 0 0 0 0 15 0 1 0 747497191 306421760 5611 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 6540616445934403844 0 0 17 5 0 0 0
/proc/20060/statm: 74810 5611 348 364 0 10722 0
[pid=20062] ppid=20060 vsize=2172 CPUtime=1.35 cores=0-7
/proc/20062/stat : 20062 (march_hi) R 20060 20062 20062 0 -1 4202496 626 0 0 0 135 0 0 0 25 0 1 0 747497192 2224128 361 33554432000 4194304 4955518 140734957602736 18446744073709551615 4228014 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/20062/statm: 543 361 104 186 0 356 0
[pid=20063] ppid=20048 vsize=309948 CPUtime=0 cores=0-7
/proc/20063/stat : 20063 (python) S 20048 20048 19445 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 747497194 317386752 5707 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 3631150349164708098 0 0 17 2 0 0 0
/proc/20063/statm: 77487 5707 348 364 0 13396 0
[pid=20065] ppid=20063 vsize=2172 CPUtime=1.11 cores=0-7
/proc/20065/stat : 20065 (march_hi) R 20063 20065 20065 0 -1 4202496 622 0 0 0 111 0 0 0 25 0 1 0 747497197 2224128 357 33554432000 4194304 4955518 140733714518400 18446744073709551615 4227032 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/20065/statm: 543 357 104 186 0 356 0
[pid=20066] ppid=20048 vsize=320204 CPUtime=0 cores=0-7
/proc/20066/stat : 20066 (python) S 20048 20048 19445 0 -1 4202560 753 0 0 0 0 0 0 0 18 0 1 0 747497198 327888896 5711 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 17370001411894378753 0 0 17 2 0 0 0
/proc/20066/statm: 80051 5711 348 364 0 15957 0
[pid=20068] ppid=20066 vsize=1017800 CPUtime=2.31 cores=0-7
/proc/20068/stat : 20068 (gnovelty+2) R 20066 20068 20068 0 -1 4202496 441 0 0 0 231 0 0 0 25 0 1 0 747497200 1042227200 177 33554432000 134512640 135663704 4292995664 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/20068/statm: 254450 177 113 282 0 254166 0
[pid=20069] ppid=20048 vsize=330460 CPUtime=0 cores=0-7
/proc/20069/stat : 20069 (python) S 20048 20048 19445 0 -1 4202560 769 0 0 0 0 0 0 0 15 0 1 0 747497201 338391040 5715 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14649624651945181443 0 0 17 2 0 0 0
/proc/20069/statm: 82615 5715 348 364 0 18518 0
[pid=20071] ppid=20069 vsize=2172 CPUtime=1.09 cores=0-7
/proc/20071/stat : 20071 (march_hi) R 20069 20071 20071 0 -1 4202496 624 0 0 0 109 0 0 0 25 0 1 0 747497204 2224128 358 33554432000 4194304 4955518 140735692014512 18446744073709551615 4224646 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/20071/statm: 543 358 104 186 0 356 0
[pid=20072] ppid=20048 vsize=340716 CPUtime=0 cores=0-7
/proc/20072/stat : 20072 (python) S 20048 20048 19445 0 -1 4202560 775 0 0 0 0 0 0 0 15 0 1 0 747497204 348893184 5736 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 13360478054703399171 0 0 17 2 0 0 0
/proc/20072/statm: 85179 5736 348 364 0 21079 0
[pid=20074] ppid=20072 vsize=2172 CPUtime=1.12 cores=0-7
/proc/20074/stat : 20074 (march_hi) R 20072 20074 20074 0 -1 4202496 624 0 0 0 112 0 0 0 25 0 1 0 747497207 2224128 357 33554432000 4194304 4955518 140736667254528 18446744073709551615 4209776 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/20074/statm: 543 357 104 186 0 356 0
[pid=20075] ppid=20048 vsize=351108 CPUtime=0 cores=0-7
/proc/20075/stat : 20075 (python) S 20048 20048 19445 0 -1 4202560 813 0 0 0 0 0 0 0 15 0 1 0 747497208 359534592 5763 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 5990077779767427330 0 0 17 7 0 0 0
/proc/20075/statm: 87777 5763 354 364 0 23674 0
[pid=20077] ppid=20075 vsize=2304 CPUtime=2.18 cores=0-7
/proc/20077/stat : 20077 (march_hi) R 20075 20077 20077 0 -1 4202496 641 0 0 0 218 0 0 0 25 0 1 0 747497208 2359296 368 33554432000 4194304 4955518 140733288599696 18446744073709551615 4224615 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/20077/statm: 576 368 104 186 0 389 0
[pid=20048/tid=20055] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20055/stat : 20055 (python) S 20046 20048 19445 0 -1 4202560 27 931 0 0 0 0 0 0 25 0 9 0 747497185 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20058] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20058/stat : 20058 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 9 0 747497188 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 6 0 0 0
[pid=20048/tid=20061] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20061/stat : 20061 (python) S 20046 20048 19445 0 -1 4202560 20 931 0 0 0 0 0 0 25 0 9 0 747497191 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
[pid=20048/tid=20064] ppid=20046 vsize=361352 CPUtime=0 cores=0-7
/proc/20048/task/20064/stat : 20064 (python) S 20046 20048 19445 0 -1 4202560 24 931 0 0 0 0 0 0 25 0 9 0 747497194 370024448 7302 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20067] ppid=20046 vsize=361352 CPUtime=0 cores=0-7

################
# More data... #
################

/proc/20060/stat : 20060 (python) S 20048 20048 19445 0 -1 4202560 991 0 0 0 69 59 0 0 15 0 1 0 747497191 306421760 5634 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 6540616445934403844 0 0 17 5 0 0 0
/proc/20060/statm: 74810 5634 371 364 0 10722 0
[pid=20062] ppid=20060 vsize=10868 CPUtime=380.24 cores=0-7
/proc/20062/stat : 20062 (march_hi) R 20060 20062 20062 0 -1 4202496 2158 0 0 0 38024 0 0 0 25 0 1 0 747497192 11128832 1893 33554432000 4194304 4955518 140734957602736 18446744073709551615 4227892 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/20062/statm: 2717 1893 104 186 0 2530 0
[pid=20063] ppid=20048 vsize=309948 CPUtime=1.24 cores=0-7
/proc/20063/stat : 20063 (python) S 20048 20048 19445 0 -1 4202560 1175 0 0 0 63 61 0 0 15 0 2 0 747497194 317386752 5760 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/20063/statm: 77487 5760 400 364 0 13396 0
[pid=20065] ppid=20063 vsize=6088 CPUtime=242.05 cores=0-7
/proc/20065/stat : 20065 (march_hi) R 20063 20065 20065 0 -1 4202496 1311 0 0 0 24205 0 0 0 25 0 1 0 747497197 6234112 1046 33554432000 4194304 4955518 140733714518400 18446744073709551615 4227747 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/20065/statm: 1522 1046 104 186 0 1335 0
[pid=20063/tid=20091] ppid=20048 vsize=309948 CPUtime=0 cores=0-7
/proc/20063/task/20091/stat : 20091 (python) S 20048 20048 19445 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 747517126 317386752 5760 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 514 18446604453045365504 0 0 -1 2 0 0 0
[pid=20066] ppid=20048 vsize=320204 CPUtime=0.34 cores=0-7
/proc/20066/stat : 20066 (python) S 20048 20048 19445 0 -1 4202560 1125 0 0 0 19 15 0 0 19 0 2 0 747497198 327888896 5764 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032222667 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/20066/statm: 80051 5764 400 364 0 15957 0
[pid=20068] ppid=20066 vsize=1017800 CPUtime=97.2 cores=0-7
/proc/20068/stat : 20068 (gnovelty+2) T 20066 20068 20068 0 -1 4202496 441 0 0 0 9720 0 0 0 25 0 1 0 747497200 1042227200 177 33554432000 134512640 135663704 4292995664 18446744073709551615 134525678 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/20068/statm: 254450 177 113 282 0 254166 0
[pid=20066/tid=20082] ppid=20048 vsize=320204 CPUtime=0 cores=0-7
/proc/20066/task/20082/stat : 20082 (python) S 20048 20048 19445 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 747511940 327888896 5764 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 514 18446604444072059968 0 0 -1 0 0 0 0
[pid=20069] ppid=20048 vsize=330460 CPUtime=1.25 cores=0-7
/proc/20069/stat : 20069 (python) S 20048 20048 19445 0 -1 4202560 1023 0 0 0 68 57 0 0 15 0 1 0 747497201 338391040 5738 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14454918730557128963 0 0 17 2 0 0 0
/proc/20069/statm: 82615 5738 371 364 0 18518 0
[pid=20071] ppid=20069 vsize=8044 CPUtime=308.35 cores=0-7
/proc/20071/stat : 20071 (march_hi) R 20069 20071 20071 0 -1 4202496 1654 0 0 0 30835 0 0 0 25 0 1 0 747497204 8237056 1388 33554432000 4194304 4955518 140735692014512 18446744073709551615 4238626 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/20071/statm: 2011 1388 104 186 0 1824 0
[pid=20072] ppid=20048 vsize=340716 CPUtime=1.29 cores=0-7
/proc/20072/stat : 20072 (python) S 20048 20048 19445 0 -1 4202560 1035 0 0 0 67 62 0 0 15 0 1 0 747497204 348893184 5759 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 206730180549378307 0 0 17 0 0 0 0
/proc/20072/statm: 85179 5759 371 364 0 21079 0
[pid=20074] ppid=20072 vsize=15212 CPUtime=448.93 cores=0-7
/proc/20074/stat : 20074 (march_hi) R 20072 20074 20074 0 -1 4202496 2914 0 0 0 44888 5 0 0 25 0 1 0 747497207 15577088 2647 33554432000 4194304 4955518 140736667254528 18446744073709551615 4282888 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/20074/statm: 3803 2647 104 186 0 3616 0
[pid=20075] ppid=20048 vsize=351108 CPUtime=1.31 cores=0-7
/proc/20075/stat : 20075 (python) S 20048 20048 19445 0 -1 4202560 1062 0 0 0 64 67 0 0 15 0 1 0 747497208 359534592 5786 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 5990077779767427330 0 0 17 7 0 0 0
/proc/20075/statm: 87777 5786 377 364 0 23674 0
[pid=20077] ppid=20075 vsize=15212 CPUtime=451.17 cores=0-7
/proc/20077/stat : 20077 (march_hi) R 20075 20077 20077 0 -1 4202496 2950 0 0 0 45117 0 0 0 25 0 1 0 747497208 15577088 2677 33554432000 4194304 4955518 140733288599696 18446744073709551615 4227752 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/20077/statm: 3803 2677 104 186 0 3616 0
[pid=20083] ppid=20048 vsize=361364 CPUtime=0.88 cores=0-7
/proc/20083/stat : 20083 (python) S 20048 20048 19445 0 -1 4202560 1037 0 0 0 47 41 0 0 15 0 1 0 747511944 370036736 5784 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14454918730557128961 0 0 17 4 0 0 0
/proc/20083/statm: 90341 5784 371 364 0 26235 0
[pid=20085] ppid=20083 vsize=4172 CPUtime=168.39 cores=0-7
/proc/20085/stat : 20085 (march_hi) R 20083 20085 20085 0 -1 4202496 1057 0 0 0 16838 1 0 0 25 0 1 0 747511944 4272128 787 33554432000 4194304 4955518 140734487616032 18446744073709551615 4235344 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/20085/statm: 1043 787 104 186 0 856 0
[pid=20088] ppid=20048 vsize=437156 CPUtime=0.67 cores=0-7
/proc/20088/stat : 20088 (python) S 20048 20048 19445 0 -1 4202560 1030 0 0 0 37 30 0 0 18 0 1 0 747516832 447647744 5795 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14454918730557128960 0 0 17 1 0 0 0
/proc/20088/statm: 109289 5795 371 364 0 45180 0
[pid=20090] ppid=20088 vsize=182444 CPUtime=159.35 cores=0-7
/proc/20090/stat : 20090 (kcnfs-2006) R 20088 20090 20090 0 -1 4202496 597 0 0 0 15935 0 0 0 25 0 1 0 747516833 186822656 330 33554432000 134512640 135163436 4289090528 18446744073709551615 134564786 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/20090/statm: 45611 330 77 159 0 45451 0
[pid=20048/tid=20055] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20055/stat : 20055 (python) S 20046 20048 19445 0 -1 4202560 27 931 0 0 0 0 0 0 25 0 11 0 747497185 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20058] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20058/stat : 20058 (python) S 20046 20048 19445 0 -1 4202560 28 931 0 0 0 0 0 0 18 0 11 0 747497188 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 6 0 0 0
[pid=20048/tid=20061] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20061/stat : 20061 (python) S 20046 20048 19445 0 -1 4202560 20 931 0 0 0 0 0 0 25 0 11 0 747497191 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
[pid=20048/tid=20064] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20064/stat : 20064 (python) S 20046 20048 19445 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 11 0 747497194 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20067] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20067/stat : 20067 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 11 0 747497198 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20070] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20070/stat : 20070 (python) S 20046 20048 19445 0 -1 4202560 26 931 0 0 0 0 0 0 25 0 11 0 747497201 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20073] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20073/stat : 20073 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 11 0 747497205 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20076] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20076/stat : 20076 (python) S 20046 20048 19445 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 11 0 747497208 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20084] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20084/stat : 20084 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 11 0 747511944 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
[pid=20048/tid=20089] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20089/stat : 20089 (python) S 20046 20048 19445 0 -1 4202560 26 931 0 0 0 0 0 0 19 0 11 0 747516833 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2773.22
Current children cumulated vsize (KiB) 6064824

[startup+485.501 s]
/proc/loadavg: 11.12 10.69 9.87 12/358 20100
/proc/meminfo: memFree=25229660/32951124 swapFree=67083432/67111528
[pid=20048] ppid=20046 vsize=447400 CPUtime=0.85 cores=0-7
/proc/20048/stat : 20048 (python) S 20046 20048 19445 0 -1 4202496 18251 931 1 0 77 8 0 0 18 0 11 0 747497128 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032222667 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/20048/statm: 111850 7472 1894 364 0 47741 0
[pid=20054] ppid=20048 vsize=288920 CPUtime=0.46 cores=0-7
/proc/20054/stat : 20054 (python) S 20048 20048 19445 0 -1 4202560 1112 0 0 0 24 22 0 0 19 0 2 0 747497185 295854080 5636 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032222667 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/20054/statm: 72230 5636 410 364 0 8148 0
[pid=20056] ppid=20054 vsize=1017800 CPUtime=98.13 cores=0-7
/proc/20056/stat : 20056 (gnovelty+2) T 20054 20056 20056 0 -1 4202496 439 0 0 0 9813 0 0 0 25 0 1 0 747497186 1042227200 178 33554432000 134512640 135663704 4291797328 18446744073709551615 134525535 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/20056/statm: 254450 178 113 282 0 254166 0
[pid=20054/tid=20087] ppid=20048 vsize=288920 CPUtime=0 cores=0-7
/proc/20054/task/20087/stat : 20087 (python) S 20048 20048 19445 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 747516829 295854080 5636 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 514 18446604443857797184 0 0 -1 2 0 0 0
[pid=20057] ppid=20048 vsize=289336 CPUtime=1.29 cores=0-7
/proc/20057/stat : 20057 (python) S 20048 20048 19445 0 -1 4202560 1144 0 0 0 67 62 0 0 15 0 2 0 747497188 296280064 5745 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/20057/statm: 72334 5745 401 364 0 8249 0
[pid=20059] ppid=20057 vsize=11332 CPUtime=408.75 cores=0-7
/proc/20059/stat : 20059 (march_hi) R 20057 20059 20059 0 -1 4202496 2455 0 0 0 40874 1 0 0 25 0 1 0 747497188 11603968 2191 33554432000 4194304 4955518 140733935430288 18446744073709551615 4224615 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/20059/statm: 2833 2191 104 186 0 2646 0
[pid=20057/tid=20099] ppid=20048 vsize=289336 CPUtime=0 cores=0-7
/proc/20057/task/20099/stat : 20099 (python) S 20048 20048 19445 0 -1 4202560 44 0 0 0 0 0 0 0 16 0 2 0 747542223 296280064 5745 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 514 18446604453907998592 0 0 -1 6 0 0 0
[pid=20060] ppid=20048 vsize=299240 CPUtime=1.28 cores=0-7
/proc/20060/stat : 20060 (python) S 20048 20048 19445 0 -1 4202560 991 0 0 0 69 59 0 0 15 0 1 0 747497191 306421760 5634 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 6540616445934403844 0 0 17 5 0 0 0
/proc/20060/statm: 74810 5634 371 364 0 10722 0
[pid=20062] ppid=20060 vsize=10868 CPUtime=380.63 cores=0-7
/proc/20062/stat : 20062 (march_hi) R 20060 20062 20062 0 -1 4202496 2161 0 0 0 38063 0 0 0 25 0 1 0 747497192 11128832 1896 33554432000 4194304 4955518 140734957602736 18446744073709551615 4227677 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/20062/statm: 2717 1896 104 186 0 2530 0
[pid=20063] ppid=20048 vsize=309948 CPUtime=1.24 cores=0-7
/proc/20063/stat : 20063 (python) S 20048 20048 19445 0 -1 4202560 1175 0 0 0 63 61 0 0 15 0 2 0 747497194 317386752 5760 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/20063/statm: 77487 5760 400 364 0 13396 0
[pid=20065] ppid=20063 vsize=6088 CPUtime=242.24 cores=0-7
/proc/20065/stat : 20065 (march_hi) R 20063 20065 20065 0 -1 4202496 1311 0 0 0 24224 0 0 0 25 0 1 0 747497197 6234112 1046 33554432000 4194304 4955518 140733714518400 18446744073709551615 4209717 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/20065/statm: 1522 1046 104 186 0 1335 0
[pid=20063/tid=20091] ppid=20048 vsize=309948 CPUtime=0 cores=0-7
/proc/20063/task/20091/stat : 20091 (python) S 20048 20048 19445 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 747517126 317386752 5760 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 514 18446604453045365504 0 0 -1 2 0 0 0
[pid=20066] ppid=20048 vsize=320204 CPUtime=0.34 cores=0-7
/proc/20066/stat : 20066 (python) S 20048 20048 19445 0 -1 4202560 1125 0 0 0 19 15 0 0 19 0 2 0 747497198 327888896 5764 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032222667 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/20066/statm: 80051 5764 400 364 0 15957 0
[pid=20068] ppid=20066 vsize=1017800 CPUtime=97.2 cores=0-7
/proc/20068/stat : 20068 (gnovelty+2) T 20066 20068 20068 0 -1 4202496 441 0 0 0 9720 0 0 0 25 0 1 0 747497200 1042227200 177 33554432000 134512640 135663704 4292995664 18446744073709551615 134525678 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/20068/statm: 254450 177 113 282 0 254166 0
[pid=20066/tid=20082] ppid=20048 vsize=320204 CPUtime=0 cores=0-7
/proc/20066/task/20082/stat : 20082 (python) S 20048 20048 19445 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 747511940 327888896 5764 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 514 18446604444072059968 0 0 -1 0 0 0 0
[pid=20069] ppid=20048 vsize=330460 CPUtime=1.25 cores=0-7
/proc/20069/stat : 20069 (python) S 20048 20048 19445 0 -1 4202560 1023 0 0 0 68 57 0 0 15 0 1 0 747497201 338391040 5738 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14454918730557128963 0 0 17 2 0 0 0
/proc/20069/statm: 82615 5738 371 364 0 18518 0
[pid=20071] ppid=20069 vsize=8044 CPUtime=308.54 cores=0-7
/proc/20071/stat : 20071 (march_hi) R 20069 20071 20071 0 -1 4202496 1654 0 0 0 30854 0 0 0 25 0 1 0 747497204 8237056 1388 33554432000 4194304 4955518 140735692014512 18446744073709551615 4227576 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/20071/statm: 2011 1388 104 186 0 1824 0
[pid=20072] ppid=20048 vsize=340716 CPUtime=1.29 cores=0-7
/proc/20072/stat : 20072 (python) S 20048 20048 19445 0 -1 4202560 1035 0 0 0 67 62 0 0 15 0 1 0 747497204 348893184 5759 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 206730180549378307 0 0 17 0 0 0 0
/proc/20072/statm: 85179 5759 371 364 0 21079 0
[pid=20074] ppid=20072 vsize=15212 CPUtime=449.11 cores=0-7
/proc/20074/stat : 20074 (march_hi) R 20072 20074 20074 0 -1 4202496 2918 0 0 0 44906 5 0 0 25 0 1 0 747497207 15577088 2651 33554432000 4194304 4955518 140736667254528 18446744073709551615 4290128 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/20074/statm: 3803 2651 104 186 0 3616 0
[pid=20075] ppid=20048 vsize=351108 CPUtime=1.31 cores=0-7
/proc/20075/stat : 20075 (python) S 20048 20048 19445 0 -1 4202560 1062 0 0 0 64 67 0 0 15 0 1 0 747497208 359534592 5786 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 5990077779767427330 0 0 17 7 0 0 0
/proc/20075/statm: 87777 5786 377 364 0 23674 0
[pid=20077] ppid=20075 vsize=15212 CPUtime=451.56 cores=0-7
/proc/20077/stat : 20077 (march_hi) R 20075 20077 20077 0 -1 4202496 2956 0 0 0 45156 0 0 0 25 0 1 0 747497208 15577088 2683 33554432000 4194304 4955518 140733288599696 18446744073709551615 4224691 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/20077/statm: 3803 2683 104 186 0 3616 0
[pid=20083] ppid=20048 vsize=361364 CPUtime=0.88 cores=0-7
/proc/20083/stat : 20083 (python) S 20048 20048 19445 0 -1 4202560 1037 0 0 0 47 41 0 0 15 0 1 0 747511944 370036736 5784 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14454918730557128961 0 0 17 4 0 0 0
/proc/20083/statm: 90341 5784 371 364 0 26235 0
[pid=20085] ppid=20083 vsize=4172 CPUtime=168.59 cores=0-7
/proc/20085/stat : 20085 (march_hi) R 20083 20085 20085 0 -1 4202496 1058 0 0 0 16858 1 0 0 25 0 1 0 747511944 4272128 788 33554432000 4194304 4955518 140734487616032 18446744073709551615 4227571 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/20085/statm: 1043 788 104 186 0 856 0
[pid=20088] ppid=20048 vsize=437156 CPUtime=0.67 cores=0-7
/proc/20088/stat : 20088 (python) S 20048 20048 19445 0 -1 4202560 1030 0 0 0 37 30 0 0 18 0 1 0 747516832 447647744 5795 33554432000 4194304 5683644 140735968644192 18446744073709551615 260020416070 0 0 16781312 514 14454918730557128960 0 0 17 1 0 0 0
/proc/20088/statm: 109289 5795 371 364 0 45180 0
[pid=20090] ppid=20088 vsize=182444 CPUtime=159.74 cores=0-7
/proc/20090/stat : 20090 (kcnfs-2006) R 20088 20090 20090 0 -1 4202496 597 0 0 0 15974 0 0 0 25 0 1 0 747516833 186822656 330 33554432000 134512640 135163436 4289090528 18446744073709551615 134564823 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/20090/statm: 45611 330 77 159 0 45451 0
[pid=20048/tid=20055] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20055/stat : 20055 (python) S 20046 20048 19445 0 -1 4202560 27 931 0 0 0 0 0 0 25 0 11 0 747497185 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20058] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20058/stat : 20058 (python) S 20046 20048 19445 0 -1 4202560 28 931 0 0 0 0 0 0 18 0 11 0 747497188 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 6 0 0 0
[pid=20048/tid=20061] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20061/stat : 20061 (python) S 20046 20048 19445 0 -1 4202560 20 931 0 0 0 0 0 0 25 0 11 0 747497191 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
[pid=20048/tid=20064] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20064/stat : 20064 (python) S 20046 20048 19445 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 11 0 747497194 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20067] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20067/stat : 20067 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 11 0 747497198 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20070] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20070/stat : 20070 (python) S 20046 20048 19445 0 -1 4202560 26 931 0 0 0 0 0 0 25 0 11 0 747497201 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20073] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20073/stat : 20073 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 25 0 11 0 747497205 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20076] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20076/stat : 20076 (python) S 20046 20048 19445 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 11 0 747497208 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 2 0 0 0
[pid=20048/tid=20084] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20084/stat : 20084 (python) S 20046 20048 19445 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 11 0 747511944 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
[pid=20048/tid=20089] ppid=20046 vsize=447400 CPUtime=0 cores=0-7
/proc/20048/task/20089/stat : 20089 (python) S 20046 20048 19445 0 -1 4202560 26 931 0 0 0 0 0 0 19 0 11 0 747516833 458137600 7472 33554432000 4194304 5683644 140735968644192 18446744073709551615 260032219569 0 0 16781312 2 18446604444416100288 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2775.35
Current children cumulated vsize (KiB) 6064824

Child status: 0
Real time (s): 485.584
CPU time (s): 2775.9
CPU user time (s): 2770.87
CPU system time (s): 5.02823
CPU usage (%): 571.662
Max. virtual memory (cumulated for all children) (KiB): 6064824

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2770.87
system time used= 5.02823
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 50573
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7587
involuntary context switches= 24548

runsolver used 2.45963 second user time and 5.46417 second system time

The end

Launcher Data 

Begin job on node110 at 2011-05-16 21:04:25
IDJOB=3385736
IDBENCH=83001
IDSOLVER=1870
FILE ID=node110/3385736-1305572665
RUNJOBID= node110-1305572593-19459
PBS_JOBID= 13350157
Free space on /tmp= 73504 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v500-c2130-S247862791-097.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385736-1305572665/watcher-3385736-1305572665 -o /tmp/evaluation-result-3385736-1305572665/solver-3385736-1305572665 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385736-1305572665.cnf -seed 2093793560 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 6b69f72e1046222baee8c8d817a4adac
RANDOM SEED=2093793560

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25427236 kB
Buffers:        289488 kB
Cached:        3254660 kB
SwapCached:       1416 kB
Active:        4025916 kB
Inactive:      3162312 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25427236 kB
SwapTotal:    67111528 kB
SwapFree:     67083432 kB
Dirty:           35016 kB
Writeback:          16 kB
AnonPages:     3642520 kB
Mapped:          28744 kB
Slab:           239096 kB
PageTables:      26852 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 75933736 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73472 MiB
End job on node110 at 2011-05-16 21:12:31