Trace number 3321786

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 7991.24 1004.4

General information on the benchmark

Namerandom/large/
unif-k7-r85-v200-c17000-S1348818483-020-UNKNOWN.cnf
MD5SUM112b17aaf1cede32f350e11f6333fc88
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables200
Number of clauses17000
Sum of the clauses size119000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 517000

Solver Data 

0.37/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/0.67	c INFO: solving HOME/instance-3321786-1304367249.cnf
0.37/0.67	c INFO: computing task features
0.45/0.78	c INFO: computed variable-clause graph statistics
0.45/0.78	c INFO: computed clause balance statistics
1.05/1.37	c INFO: computed variable balance statistics
1.25/1.59	c INFO: computed small-clause counts
1.25/1.59	c INFO: computed Horn-clause counts
1.35/1.61	c INFO: computed variable graph statistics
1.35/1.61	c INFO: collected features for HOME/instance-3321786-1304367249.cnf in 0.94 s
1.35/1.64	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.35)
1.35/1.68	c INFO: running gnovelty+2@0 for 97 with 39999 remaining (b = 0.12)
1.45/1.72	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.08)
1.45/1.75	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.06)
1.45/1.79	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.05)
1.55/1.83	c INFO: running gnovelty+2@0 for 97 with 39998 remaining (b = 0.04)
1.55/1.87	c INFO: running FH@0 for 97 with 39998 remaining (b = 0.03)
1.65/1.90	c INFO: running adaptg2wsat2009++@0 for 97 with 39998 remaining (b = 0.03)
800.97/101.87	c INFO: running adaptg2wsat2009++@0 for 97 with 39898 remaining (b = 0.02)
801.70/101.91	c INFO: running FH@0 for 97 with 39898 remaining (b = 0.02)
801.70/101.96	c INFO: running hybridGM3@0 for 194 with 39898 remaining (b = 0.04)
802.18/102.01	c INFO: running gnovelty+2@0 for 97 with 39898 remaining (b = 0.02)
802.18/102.07	c INFO: running hybridGM3@0 for 194 with 39898 remaining (b = 0.03)
802.45/102.13	c INFO: running hybridGM3@0 for 194 with 39898 remaining (b = 0.03)
802.45/102.19	c INFO: running TNM@0 for 1552 with 39898 remaining (b = 0.20)
803.72/102.25	c INFO: running adaptg2wsat2009++@0 for 97 with 39898 remaining (b = 0.01)
1601.01/202.11	c INFO: running FH@0 for 97 with 39798 remaining (b = 0.01)
1601.01/202.18	c INFO: running hybridGM3@0 for 194 with 39798 remaining (b = 0.02)
1601.68/202.27	c INFO: running adaptg2wsat2009++@0 for 97 with 39798 remaining (b = 0.01)
1604.10/202.51	c INFO: running adaptg2wsat2009++@0 for 97 with 39798 remaining (b = 0.01)
2367.96/298.39	c INFO: running hybridGM3@100 for 97 with 39702 remaining (b = 0.01)
2369.53/298.52	c INFO: running hybridGM3@99 for 97 with 39702 remaining (b = 0.01)
2369.53/298.59	c INFO: running hybridGM3@100 for 97 with 39702 remaining (b = 0.01)
2377.49/299.52	c INFO: running adaptg2wsat2009++@0 for 97 with 39701 remaining (b = 0.01)
2379.07/299.75	c INFO: running adaptg2wsat2009++@100 for 97 with 39700 remaining (b = 0.01)
2432.50/306.49	c INFO: running gnovelty+2@0 for 97 with 39694 remaining (b = 0.01)
3129.35/393.83	c INFO: running hybridGM3@0 for 194 with 39606 remaining (b = 0.02)
3153.99/396.99	c INFO: running hybridGM3@0 for 194 with 39603 remaining (b = 0.02)
3158.98/397.69	c INFO: running hybridGM3@100 for 97 with 39603 remaining (b = 0.01)
3175.48/399.64	c INFO: running hybridGM3@295 for 97 with 39601 remaining (b = 0.01)
3176.27/399.76	c INFO: running hybridGM3@295 for 97 with 39600 remaining (b = 0.01)
3200.15/402.77	c INFO: running TNM@0 for 1552 with 39597 remaining (b = 0.14)
3224.10/405.79	c INFO: running adaptg2wsat2009++@100 for 97 with 39594 remaining (b = 0.01)
3934.44/494.88	c INFO: running hybridGM3@396 for 97 with 39505 remaining (b = 0.01)
3935.28/495.00	c INFO: running hybridGM3@395 for 97 with 39505 remaining (b = 0.01)
3967.11/498.93	c INFO: running FH@0 for 291 with 39501 remaining (b = 0.03)
3999.81/503.04	c INFO: running hybridGM3@0 for 194 with 39497 remaining (b = 0.01)
4695.01/590.29	c INFO: running iPAWS@0 for 97 with 39410 remaining (b = 0.01)
4712.51/592.48	c INFO: running adaptg2wsat2009++@99 for 97 with 39408 remaining (b = 0.01)
4741.98/596.13	c INFO: running FH@100 for 291 with 39404 remaining (b = 0.02)
4742.79/596.26	c INFO: running iPAWS@0 for 97 with 39404 remaining (b = 0.01)
5487.33/689.74	c INFO: running hybridGM3@97 for 97 with 39311 remaining (b = 0.01)
5541.38/696.53	c INFO: running hybridGM3@0 for 194 with 39304 remaining (b = 0.01)
5542.08/696.63	c INFO: running TNM@0 for 1552 with 39304 remaining (b = 0.11)
5597.83/703.65	c INFO: running adaptg2wsat2009++@99 for 97 with 39297 remaining (b = 0.01)
6261.91/787.01	c INFO: running adaptg2wsat2009++@97 for 97 with 39213 remaining (b = 0.01)
6273.82/788.57	c INFO: running FH@0 for 97 with 39212 remaining (b = 0.01)
6364.58/799.95	c INFO: running gnovelty+2@0 for 97 with 39200 remaining (b = 0.01)
7035.77/884.29	c INFO: running hybridGM3@0 for 194 with 39116 remaining (b = 0.01)
7047.70/885.75	c INFO: running adaptg2wsat2009++@386 for 291 with 39115 remaining (b = 0.02)
7072.44/888.87	c INFO: running iPAWS@0 for 97 with 39111 remaining (b = 0.01)
7163.14/900.28	c INFO: running FH@201 for 194 with 39100 remaining (b = 0.01)
7186.27/903.16	c INFO: running hybridGM3@102 for 97 with 39097 remaining (b = 0.01)
7870.95/989.17	c INFO: running hybridGM3@0 for 194 with 39011 remaining (b = 0.01)
7984.76/1003.47	c INFO: running NCVWr@0 for 485 with 38997 remaining (b = 0.03)
7991.24/1004.36	c INFO: running FH@100 for 291 with 38996 remaining (b = 0.02)
7991.24/1004.36	Traceback (most recent call last):
7991.24/1004.36	  File "HOME/solve", line 33, in <module>
7991.24/1004.36	    plac.call(main)
7991.24/1004.36	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
7991.24/1004.36	    cmd, result = parser_from(obj).consume(arglist)
7991.24/1004.36	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
7991.24/1004.36	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
7991.24/1004.36	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
7991.24/1004.36	    (_, answer, _) = solver(input_path, budget, cores)
7991.24/1004.36	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
7991.24/1004.36	    solver.go(max_cost * borg.defaults.machine_speed)
7991.24/1004.36	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
7991.24/1004.36	    self._process.start()
7991.24/1004.36	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
7991.24/1004.36	    assert self._popen is None, 'cannot start a process twice'
7991.24/1004.36	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321786-1304367249/watcher-3321786-1304367249 -o /tmp/evaluation-result-3321786-1304367249/solver-3321786-1304367249 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321786-1304367249.cnf -seed 1155635814 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.63 7.68 5.11 1/219 15137
/proc/meminfo: memFree=19821552/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=15380 CPUtime=0 cores=0-7
/proc/15137/stat : 15137 (python) S 15135 15137 14806 0 -1 4202496 792 0 0 0 0 0 0 0 17 0 1 0 548325319 15749120 679 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640857600 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/15137/statm: 3845 679 322 364 0 332 0

[startup+0.0920899 s]
/proc/loadavg: 8.63 7.68 5.11 1/219 15137
/proc/meminfo: memFree=19821552/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=84472 CPUtime=0.05 cores=0-7
/proc/15137/stat : 15137 (python) R 15135 15137 14806 0 -1 4202496 2399 0 0 0 4 1 0 0 17 0 1 0 548325319 86499328 2127 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640857600 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/15137/statm: 21118 2127 707 364 0 1335 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 84472

[startup+0.101089 s]
/proc/loadavg: 8.63 7.68 5.11 1/219 15137
/proc/meminfo: memFree=19821552/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=91828 CPUtime=0.06 cores=0-7
/proc/15137/stat : 15137 (python) R 15135 15137 14806 0 -1 4202496 2677 0 0 0 5 1 0 0 17 0 1 0 548325319 94031872 2392 33554432000 4194304 5683644 140735003009744 18446744073709551615 47995326759200 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15137/statm: 22957 2393 754 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 91828

[startup+0.301063 s]
/proc/loadavg: 8.63 7.68 5.11 1/219 15137
/proc/meminfo: memFree=19821552/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=135748 CPUtime=0.19 cores=0-7
/proc/15137/stat : 15137 (python) S 15135 15137 14806 0 -1 4202496 5969 932 1 0 14 5 0 0 18 0 1 0 548325319 139005952 4151 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640855557 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/15137/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.70098 s]
/proc/loadavg: 8.63 7.68 5.11 1/219 15137
/proc/meminfo: memFree=19821552/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=279616 CPUtime=0.45 cores=0-7
/proc/15137/stat : 15137 (python) R 15135 15137 14806 0 -1 4202496 9681 932 1 0 33 12 0 0 17 0 1 0 548325319 286326784 7244 33554432000 4194304 5683644 140735003009744 18446744073709551615 4602436 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15137/statm: 69904 7244 1761 364 0 5828 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 279616

[startup+1.50085 s]
/proc/loadavg: 8.63 7.68 5.11 2/220 15142
/proc/meminfo: memFree=19795876/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=282588 CPUtime=1.25 cores=0-7
/proc/15137/stat : 15137 (python) R 15135 15137 14806 0 -1 4202496 11333 932 1 0 113 12 0 0 18 0 1 0 548325319 289370112 8060 33554432000 4194304 5683644 140735003009744 18446744073709551615 4500320 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15137/statm: 70647 8060 1827 364 0 6571 0
Current children cumulated CPU time (s) 1.25
Current children cumulated vsize (KiB) 282588

[startup+3.1015 s]
/proc/loadavg: 8.02 7.57 5.09 9/244 15166
/proc/meminfo: memFree=19750176/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=426940 CPUtime=1.66 cores=0-7
/proc/15137/stat : 15137 (python) S 15135 15137 14806 0 -1 4202496 17166 932 1 0 152 14 0 0 25 0 9 0 548325319 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15137/statm: 106735 7589 1891 364 0 42632 0
[pid=15143] ppid=15137 vsize=279232 CPUtime=0 cores=0-7
/proc/15143/stat : 15143 (python) S 15137 15137 14806 0 -1 4202560 728 0 0 0 0 0 0 0 18 0 1 0 548325484 285933568 5715 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879631 0 0 16781312 514 4465900401605968132 0 0 17 1 0 0 0
/proc/15143/statm: 69808 5715 354 364 0 5726 0
[pid=15145] ppid=15143 vsize=1018616 CPUtime=1.44 cores=0-7
/proc/15145/stat : 15145 (gnovelty+2) R 15143 15145 15145 0 -1 4202496 764 0 0 0 144 0 0 0 25 0 1 0 548325484 1043062784 501 33554432000 134512640 135663704 4292301552 18446744073709551615 134525043 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/15145/statm: 254654 501 113 282 0 254370 0
[pid=15146] ppid=15137 vsize=289624 CPUtime=0 cores=0-7
/proc/15146/stat : 15146 (python) S 15137 15137 14806 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 548325488 296574976 5763 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4224319005285908738 0 0 17 1 0 0 0
/proc/15146/statm: 72406 5763 348 364 0 8321 0
[pid=15148] ppid=15146 vsize=1018616 CPUtime=1.41 cores=0-7
/proc/15148/stat : 15148 (gnovelty+2) R 15146 15148 15148 0 -1 4202496 762 0 0 0 141 0 0 0 25 0 1 0 548325488 1043062784 501 33554432000 134512640 135663704 4286625760 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/15148/statm: 254654 501 113 282 0 254370 0
[pid=15149] ppid=15137 vsize=365416 CPUtime=0 cores=0-7
/proc/15149/stat : 15149 (python) S 15137 15137 14806 0 -1 4202560 700 0 0 0 0 0 0 0 18 0 1 0 548325491 374185984 5940 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4338189652138655743 0 0 17 7 0 0 0
/proc/15149/statm: 91354 5940 348 364 0 27266 0
[pid=15151] ppid=15149 vsize=418500 CPUtime=1.37 cores=0-7
/proc/15151/stat : 15151 (adaptg2wsat2009) R 15149 15151 15151 0 -1 4202496 1064 0 0 0 137 0 0 0 25 0 1 0 548325492 428544000 808 33554432000 134512640 135107411 4289607568 18446744073709551615 134523505 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15151/statm: 104625 808 49 146 0 103501 0
[pid=15152] ppid=15137 vsize=375672 CPUtime=0 cores=0-7
/proc/15152/stat : 15152 (python) S 15137 15137 14806 0 -1 4202560 712 0 0 0 0 0 0 0 18 0 1 0 548325495 384688128 5898 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4437760600516329471 0 0 17 1 0 0 0
/proc/15152/statm: 93918 5898 348 364 0 29827 0
[pid=15154] ppid=15152 vsize=418500 CPUtime=1.34 cores=0-7
/proc/15154/stat : 15154 (adaptg2wsat2009) R 15152 15154 15154 0 -1 4202496 1064 0 0 0 134 0 0 0 23 0 1 0 548325495 428544000 810 33554432000 134512640 135107411 4290081536 18446744073709551615 134523448 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/15154/statm: 104625 810 49 146 0 103501 0
[pid=15155] ppid=15137 vsize=385928 CPUtime=0 cores=0-7
/proc/15155/stat : 15155 (python) S 15137 15137 14806 0 -1 4202560 722 0 0 0 0 0 0 0 18 0 1 0 548325499 395190272 6022 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4381884519104380927 0 0 17 1 0 0 0
/proc/15155/statm: 96482 6022 348 364 0 32388 0
[pid=15157] ppid=15155 vsize=418500 CPUtime=1.29 cores=0-7
/proc/15157/stat : 15157 (adaptg2wsat2009) R 15155 15157 15157 0 -1 4202496 1065 0 0 0 129 0 0 0 25 0 1 0 548325499 428544000 808 33554432000 134512640 135107411 4288058512 18446744073709551615 134523331 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15157/statm: 104625 808 49 146 0 103501 0
[pid=15158] ppid=15137 vsize=396184 CPUtime=0 cores=0-7
/proc/15158/stat : 15158 (python) S 15137 15137 14806 0 -1 4202560 729 0 0 0 0 0 0 0 18 0 1 0 548325502 405692416 6026 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4381888367395045636 0 0 17 4 0 0 0
/proc/15158/statm: 99046 6026 348 364 0 34949 0
[pid=15160] ppid=15158 vsize=1018616 CPUtime=1.26 cores=0-7
/proc/15160/stat : 15160 (gnovelty+2) R 15158 15160 15160 0 -1 4202496 757 0 0 0 126 0 0 0 25 0 1 0 548325503 1043062784 500 33554432000 134512640 135663704 4288611600 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15160/statm: 254654 500 113 282 0 254370 0
[pid=15161] ppid=15137 vsize=406440 CPUtime=0 cores=0-7
/proc/15161/stat : 15161 (python) S 15137 15137 14806 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 548325506 416194560 5918 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4281366341459378175 0 0 17 0 0 0 0
/proc/15161/statm: 101610 5918 348 364 0 37510 0
[pid=15163] ppid=15161 vsize=246900 CPUtime=1.22 cores=0-7
/proc/15163/stat : 15163 (FH) R 15161 15163 15163 0 -1 4202496 938 0 0 0 122 0 0 0 25 0 1 0 548325506 252825600 678 33554432000 134512640 134936988 4287524128 18446744073709551615 134523462 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15163/statm: 61725 678 37 104 0 61229 0
[pid=15164] ppid=15137 vsize=416696 CPUtime=0 cores=0-7
/proc/15164/stat : 15164 (python) S 15137 15137 14806 0 -1 4202560 773 0 0 0 0 0 0 0 18 0 1 0 548325510 426696704 6048 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4144478518190800895 0 0 17 4 0 0 0
/proc/15164/statm: 104174 6048 354 364 0 40071 0
[pid=15166] ppid=15164 vsize=418500 CPUtime=1.17 cores=0-7
/proc/15166/stat : 15166 (adaptg2wsat2009) R 15164 15166 15166 0 -1 4202496 1073 0 0 0 117 0 0 0 25 0 1 0 548325510 428544000 809 33554432000 134512640 135107411 4288671680 18446744073709551615 134523360 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15166/statm: 104625 809 49 146 0 103501 0
[pid=15137/tid=15144] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15144/stat : 15144 (python) S 15135 15137 14806 0 -1 4202560 25 932 0 0 0 0 0 0 19 0 9 0 548325484 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 6 0 0 0
[pid=15137/tid=15147] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15147/stat : 15147 (python) S 15135 15137 14806 0 -1 4202560 24 932 0 0 0 0 0 0 18 0 9 0 548325488 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 3 0 0 0
[pid=15137/tid=15150] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15150/stat : 15150 (python) S 15135 15137 14806 0 -1 4202560 93 932 0 0 0 0 0 0 19 0 9 0 548325491 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15153] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15153/stat : 15153 (python) S 15135 15137 14806 0 -1 4202560 31 932 0 0 0 0 0 0 18 0 9 0 548325495 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 5 0 0 0
[pid=15137/tid=15156] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15156/stat : 15156 (python) S 15135 15137 14806 0 -1 4202560 32 932 0 0 0 0 0 0 19 0 9 0 548325499 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15159] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15159/stat : 15159 (python) S 15135 15137 14806 0 -1 4202560 32 932 0 0 0 0 0 0 21 0 9 0 548325502 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 4 0 0 0
[pid=15137/tid=15162] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15162/stat : 15162 (python) S 15135 15137 14806 0 -1 4202560 34 932 0 0 0 0 0 0 23 0 9 0 548325506 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15165] ppid=15135 vsize=426940 CPUtime=0 cores=0-7
/proc/15137/task/15165/stat : 15165 (python) S 15135 15137 14806 0 -1 4202560 86 932 0 0 0 0 0 0 25 0 9 0 548325510 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 12.16
Current children cumulated vsize (KiB) 8318880

[startup+6.30092 s]
/proc/loadavg: 8.02 7.57 5.09 9/244 15166
/proc/meminfo: memFree=19746828/32951124 swapFree=67111364/67111528
[pid=15137] ppid=15135 vsize=426940 CPUtime=1.66 cores=0-7
/proc/15137/stat : 15137 (python) S 15135 15137 14806 0 -1 4202496 17166 932 1 0 152 14 0 0 25 0 9 0 548325319 437186560 7589 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15137/statm: 106735 7589 1891 364 0 42632 0
[pid=15143] ppid=15137 vsize=279232 CPUtime=0 cores=0-7
/proc/15143/stat : 15143 (python) S 15137 15137 14806 0 -1 4202560 879 0 0 0 0 0 0 0 18 0 1 0 548325484 285933568 5738 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879631 0 0 16781312 514 4465900401605968132 0 0 17 1 0 0 0
/proc/15143/statm: 69808 5738 377 364 0 5726 0
[pid=15145] ppid=15143 vsize=1018616 CPUtime=4.64 cores=0-7
/proc/15145/stat : 15145 (gnovelty+2) R 15143 15145 15145 0 -1 4202496 764 0 0 0 464 0 0 0 25 0 1 0 548325484 1043062784 501 33554432000 134512640 135663704 4292301552 18446744073709551615 134524768 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/15145/statm: 254654 501 113 282 0 254370 0
[pid=15146] ppid=15137 vsize=289624 CPUtime=0 cores=0-7
/proc/15146/stat : 15146 (python) S 15137 15137 14806 0 -1 4202560 891 0 0 0 0 0 0 0 18 0 1 0 548325488 296574976 5786 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4224319005285908738 0 0 17 1 0 0 0
/proc/15146/statm: 72406 5786 371 364 0 8321 0
[pid=15148] ppid=15146 vsize=1018616 CPUtime=4.61 cores=0-7
/proc/15148/stat : 15148 (gnovelty+2) R 15146 15148 15148 0 -1 4202496 762 0 0 0 461 0 0 0 25 0 1 0 548325488 1043062784 501 33554432000 134512640 135663704 4286625760 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/15148/statm: 254654 501 113 282 0 254370 0
[pid=15149] ppid=15137 vsize=365416 CPUtime=0 cores=0-7
/proc/15149/stat : 15149 (python) S 15137 15137 14806 0 -1 4202560 774 0 0 0 0 0 0 0 18 0 1 0 548325491 374185984 5962 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4338189652138623235 0 0 17 7 0 0 0
/proc/15149/statm: 91354 5962 370 364 0 27266 0
[pid=15151] ppid=15149 vsize=418636 CPUtime=4.57 cores=0-7
/proc/15151/stat : 15151 (adaptg2wsat2009) R 15149 15151 15151 0 -1 4202496 1211 0 0 0 457 0 0 0 25 0 1 0 548325492 428683264 955 33554432000 134512640 135107411 4289607568 18446744073709551615 134531903 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15151/statm: 104659 955 59 146 0 103535 0
[pid=15152] ppid=15137 vsize=375672 CPUtime=0 cores=0-7
/proc/15152/stat : 15152 (python) S 15137 15137 14806 0 -1 4202560 788 0 0 0 0 0 0 0 18 0 1 0 548325495 384688128 5921 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4437760600516296961 0 0 17 1 0 0 0
/proc/15152/statm: 93918 5921 371 364 0 29827 0
[pid=15154] ppid=15152 vsize=418636 CPUtime=4.53 cores=0-7
/proc/15154/stat : 15154 (adaptg2wsat2009) R 15152 15154 15154 0 -1 4202496 1211 0 0 0 453 0 0 0 25 0 1 0 548325495 428683264 957 33554432000 134512640 135107411 4290081536 18446744073709551615 134531456 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/15154/statm: 104659 957 59 146 0 103535 0
[pid=15155] ppid=15137 vsize=385928 CPUtime=0 cores=0-7
/proc/15155/stat : 15155 (python) S 15137 15137 14806 0 -1 4202560 798 0 0 0 0 0 0 0 18 0 1 0 548325499 395190272 6045 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4381884519104348417 0 0 17 1 0 0 0
/proc/15155/statm: 96482 6045 371 364 0 32388 0
[pid=15157] ppid=15155 vsize=418636 CPUtime=4.45 cores=0-7
/proc/15157/stat : 15157 (adaptg2wsat2009) R 15155 15157 15157 0 -1 4202496 1212 0 0 0 445 0 0 0 25 0 1 0 548325499 428683264 955 33554432000 134512640 135107411 4288058512 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15157/statm: 104659 955 59 146 0 103535 0
[pid=15158] ppid=15137 vsize=396184 CPUtime=0 cores=0-7
/proc/15158/stat : 15158 (python) S 15137 15137 14806 0 -1 4202560 810 0 0 0 0 0 0 0 18 0 1 0 548325502 405692416 6049 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4381888367395045636 0 0 17 4 0 0 0
/proc/15158/statm: 99046 6049 371 364 0 34949 0
[pid=15160] ppid=15158 vsize=1018616 CPUtime=4.42 cores=0-7
/proc/15160/stat : 15160 (gnovelty+2) R 15158 15160 15160 0 -1 4202496 757 0 0 0 442 0 0 0 25 0 1 0 548325503 1043062784 500 33554432000 134512640 135663704 4288611600 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15160/statm: 254654 500 113 282 0 254370 0
[pid=15161] ppid=15137 vsize=406440 CPUtime=0 cores=0-7
/proc/15161/stat : 15161 (python) S 15137 15137 14806 0 -1 4202560 814 0 0 0 0 0 0 0 18 0 1 0 548325506 416194560 5941 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4281366341459345665 0 0 17 0 0 0 0
/proc/15161/statm: 101610 5941 371 364 0 37510 0
[pid=15163] ppid=15161 vsize=248652 CPUtime=4.41 cores=0-7
/proc/15163/stat : 15163 (FH) R 15161 15163 15163 0 -1 4202496 1119 0 0 0 441 0 0 0 25 0 1 0 548325506 254619648 857 33554432000 134512640 134936988 4287524128 18446744073709551615 134554229 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15163/statm: 62163 857 41 104 0 61667 0
[pid=15164] ppid=15137 vsize=416696 CPUtime=0 cores=0-7
/proc/15164/stat : 15164 (python) S 15137 15137 14806 0 -1 4202560 853 0 0 0 0 0 0 0 18 0 1 0 548325510 426696704 6070 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4144478518190768386 0 0 17 4 0 0 0
/proc/15164/statm: 104174 6070 376 364 0 40071 0
[pid=15166] ppid=15164 vsize=418636 CPUtime=4.34 cores=0-7
/proc/15166/stat : 15166 (adaptg2wsat2009) R 15164 15166 15166 0 -1 4202496 1219 0 0 0 434 0 0 0 25 0 1 0 548325510 428683264 955 33554432000 134512640 135107411 4288671680 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15166/statm: 104659 955 59 146 0 103535 0

################
# More data... #
################

[pid=15235] ppid=15233 vsize=457124 CPUtime=600.92 cores=0-7
/proc/15235/stat : 15235 (TNM) R 15233 15235 15235 0 -1 4202496 1117 0 0 0 60091 1 0 0 25 0 1 0 548365597 468094976 837 33554432000 134512640 135008748 4287393552 18446744073709551615 134527734 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15235/statm: 114281 837 46 122 0 113181 0
[pid=15240] ppid=15137 vsize=570536 CPUtime=0.37 cores=0-7
/proc/15240/stat : 15240 (python) S 15137 15137 14806 0 -1 4202560 1185 0 0 0 20 17 0 0 19 0 2 0 548375623 584228864 6251 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15240/statm: 142634 6251 408 364 0 78486 0
[pid=15242] ppid=15240 vsize=268428 CPUtime=199.8 cores=0-7
/proc/15242/stat : 15242 (hybridGM3) T 15240 15242 15242 0 -1 4202496 5176 0 0 0 19979 1 0 0 25 0 1 0 548375624 274870272 4897 33554432000 134512640 135204385 4287080320 18446744073709551615 134538016 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15242/statm: 67107 4897 63 169 0 66937 0
[pid=15240/tid=15264] ppid=15137 vsize=570536 CPUtime=0 cores=0-7
/proc/15240/task/15264/stat : 15264 (python) S 15137 15137 14806 0 -1 4202560 80 0 0 0 0 0 0 0 19 0 2 0 548395675 584228864 6251 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604442868132608 0 0 -1 1 0 0 0
[pid=15246] ppid=15137 vsize=580792 CPUtime=0.19 cores=0-7
/proc/15246/stat : 15246 (python) S 15137 15137 14806 0 -1 4202560 1177 0 0 0 10 9 0 0 19 0 2 0 548384349 594731008 6334 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/15246/statm: 145198 6334 408 364 0 81047 0
[pid=15248] ppid=15246 vsize=335996 CPUtime=102.29 cores=0-7
/proc/15248/stat : 15248 (iPAWS) T 15246 15248 15248 0 -1 4202496 1601 0 0 0 10229 0 0 0 25 0 1 0 548384350 344059904 1320 33554432000 134512640 135114423 4294665552 18446744073709551615 134516967 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15248/statm: 83999 1320 59 147 0 83851 0
[pid=15246/tid=15257] ppid=15137 vsize=580792 CPUtime=0 cores=0-7
/proc/15246/task/15257/stat : 15257 (python) S 15137 15137 14806 0 -1 4202560 78 0 0 0 0 0 0 0 19 0 2 0 548394969 594731008 6334 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604442868130816 0 0 -1 1 0 0 0
[pid=15250] ppid=15137 vsize=591048 CPUtime=0.18 cores=0-7
/proc/15250/stat : 15250 (python) S 15137 15137 14806 0 -1 4202560 1176 0 0 0 9 9 0 0 19 0 2 0 548384946 605233152 6391 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/15250/statm: 147762 6391 401 364 0 83608 0
[pid=15252] ppid=15250 vsize=335996 CPUtime=99.48 cores=0-7
/proc/15252/stat : 15252 (iPAWS) T 15250 15252 15252 0 -1 4202496 1600 0 0 0 9947 1 0 0 25 0 1 0 548384947 344059904 1320 33554432000 134512640 135114423 4289635376 18446744073709551615 134516960 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/15252/statm: 83999 1320 59 147 0 83851 0
[pid=15250/tid=15256] ppid=15137 vsize=591048 CPUtime=0 cores=0-7
/proc/15250/task/15256/stat : 15256 (python) S 15137 15137 14806 0 -1 4202560 80 0 0 0 0 0 0 0 19 0 2 0 548394964 605233152 6391 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604453931310080 0 0 -1 1 0 0 0
[pid=15258] ppid=15137 vsize=601304 CPUtime=0.62 cores=0-7
/proc/15258/stat : 15258 (python) S 15137 15137 14806 0 -1 4202560 1189 0 0 0 33 29 0 0 18 0 2 0 548394973 615735296 6279 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15258/statm: 150326 6279 401 364 0 86169 0
[pid=15260] ppid=15258 vsize=268420 CPUtime=294.42 cores=0-7
/proc/15260/stat : 15260 (hybridGM3) T 15258 15260 15260 0 -1 4202496 5179 0 0 0 29442 0 0 0 25 0 1 0 548394974 274862080 4895 33554432000 134512640 135204385 4293488928 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/15260/statm: 67105 4895 63 169 0 66935 0
[pid=15258/tid=15285] ppid=15137 vsize=601304 CPUtime=0 cores=0-7
/proc/15258/task/15285/stat : 15285 (python) S 15137 15137 14806 0 -1 4202560 81 0 0 0 0 0 0 0 18 0 2 0 548415625 615735296 6279 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604453942287360 0 0 -1 7 0 0 0
[pid=15261] ppid=15137 vsize=611560 CPUtime=0.6 cores=0-7
/proc/15261/stat : 15261 (python) S 15137 15137 14806 0 -1 4202560 987 0 0 0 33 27 0 0 18 0 1 0 548394983 626237440 6337 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 15213423424048169217 0 0 17 0 0 0 0
/proc/15261/statm: 152890 6337 372 364 0 88730 0
[pid=15263] ppid=15261 vsize=457124 CPUtime=306.48 cores=0-7
/proc/15263/stat : 15263 (TNM) R 15261 15263 15263 0 -1 4202496 1118 0 0 0 30646 2 0 0 25 0 1 0 548394983 468094976 837 33554432000 134512640 135008748 4293520432 18446744073709551615 134527111 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/15263/statm: 114281 837 46 122 0 113181 0
[pid=15268] ppid=15137 vsize=621816 CPUtime=0.19 cores=0-7
/proc/15268/stat : 15268 (python) S 15137 15137 14806 0 -1 4202560 1196 0 0 0 10 9 0 0 19 0 2 0 548404177 636739584 6361 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15268/statm: 155454 6361 402 364 0 91291 0
[pid=15270] ppid=15268 vsize=248648 CPUtime=100.09 cores=0-7
/proc/15270/stat : 15270 (FH) T 15268 15270 15270 0 -1 4202496 1139 0 0 0 10009 0 0 0 25 0 1 0 548404177 254615552 856 33554432000 134512640 134936988 4292625600 18446744073709551615 134544814 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15270/statm: 62162 856 41 104 0 61666 0
[pid=15268/tid=15279] ppid=15137 vsize=621816 CPUtime=0 cores=0-7
/proc/15268/task/15279/stat : 15279 (python) S 15137 15137 14806 0 -1 4202560 79 0 0 0 0 0 0 0 19 0 2 0 548414196 636739584 6361 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604453891429696 0 0 -1 6 0 0 0
[pid=15271] ppid=15137 vsize=632072 CPUtime=0.2 cores=0-7
/proc/15271/stat : 15271 (python) S 15137 15137 14806 0 -1 4202560 1232 0 0 0 10 10 0 0 19 0 2 0 548405315 647241728 6426 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15271/statm: 158018 6426 407 364 0 93852 0
[pid=15273] ppid=15271 vsize=1018616 CPUtime=99.83 cores=0-7
/proc/15273/stat : 15273 (gnovelty+2) T 15271 15273 15273 0 -1 4202496 792 0 0 0 9983 0 0 0 25 0 1 0 548405315 1043062784 501 33554432000 134512640 135663704 4288357984 18446744073709551615 134524677 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15273/statm: 254654 501 113 282 0 254370 0
[pid=15271/tid=15284] ppid=15137 vsize=632072 CPUtime=0 cores=0-7
/proc/15271/task/15284/stat : 15284 (python) S 15137 15137 14806 0 -1 4202560 78 0 0 0 0 0 0 0 19 0 2 0 548415337 647241728 6426 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604453879545984 0 0 -1 1 0 0 0
[pid=15276] ppid=15137 vsize=642328 CPUtime=0.24 cores=0-7
/proc/15276/stat : 15276 (python) S 15137 15137 14806 0 -1 4202560 1037 0 0 0 14 10 0 0 18 0 1 0 548413749 657743872 6458 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 15213423424048169219 0 0 17 0 0 0 0
/proc/15276/statm: 160582 6458 378 364 0 96413 0
[pid=15278] ppid=15276 vsize=268424 CPUtime=118.99 cores=0-7
/proc/15278/stat : 15278 (hybridGM3) R 15276 15278 15278 0 -1 4202496 5181 0 0 0 11898 1 0 0 25 0 1 0 548413749 274866176 4895 33554432000 134512640 135204385 4291890624 18446744073709551615 134537804 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15278/statm: 67106 4895 63 169 0 66936 0
[pid=15280] ppid=15137 vsize=652584 CPUtime=0.2 cores=0-7
/proc/15280/stat : 15280 (python) S 15137 15137 14806 0 -1 4202560 1253 0 0 0 11 9 0 0 19 0 2 0 548414207 668246016 6473 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652686283 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15280/statm: 163146 6473 408 364 0 98974 0
[pid=15282] ppid=15280 vsize=335996 CPUtime=99.7 cores=0-7
/proc/15282/stat : 15282 (iPAWS) T 15280 15282 15282 0 -1 4202496 1605 0 0 0 9970 0 0 0 25 0 1 0 548414207 344059904 1321 33554432000 134512640 135114423 4291864448 18446744073709551615 134514760 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15282/statm: 83999 1321 59 147 0 83851 0
[pid=15280/tid=15288] ppid=15137 vsize=652584 CPUtime=0 cores=0-7
/proc/15280/task/15288/stat : 15288 (python) S 15137 15137 14806 0 -1 4202560 88 0 0 0 0 0 0 0 19 0 2 0 548424226 668246016 6473 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 514 18446604453904140224 0 0 -1 1 0 0 0
[pid=15289] ppid=15137 vsize=662840 CPUtime=0.02 cores=0-7
/proc/15289/stat : 15289 (python) S 15137 15137 14806 0 -1 4202560 1063 0 0 0 1 1 0 0 18 0 1 0 548424236 678748160 6490 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 15213423424048169217 0 0 17 1 0 0 0
/proc/15289/statm: 165710 6490 379 364 0 101535 0
[pid=15291] ppid=15289 vsize=268420 CPUtime=14.94 cores=0-7
/proc/15291/stat : 15291 (hybridGM3) R 15289 15291 15291 0 -1 4202496 5185 0 0 0 1494 0 0 0 25 0 1 0 548424237 274862080 4896 33554432000 134512640 135204385 4294451984 18446744073709551615 134538375 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15291/statm: 67105 4896 63 169 0 66935 0
[pid=15292] ppid=15137 vsize=673096 CPUtime=0 cores=0-7
/proc/15292/stat : 15292 (python) S 15137 15137 14806 0 -1 4202560 912 0 0 0 0 0 0 0 19 0 1 0 548425666 689250304 6524 33554432000 4194304 5683644 140735003009744 18446744073709551615 253640879686 0 0 16781312 514 4391858189079937023 0 0 17 0 0 0 0
/proc/15292/statm: 168274 6524 345 364 0 104096 0
[pid=15294] ppid=15292 vsize=414068 CPUtime=0.73 cores=0-7
/proc/15294/stat : 15294 (NCVWr) R 15292 15294 15294 0 -1 4202496 989 0 0 0 73 0 0 0 22 0 1 0 548425667 424005632 700 33554432000 134512640 135113558 4292854512 18446744073709551615 134526474 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15294/statm: 103517 700 55 147 0 102392 0
[pid=15137/tid=15144] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15144/stat : 15144 (python) S 15135 15137 14806 0 -1 4202560 25 932 0 0 0 0 0 0 19 0 34 0 548325484 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 6 0 0 0
[pid=15137/tid=15147] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15147/stat : 15147 (python) S 15135 15137 14806 0 -1 4202560 24 932 0 0 0 0 0 0 18 0 34 0 548325488 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 3 0 0 0
[pid=15137/tid=15150] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15150/stat : 15150 (python) S 15135 15137 14806 0 -1 4202560 104 932 0 0 0 0 0 0 15 0 34 0 548325491 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15153] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15153/stat : 15153 (python) S 15135 15137 14806 0 -1 4202560 35 932 0 0 0 0 0 0 18 0 34 0 548325495 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 5 0 0 0
[pid=15137/tid=15156] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15156/stat : 15156 (python) S 15135 15137 14806 0 -1 4202560 50 932 0 0 0 0 0 0 18 0 34 0 548325499 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15159] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15159/stat : 15159 (python) S 15135 15137 14806 0 -1 4202560 32 932 0 0 0 0 0 0 21 0 34 0 548325502 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 4 0 0 0
[pid=15137/tid=15162] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15162/stat : 15162 (python) S 15135 15137 14806 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 34 0 548325506 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15165] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15165/stat : 15165 (python) S 15135 15137 14806 0 -1 4202560 93 932 0 0 0 0 0 0 18 0 34 0 548325510 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 4 0 0 0
[pid=15137/tid=15172] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15172/stat : 15172 (python) S 15135 15137 14806 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 34 0 548335507 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 6 0 0 0
[pid=15137/tid=15177] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15177/stat : 15177 (python) S 15135 15137 14806 0 -1 4202560 45 932 0 0 0 0 0 0 18 0 34 0 548335511 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 5 0 0 0
[pid=15137/tid=15181] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15181/stat : 15181 (python) S 15135 15137 14806 0 -1 4202560 50 932 0 0 0 0 0 0 18 0 34 0 548335516 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 7 0 0 0
[pid=15137/tid=15184] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15184/stat : 15184 (python) S 15135 15137 14806 0 -1 4202560 33 932 0 0 0 0 0 0 19 0 34 0 548335521 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15189] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15189/stat : 15189 (python) S 15135 15137 14806 0 -1 4202560 48 932 0 0 0 0 0 0 18 0 34 0 548335527 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 4 0 0 0
[pid=15137/tid=15193] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15193/stat : 15193 (python) S 15135 15137 14806 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 34 0 548335533 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15196] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15196/stat : 15196 (python) S 15135 15137 14806 0 -1 4202560 35 932 0 0 0 0 0 0 25 0 34 0 548335539 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 2 0 0 0
[pid=15137/tid=15199] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15199/stat : 15199 (python) S 15135 15137 14806 0 -1 4202560 42 932 0 0 0 0 0 0 18 0 34 0 548335545 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15207] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15207/stat : 15207 (python) S 15135 15137 14806 0 -1 4202560 34 932 0 0 0 0 0 0 19 0 34 0 548345531 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 7 0 0 0
[pid=15137/tid=15210] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15210/stat : 15210 (python) S 15135 15137 14806 0 -1 4202560 40 932 0 0 0 0 0 0 18 0 34 0 548345538 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15222] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15222/stat : 15222 (python) S 15135 15137 14806 0 -1 4202560 41 932 0 0 0 0 0 0 20 0 34 0 548355969 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15227] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15227/stat : 15227 (python) S 15135 15137 14806 0 -1 4202560 38 932 0 0 0 0 0 0 18 0 34 0 548364702 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15230] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15230/stat : 15230 (python) S 15135 15137 14806 0 -1 4202560 33 932 0 0 0 0 0 0 20 0 34 0 548365019 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15234] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15234/stat : 15234 (python) S 15135 15137 14806 0 -1 4202560 38 932 0 0 0 0 0 0 20 0 34 0 548365597 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15241] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15241/stat : 15241 (python) S 15135 15137 14806 0 -1 4202560 38 932 0 0 0 0 0 0 20 0 34 0 548375624 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15247] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15247/stat : 15247 (python) S 15135 15137 14806 0 -1 4202560 33 932 0 0 0 0 0 0 19 0 34 0 548384349 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15251] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15251/stat : 15251 (python) S 15135 15137 14806 0 -1 4202560 36 932 0 0 0 0 0 0 19 0 34 0 548384946 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15259] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15259/stat : 15259 (python) S 15135 15137 14806 0 -1 4202560 40 932 0 0 0 0 0 0 15 0 34 0 548394973 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15262] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15262/stat : 15262 (python) S 15135 15137 14806 0 -1 4202560 31 932 0 0 0 0 0 0 21 0 34 0 548394983 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15269] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15269/stat : 15269 (python) S 15135 15137 14806 0 -1 4202560 36 932 0 0 0 0 0 0 20 0 34 0 548404177 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15272] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15272/stat : 15272 (python) S 15135 15137 14806 0 -1 4202560 32 932 0 0 0 0 0 0 20 0 34 0 548405315 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15277] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15277/stat : 15277 (python) S 15135 15137 14806 0 -1 4202560 33 932 0 0 0 0 0 0 20 0 34 0 548413749 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 1 0 0 0
[pid=15137/tid=15281] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15281/stat : 15281 (python) S 15135 15137 14806 0 -1 4202560 35 932 0 0 0 0 0 0 20 0 34 0 548414207 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15290] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15290/stat : 15290 (python) S 15135 15137 14806 0 -1 4202560 36 932 0 0 0 0 0 0 19 0 34 0 548424237 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
[pid=15137/tid=15293] ppid=15135 vsize=683340 CPUtime=0 cores=0-7
/proc/15137/task/15293/stat : 15293 (python) S 15135 15137 14806 0 -1 4202560 34 932 0 0 0 0 0 0 20 0 34 0 548425667 699740160 8125 33554432000 4194304 5683644 140735003009744 18446744073709551615 253652683185 0 0 16781312 2 18446604445725571776 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 7991.24
Current children cumulated vsize (KiB) 32520696

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15137 and gives
#  childrusage.ru_utime.tv_sec=1643
#  childrusage.ru_utime.tv_usec=424161
#  childrusage.ru_stime.tv_sec=7
#  childrusage.ru_stime.tv_usec=239899
# CPU time returned by wait4() is 1650.66
# while last known CPU time is 7991.24
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1004.4
CPU time (s): 7991.24
CPU user time (s): 7984.19
CPU system time (s): 7.05
CPU usage (%): 795.622
Max. virtual memory (cumulated for all children) (KiB): 32520696

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1643.42
system time used= 7.2399
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 114294
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 13328
involuntary context switches= 7127

runsolver used 8.72967 second user time and 19.759 second system time

The end

Launcher Data 

Begin job on node106 at 2011-05-02 22:14:09
IDJOB=3321786
IDBENCH=82564
IDSOLVER=1712
FILE ID=node106/3321786-1304367249
RUNJOBID= node106-1304366468-14820
PBS_JOBID= 13181764
Free space on /tmp= 73436 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k7-r85-v200-c17000-S1348818483-020-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321786-1304367249/watcher-3321786-1304367249 -o /tmp/evaluation-result-3321786-1304367249/solver-3321786-1304367249 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321786-1304367249.cnf -seed 1155635814 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 112b17aaf1cede32f350e11f6333fc88
RANDOM SEED=1155635814

node106.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19821708 kB
Buffers:        321852 kB
Cached:        3593640 kB
SwapCached:         88 kB
Active:       10247796 kB
Inactive:      2622072 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19821708 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           32228 kB
Writeback:           0 kB
AnonPages:     8953912 kB
Mapped:          23056 kB
Slab:           169280 kB
PageTables:      25616 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 10339244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73404 MiB
End job on node106 at 2011-05-02 22:30:56